Installing software on Linux without root : managing packages in user space

First step : don’t. Reconsider. There really isn’t an alternative ? Using something else that is already installed ? Sweet-talking your sysadmin into doing the installation ? Giving up that particular task ? Giving up Computer Sciences altogether and moving to the country to raise pigs ?

Ok, so you really don’t have an alternative. May the gods have mercy on your soul, because Linux won’t. By necessity, this won’t be a step-by-step guide, because each system has its quirks. I’m not promising you heaven, just a tolerable purgatory instead of a living hell.

Take a deep breath.

(And as always : follow those instructions at your own risk. If you turn your Linux box into a Linux brick, or into Skynet, I’m not liable.)

The problem : dependency hell

There’s a reason why installing software from sources is so painful : dependencies. Sure, you just want to install Caffe, or Torch7, or Theano. But Theano needs python, python needs openssl, openssl needs… it’s endless.

High-level package managers like apt-get and yum are so popular because they deal with those. When installing from source, you’re on your own.

But here’s the catch : when installing from sources, you can almost always relocate the software to your ~home, bypassing the need for root access. High-level package managers, at least the current generation, can’t relocate.

Except for Homebrew.

The strategy : Linuxbrew

Homebrew was created as “the missing package manager for OS X”, and is required to do anything interesting on a Mac. It was designed around two principles : installation at the user home, and installation from sources.

Say that again ? Installation at the user home, without need for root. From sources. Wow. If only there was a version for Linux ! Enter Linuxbrew. Homebrew concept was so successful that, in an ironic turn, it’s now becoming “the missing package manager for Linux”.

So, case closed ? Hardly. To start, Linuxbrew has dependencies of its own, and you have to take care of those by hand. Then, the software you want to install has to be available as a brew “formula” (but the list is quite comprehensive, and growing). Finally, it doesn’t always goes smoothly. Linuxbrew is a much bumpier ride than Homebrew/OS X, at least for now. Most formulas will install without issue, but a good 20% will require tweaking, googling, and deciphering forum posts.

The strategy is most definitely not user-friendly. But contrarily to installing each and every package by hand it is just user-bearable enough to be feasible. If you really need the software. (Are you sure you don’t prefer the pig farm ?)

Okay, you are sure.

Our strategy will be :

  1. Ensuring Linuxbrew dependencies ;
  2. Installing and configuring Linuxbrew ;
  3. Using Linuxbrew to install the desired software… ;
  4. …or if you’re unlucky, using Linuxbrew to install the desired software dependencies, and then installing the desired software by hand.

The tatics

Installing Linuxbrew dependencies

Linuxbrew has, fortunately, few dependencies : Ruby, GCC, Git, and… Linux (duh !). It runs on x86 or 32-bit ARM platforms. If you’re running Linux in other architectures this is your cue to break down sobbing.

Most systems will, fortunately have those dependencies already installed. You can check the minimal versions currently required by Linuxbrew, and then check the versions installed at your system (and whether they are installed at all) calling the commands with --version :

$ ruby --version
ruby 1.8.7 (2011-12-28 patchlevel 357) [x86_64-linux]

$ gcc --version
gcc (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973]
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ git --version
If 'git' is not a typo you can run the following command to lookup the package that contains the binary:
command-not-found git
-bash: git: command not found

Linuxbrew won’t turn your dependency hell into a package management heaven, but it might turn it into a tolerable purgatory.

Linuxbrew won’t turn your dependency hell into a package management heaven, but it might turn it into a tolerable purgatory.

As you see, I got almost lucky. Ruby and GCC are good to go, but I’ll have to install Git (and its dependencies).

The trick is being minimalist. Many packages have optional features : extra commands, documentation, GUI interfaces, etc., that will fire zillions of extra dependencies. Get rid of those optionals as much as possible.

I recommend a top-down plan of attack. Start with the package you want to install and try to configure–make–install it. If it breaks (and it will, it will), find out which dependency/configuration is missing and correct it. Do this recursively, depth-first, until you get everything installed.

Resist perfectionism. You might spend a lot of time smoothing out every wrinkle of package-1.73.03 just to find a bit later that it breaks your installation and has to be removed to make room for package-1.73.04. This is war, kid : not a time for bells and whistles. Once you get a dependency working, move on to the next.

In more detail, each cycle will consist of :

  1.  Finding, downloading, and unpacking the source package ;
  2.  Configuring the package to work with a $HOME prefix ;
  3. Building the package ;
  4. Installing the package.

Step 1 is usually trivial after bit of googling. If your Linux distribution is Debian-based, you might be able to use a single command-line operation :

apt-get source git

There are similar facilities for other high-level package managers.

Otherwise, you might download either a compressed source file, or even the bleeding edge version from the source repository (sourceforge, github, etc.) In the case of Git, this would be at https://github.com/git/git. (Be careful, because those hyper-up-to-date versions might be unstable.)

Step 2 varies a bit from package to package, but usually consists in calling a ./configure script. Sometimes pre-configuration is involved : a call to make configure or make config, or another script, e.g., ./buildconf. Sometimes it involves cmake (cross your fingers for having autoconf/automake already installed). Sometimes there’s no step 2, all options being passed directly to make during step 3. It varies.

How will you know ? Try to locate a INSTALL.* or README.* file. Usually the instructions are there. No luck ? Try browsing the official website of the package for installations instructions. Googling <package> installation instructions usually will point you to the right direction.

For git, this will work :

cd git-2.1.4/
./configure --prefix=$HOME

Well, sort of. It will probably break, because one or more dependencies will be missing. Install those (and their recursive dependencies) and try again.

Step 3 is almost always :

make

or sometimes :

make all

Sometimes this is the moment when things break down for lack of dependencies (or wrong versions, or wrong settings, or the universe showing its lack of care). Sometimes the --prefix=$HOME option comes here instead of Step 2.

Step 4 is almost always :

make install

If you set the prefixes right, that will automagically put everything in the right place, under your ~home directory. And you won’t need root permissions.

Got it ? Good. I hope you enjoy typing command-line commands : you’ll be doing it all day. For extra enjoyment, get a second monitor and close the shades.

Installing Linuxbrew

Once you have all dependencies working, installing Linuxbrew itself is a breeze :

git clone https://github.com/Homebrew/linuxbrew.git ~/.linuxbrew

Aaand… that’s it. It won’t work immediately because you have to set the paths (see below). After you do it you can simply type :

brew install $WHATEVER_YOU_WANT

And it should take care of everything.

Before you do it, however it is a good idea to call

brew doctor

and check if everything is ok. Again, be minimalist : you don’t have to correct every tiny issue. Take a good look and make the smallest needed intervention.

Linuxbrew comes ready with a lot of recipes for installing packages, or as they call, formulas. You can keep them up do date by typing

brew update

Depending on what you want to install, however, you’ll need extra formulas. In Homebrew/Linuxbrew parlance this is called tapping. For example :

brew tap homebrew/science

will install a lot of new formulas related to science, data analysis, etc.

PATH configurations

Both phases (manual dependency installations; Linuxbrew operation) won’t do you much good if your paths aren’t configured. There are at least four important paths, maybe more depending on your setup : executables (PATH), static libraries (LIBRARY_PATH), dynamic libraries (LD_LIBRARY_PATH), and include files (CPATH).

The usual place to set up those is your your shell configuration file. The examples below assume you’re using bash. If that’s your case, decide whether .bashrc or .bash_profile is better for you (usually it’s the former).

During the manual installation of dependencies add the following lines :

# Manually installed packages
export PATH="$HOME/bin:$PATH"
export LIBRARY_PATH="$HOME/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="$HOME/lib:$LD_LIBRARY_PATH"
export CPATH="$HOME/include:$CPATH"

During Linuxbrew operation put those additional lines :

# HomeBrew / LinuxBrew
export HOMEBREW_PREFIX="$HOME/.linuxbrew"
export PATH="$HOMEBREW_PREFIX/bin:$PATH"
export LIBRARY_PATH="$HOMEBREW_PREFIX/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="$HOMEBREW_PREFIX/lib:$LD_LIBRARY_PATH"
export CPATH="$HOMEBREW_PREFIX/include:$CPATH"

Remember that shell configurations are not effective immediately, only on the next start. You don’t have to reboot the system : simple closing and reopening the terminal, or logging out and back in suffices.

An ugly ugly ugly workaround

During my installations, I faced an issue with CA certificates that I could not bypass. Many formulas would refuse to proceed, stopping during download with the error : “cURL error 60: SSL certificate problem: unable to get local issuer certificate”.

Yes : I tried downloading updated certificates from Mozilla Corp. Yes : I checked my curl-config --ca. Yes : I tried reinstalling cURL. And Git. And OpenSSL. I spent, litteraly, hours trying to solve the problem in an elegant way.

I concede defeat. Here’s the very inelegant solution. Be aware that it opens your package manager to man-in-the-middle attacks. That is more than a theoretical risk : it has been done. This is a huge security hole. If you decide to apply it, don’t do it preemptively, wait to see if you’ll actually get the SSL certificate problem.

So you got the error, and you’re willing to expose your neck to the wolves ? Sucks to be you. Open the file download_strategy.rb at ~/.linuxbrew/Library/Homebrew and find the lines below

# Curl options to be always passed to curl,
# with raw head calls (`curl -I`) or with actual `fetch`.
def _curl_opts
  copts = []
  copts << "--user" << meta.fetch(:user) if meta.key?(:user)
  copts
end

Change line four to

# Curl options to be always passed to curl,
# with raw head calls (`curl -I`) or with actual `fetch`.
def _curl_opts
  copts = ["-k"] # Disable certificate verification
  copts << "--user" << meta.fetch(:user) if meta.key?(:user)
  copts
end

And that’s it. You’re ready to proceed installing source packages. And to be a victim of cyber mafias, and of terrorists, and of tyrannical governments.

(Note to security people : if your watertight security solution makes a system unusable, guess what will happen ?)

Extra tips

First and foremost, source code relocation is not a panacea. Some things require root access, for example, driver installations, kernel recompilations, boot sector modifications, etc. You might want to check if your software require one of those before you start this whole adventure.

You can learn a lot about a formula with the info option

$ brew info python
python: stable 2.7.10 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org
Not installed
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python.rb
==> Dependencies
Build: pkg-config ✘
Required: openssl ✘
Recommended: readline ✘, sqlite ✘, gdbm ✘
Optional: homebrew/dupes/tcl-tk ✘, berkeley-db4 ✘
==> Options
--universal
  Build a universal binary
--with-berkeley-db4
  Build with berkeley-db4 support
--with-poll
  Enable select.poll, which is not fully implemented on OS X (https://bugs.python.org/issue5154)
--with-quicktest
  Run `make quicktest` after the build (for devs; may fail)
--with-tcl-tk
  Use Homebrew's Tk instead of OS X Tk (has optional Cocoa and threads support)
--without-gdbm
  Build without gdbm support
--without-readline
  Build without readline support
--without-sqlite
  Build without sqlite support
--HEAD
  Install HEAD version

Take a good look at the --without-* options because they are sometimes a lifesaver. Some packages have optional GUI extras. They might fire hundreds of useless extra dependencies — especially if you are installing on a headless server.

Sometimes Linuxbrew breaks down for the lack of a dependency, refusing to install it, but will gladly do it if you explicitly ask for it. For example : brew install package1 breaks for the lack of package2, and all it takes is typing brew install package2 and retrying package1. Mysteries.

Installation time is highly unpredictable. Sometimes a small innocent little package will require a precise version of GCC… that Linuxbrew will then have to install from the sources. Time for a tea.

If your installation becomes so corrupted with conflicting packages that you have to restart from scratch (nooooooo !), it can be — small consolation — accomplished easily :

rem ~/.linuxbrew -rf

For extra OCD cred, clean-up the download cache as well :

rm ~/.cache/Homebrew/ -rf

If the whole thing becomes so messed up that you have to scratch even the manual dependencies (two words : pig farm), it is also easily done :

rm ~/bin/ ~/lib/ -rf

You might also consider :

rm ~/include ~/etc -rf

but be careful because that might erase innocent third parties.

You might be forced to install multiple versions of the same package. That adds another nightmare layer to the ongoing nightmare, but it’s doable. Linuxbrew will usually be friendly enough to tell you what to do.

For example, when I had to install both opencv2 and opencv3 I got this :

opencv3 and opencv install many of the same files.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/home/valle/.linuxbrew/opt/opencv3/lib
    CPPFLAGS: -I/home/valle/.linuxbrew/opt/opencv3/include

Those little displays of care are the reason why I like Homebrew/Linuxbrew so much. Love breeds love : a truism even for human–computer interaction.

One thought on “Installing software on Linux without root : managing packages in user space

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s