First step : don’t. Reconsider. There really isn’t an alternative ? Using something else that is already installed ? Sweet-talking your sysadmin into doing the installation ? Giving up that particular task ? Giving up Computer Sciences altogether and moving to the country to raise pigs ?
Ok, so you really don’t have an alternative. May the gods have mercy on your soul, because Linux won’t. By necessity, this won’t be a step-by-step guide, because each system has its quirks. I’m not promising you heaven, just a tolerable purgatory instead of a living hell.
Take a deep breath.
(And as always : follow those instructions at your own risk. If you turn your Linux box into a Linux brick, or into Skynet, I’m not liable.)
The problem : dependency hell
There’s a reason why installing software from sources is so painful : dependencies. Sure, you just want to install Caffe, or Torch7, or Theano. But Theano needs python, python needs openssl, openssl needs… it’s endless.
High-level package managers like apt-get and yum are so popular because they deal with those. When installing from source, you’re on your own.
But here’s the catch : when installing from sources, you can almost always relocate the software to your
~home, bypassing the need for root access. High-level package managers, at least the current generation, can’t relocate.
Except for Homebrew.
The strategy : Linuxbrew
Homebrew was created as “the missing package manager for OS X”, and is required to do anything interesting on a Mac. It was designed around two principles : installation at the user home, and installation from sources.
Say that again ? Installation at the user home, without need for root. From sources. Wow. If only there was a version for Linux ! Enter Linuxbrew. Homebrew concept was so successful that, in an ironic turn, it’s now becoming “the missing package manager for Linux”.
So, case closed ? Hardly. To start, Linuxbrew has dependencies of its own, and you have to take care of those by hand. Then, the software you want to install has to be available as a brew “formula” (but the list is quite comprehensive, and growing). Finally, it doesn’t always goes smoothly. Linuxbrew is a much bumpier ride than Homebrew/OS X, at least for now. Most formulas will install without issue, but a good 20% will require tweaking, googling, and deciphering forum posts.
The strategy is most definitely not user-friendly. But contrarily to installing each and every package by hand it is just user-bearable enough to be feasible. If you really need the software. (Are you sure you don’t prefer the pig farm ?)
Okay, you are sure.
Our strategy will be :
- Ensuring Linuxbrew dependencies ;
- Installing and configuring Linuxbrew ;
- Using Linuxbrew to install the desired software… ;
- …or if you’re unlucky, using Linuxbrew to install the desired software dependencies, and then installing the desired software by hand.
Installing Linuxbrew dependencies
Linuxbrew has, fortunately, few dependencies : Ruby, GCC, Git, and… Linux (duh !). It runs on x86 or 32-bit ARM platforms. If you’re running Linux in other architectures this is your cue to break down sobbing.
Most systems will, fortunately have those dependencies already installed. You can check the minimal versions currently required by Linuxbrew, and then check the versions installed at your system (and whether they are installed at all) calling the commands with
$ ruby --version ruby 1.8.7 (2011-12-28 patchlevel 357) [x86_64-linux] $ gcc --version gcc (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973] Copyright (C) 2008 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. $ git --version If 'git' is not a typo you can run the following command to lookup the package that contains the binary: command-not-found git -bash: git: command not foundAs you see, I got almost lucky. Ruby and GCC are good to go, but I’ll have to install Git (and its dependencies).
The trick is being minimalist. Many packages have optional features : extra commands, documentation, GUI interfaces, etc., that will fire zillions of extra dependencies. Get rid of those optionals as much as possible.
I recommend a top-down plan of attack. Start with the package you want to install and try to configure–make–install it. If it breaks (and it will, it will), find out which dependency/configuration is missing and correct it. Do this recursively, depth-first, until you get everything installed.
Resist perfectionism. You might spend a lot of time smoothing out every wrinkle of package-1.73.03 just to find a bit later that it breaks your installation and has to be removed to make room for package-1.73.04. This is war, kid : not a time for bells and whistles. Once you get a dependency working, move on to the next.
In more detail, each cycle will consist of :
- Finding, downloading, and unpacking the source package ;
- Configuring the package to work with a
- Building the package ;
- Installing the package.
Step 1 is usually trivial after bit of googling. If your Linux distribution is Debian-based, you might be able to use a single command-line operation :
apt-get source git
There are similar facilities for other high-level package managers.
Otherwise, you might download either a compressed source file, or even the bleeding edge version from the source repository (sourceforge, github, etc.) In the case of Git, this would be at https://github.com/git/git. (Be careful, because those hyper-up-to-date versions might be unstable.)
Step 2 varies a bit from package to package, but usually consists in calling a
./configure script. Sometimes pre-configuration is involved : a call to
make configure or
make config, or another script, e.g.,
./buildconf. Sometimes it involves cmake (cross your fingers for having autoconf/automake already installed). Sometimes there’s no step 2, all options being passed directly to make during step 3. It varies.
How will you know ? Try to locate a
README.* file. Usually the instructions are there. No luck ? Try browsing the official website of the package for installations instructions. Googling <package> installation instructions usually will point you to the right direction.
For git, this will work :
cd git-2.1.4/ ./configure --prefix=$HOME
Well, sort of. It will probably break, because one or more dependencies will be missing. Install those (and their recursive dependencies) and try again.
Step 3 is almost always :
or sometimes :
Sometimes this is the moment when things break down for lack of dependencies (or wrong versions, or wrong settings, or the universe showing its lack of care). Sometimes the
--prefix=$HOME option comes here instead of Step 2.
Step 4 is almost always :
If you set the prefixes right, that will automagically put everything in the right place, under your
~home directory. And you won’t need root permissions.
Got it ? Good. I hope you enjoy typing command-line commands : you’ll be doing it all day. For extra enjoyment, get a second monitor and close the shades.
Once you have all dependencies working, installing Linuxbrew itself is a breeze :
git clone https://github.com/Homebrew/linuxbrew.git ~/.linuxbrew
Aaand… that’s it. It won’t work immediately because you have to set the paths (see below). After you do it you can simply type :
brew install $WHATEVER_YOU_WANT
And it should take care of everything.
Before you do it, however it is a good idea to call
and check if everything is ok. Again, be minimalist : you don’t have to correct every tiny issue. Take a good look and make the smallest needed intervention.
Linuxbrew comes ready with a lot of recipes for installing packages, or as they call, formulas. You can keep them up do date by typing
Depending on what you want to install, however, you’ll need extra formulas. In Homebrew/Linuxbrew parlance this is called tapping. For example :
brew tap homebrew/science
will install a lot of new formulas related to science, data analysis, etc.
Both phases (manual dependency installations; Linuxbrew operation) won’t do you much good if your paths aren’t configured. There are at least four important paths, maybe more depending on your setup : executables (
PATH), static libraries (
LIBRARY_PATH), dynamic libraries (
LD_LIBRARY_PATH), and include files (
The usual place to set up those is your your shell configuration file. The examples below assume you’re using bash. If that’s your case, decide whether
.bash_profile is better for you (usually it’s the former).
During the manual installation of dependencies add the following lines :
# Manually installed packages export PATH="$HOME/bin:$PATH" export LIBRARY_PATH="$HOME/lib:$LIBRARY_PATH" export LD_LIBRARY_PATH="$HOME/lib:$LD_LIBRARY_PATH" export CPATH="$HOME/include:$CPATH"
During Linuxbrew operation put those additional lines :
# HomeBrew / LinuxBrew export HOMEBREW_PREFIX="$HOME/.linuxbrew" export PATH="$HOMEBREW_PREFIX/bin:$PATH" export LIBRARY_PATH="$HOMEBREW_PREFIX/lib:$LIBRARY_PATH" export LD_LIBRARY_PATH="$HOMEBREW_PREFIX/lib:$LD_LIBRARY_PATH" export CPATH="$HOMEBREW_PREFIX/include:$CPATH"
Remember that shell configurations are not effective immediately, only on the next start. You don’t have to reboot the system : simple closing and reopening the terminal, or logging out and back in suffices.
An ugly ugly ugly workaround
During my installations, I faced an issue with CA certificates that I could not bypass. Many formulas would refuse to proceed, stopping during download with the error : “cURL error 60: SSL certificate problem: unable to get local issuer certificate”.
Yes : I tried downloading updated certificates from Mozilla Corp. Yes : I checked my
curl-config --ca. Yes : I tried reinstalling cURL. And Git. And OpenSSL. I spent, litteraly, hours trying to solve the problem in an elegant way.
I concede defeat. Here’s the very inelegant solution. Be aware that it opens your package manager to man-in-the-middle attacks. That is more than a theoretical risk : it has been done. This is a huge security hole. If you decide to apply it, don’t do it preemptively, wait to see if you’ll actually get the SSL certificate problem.
So you got the error, and you’re willing to expose your neck to the wolves ? Sucks to be you. Open the file download_strategy.rb at ~/.linuxbrew/Library/Homebrew and find the lines below
# Curl options to be always passed to curl, # with raw head calls (`curl -I`) or with actual `fetch`. def _curl_opts copts =  copts << "--user" << meta.fetch(:user) if meta.key?(:user) copts end
Change line four to
# Curl options to be always passed to curl, # with raw head calls (`curl -I`) or with actual `fetch`. def _curl_opts copts = ["-k"] # Disable certificate verification copts << "--user" << meta.fetch(:user) if meta.key?(:user) copts end
And that’s it. You’re ready to proceed installing source packages. And to be a victim of cyber mafias, and of terrorists, and of tyrannical governments.
(Note to security people : if your watertight security solution makes a system unusable, guess what will happen ?)
First and foremost, source code relocation is not a panacea. Some things require root access, for example, driver installations, kernel recompilations, boot sector modifications, etc. You might want to check if your software require one of those before you start this whole adventure.
You can learn a lot about a formula with the info option
$ brew info python python: stable 2.7.10 (bottled), HEAD Interpreted, interactive, object-oriented programming language https://www.python.org Not installed From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python.rb ==> Dependencies Build: pkg-config ✘ Required: openssl ✘ Recommended: readline ✘, sqlite ✘, gdbm ✘ Optional: homebrew/dupes/tcl-tk ✘, berkeley-db4 ✘ ==> Options --universal Build a universal binary --with-berkeley-db4 Build with berkeley-db4 support --with-poll Enable select.poll, which is not fully implemented on OS X (https://bugs.python.org/issue5154) --with-quicktest Run `make quicktest` after the build (for devs; may fail) --with-tcl-tk Use Homebrew's Tk instead of OS X Tk (has optional Cocoa and threads support) --without-gdbm Build without gdbm support --without-readline Build without readline support --without-sqlite Build without sqlite support --HEAD Install HEAD version
Take a good look at the
--without-* options because they are sometimes a lifesaver. Some packages have optional GUI extras. They might fire hundreds of useless extra dependencies — especially if you are installing on a headless server.
Sometimes Linuxbrew breaks down for the lack of a dependency, refusing to install it, but will gladly do it if you explicitly ask for it. For example :
brew install package1 breaks for the lack of package2, and all it takes is typing
brew install package2 and retrying package1. Mysteries.
Installation time is highly unpredictable. Sometimes a small innocent little package will require a precise version of GCC… that Linuxbrew will then have to install from the sources. Time for a tea.
If your installation becomes so corrupted with conflicting packages that you have to restart from scratch (nooooooo !), it can be — small consolation — accomplished easily :
rem ~/.linuxbrew -rf
For extra OCD cred, clean-up the download cache as well :
rm ~/.cache/Homebrew/ -rf
If the whole thing becomes so messed up that you have to scratch even the manual dependencies (two words : pig farm), it is also easily done :
rm ~/bin/ ~/lib/ -rf
You might also consider :
rm ~/include ~/etc -rf
but be careful because that might erase innocent third parties.
You might be forced to install multiple versions of the same package. That adds another nightmare layer to the ongoing nightmare, but it’s doable. Linuxbrew will usually be friendly enough to tell you what to do.
For example, when I had to install both opencv2 and opencv3 I got this :
opencv3 and opencv install many of the same files. Generally there are no consequences of this for you. If you build your own software and it requires this formula, you'll need to add to your build variables: LDFLAGS: -L/home/valle/.linuxbrew/opt/opencv3/lib CPPFLAGS: -I/home/valle/.linuxbrew/opt/opencv3/include
Those little displays of care are the reason why I like Homebrew/Linuxbrew so much. Love breeds love : a truism even for human–computer interaction.