Wow ! Much Homebrew. Very Numpy. So Scipy. Such OpenCV

The first time I tried to install NumPy+SciPy in my Mac, it turned into a Kafkaesque nightmare, out of which I only managed to surface due to luck and grit. (Only to have, a few weeks later, a system update breaking my MacPorts and sending everything back to hell.)

The second time around, I traded freedom for comfort, and went with Enthought Python Distribution (now Enthought Canopy).  EPD came with an impressive list of available packages, and, more importantly : it just worked. It was also generously available at no fee for academic use, an offer from which I’ve profited.

Recently though, I became a latecomer to Homebrew, enticed by their taglines (‘The missing package manager of OS X’, ‘MacPorts driving you to drink ? Try Homebrew !’) and by their oneliner installation procedure (look for ‘Install Homebrew’ at their homepage).

So far, I am incredibly impressed — I’ve done fresh installations of Python, Nose, NumPy, OpenCV, GCC (!), SciPy, Bottleneck, wxPython and PIL. All went smoothly, installing and testing without smoke. My command-line history reveals just how easy it was :

ruby -e "$(curl -fsSL"

brew install python

/usr/local/bin/pip-2.7 install nose

/usr/local/bin/pip-2.7 install numpy

brew install opencv

brew install gcc49

brew install scipy

/usr/local/bin/pip-2.7 install bottleneck

brew install wxwidgets

/usr/local/bin/pip install pil

The only pitfall (if it may even be called so) is that some Python packages prefer the homebrew installer, and some prefer pip — but quick error and trial works just fine to find out.

Often homebrew installer will discreetly guide you through the process, like when I asked ‘brew install wxpython’, and it told me that there was no such package, but that  ‘wxwidgets’ already came with the wxPython bindings. That kind of gentle bending of Unix philosophy, on behalf of preserving the user sanity, never fails to win my respect.

Now : maybe homebrew is running so smoothly only because I have EPD already installed in this machine, all exoteric dependences having been previously solved. I also had Xcode fully installed and operational, a requirement for most interesting tools working on OS X at all. Remark also that I am still running Mountain Lion.

Homebrew’s express requirements seem to be quite modest, however : the Command-line Tools for Xcode, and a bash or zsh-compatible shell (the default terminal is fine). Additionally, it resides in a branch independent from EPD, so it probably can’t cound on the latter’s dependences. In a few weeks, I intend to do a fresh installation of Mavericks on this machine, and we will know for sure.

No easy answers

Barely a few months (6, 7 ?) since using Mac OS X and my system has got basically unusable: MacPorts has died and gone to hell, root certificates get systematically rejected by Safari, the system is sssoooo ssssllllloooowwww that using it feels like moving in water. Firefox crashes every few hours. Parallels isn’t working anymore in Coherence mode.

The system is so unreliable that my only option is to reformat the machine and restart from scratch (basically what I had to do when I used XP — but XP usually lasted a good year and a half between reformats).

With that and all the software incompatibility problems, I am seriously considering switching to Windows 7 + Cygwin and thrashing this whole Mac OS experiment in the huge pile of “tried: didn’t work”.

I can haz SciPy !!1!1!

"The Dependency Hell", rightmost panel of Hieronymus Bosch's "The Garden of Earthly Delights".
Well, at least it looks like so (the self-test does not smoke anymore).

I will try to reconstruct a walkthrough, but bear in mind that I have installed, uninstalled and reinstalled so much stuff in my Mac that the reproducibility of this recipe should be taken with a grain of salt. If you try it and it works, I would be thankful if you’d leave a note describing exactly what you did.

Without further ado, how to install the trio NumPy, SciPy and Matplotlib in Mac OS X Snow Leopard, using MacPorts:

  1. Download Apple’s Xcode for Mac OS X, without which nothing else is possible. You’ll have to register on Apple, but there is a free inscription if you are not planning to sell anything;
  2. Download MacPorts. This is the package manager which will operate all the magic — it will do for your Mac the same thing yum or apt-get does for a Linux box, so if it doesn’t work properly nothing else will;
  3. Install Xcode and then MacPorts. If you want to be extra sure, follow the intructions on the Macports page — basically instructing you to install X11 support on Mac and MacPorts. I didn’t do this (because I only became aware of those instruction very late in the whole process), but that doesn’t mean you shouldn’t. Alternatively, if you already have MacPorts installed, ensure it is up-to-date by typing:

    sudo port selfupdate

  4. You can compile your dependencies with any compiler — in theory. I found out that SciPy smokes unless compiled with GCC 4.4, but now I have some dependencies compiled with their default choice and some (NumPy, SciPy) in which I forced compilation with GCC 4.4. If you want to try your luck, start by downloading GCC 4.4 and making it default:

    sudo port install gcc44
    sudo port install gcc_select
    sudo gcc_select mp-gcc44

    If instead, you want to reproduce exactly my crazy (but successful) sequence, install GCC 4.4 but do not make it default (omit the last two lines), and also install GCC 4.5 (I know, how many versions of GCC does one need ?!):

    sudo port install gcc45

  5. Install the “non-dependencies” of PIL. Those “non-dependencies” are libraries needed by other libraries in other to provided optional (but often important) functionality, which won’t be installed by the package manager otherwise (PIL itself is a “non-dependency” of SciPy):

    sudo port install jpeg libpng tiff lcms freefont-ttf

  6. Install PIL. This will install python as a dependency. (I have chosen to install 2.7, so I’ve consistenly chosen py27-* packages through the process):

    sudo port install py27-pil

  7. Install the binary dependencies of NumPy and SciPy:

    sudo port install arpack
    sudo port install SuiteSparse

  8. Finally, install the goodies:

    sudo port install py27-numpy
    sudo port install py27-scipy

    Again, if you want to do exactly what I did, the process is more convoluted. After typing exactly those commands above, I then uninstalled then (but not their dependencies):

    sudo port uninstall py27-scipy py27-matplotlib

    And reinstalled everything again (but the dependencies) with the compiler directive:

    sudo port install py27-numpy configure.compiler=macports-gcc-4.5
    sudo port install py27-numpy configure.compiler=macports-gcc-4.4

    I don’t remember why I chose GCC 4.5 for NumPy and 4.4 for SciPy — but my history file is a more faithful testimony than my memory (probably I just mistyped NumPy’s and meant 4.4 for both — give-me some slack, it was almost 4 a.m. by then !).

  9. Last, but not least, install Matplotlib:

    sudo port install py27-matplotlib

I don’t know if all this mix-and-match of compilers is a good thing — all I know is that SciPy smokes if not compiled with GCC 4.4, but maybe compiling everything with GCC 4.4 is the way to go.

To test the installation, call python and selftest the packages one by one:

import numpy

import scipy

import matplotlib

I’ve got a pristine regression test for NumPy and SciPy, including the problematic C++ code weaving. My Matplotlib regression smoked, with several “ImageComparisonFailure: images not close” errors with RMS values slightly above acceptable. I compared the images by hand, however, and could not spot the difference — and dismissed the problem as a matter of different engines of font smoothing.

By the way, if you want to make the new python default, there is a python_select for this job:

sudo port install python_select
sudo python_select python27

Both gcc_select and python_select allow you to choose betwen a list of options which include the MacPorts and the Apple versions of GCC and python, to see the options available, type;

gcc_select -l
python_select -l

And a little something I found out at my expense. When using bash, sometimes updating the PATH is not enough: recently used commands will be at a hash table, which will have to be refreshed, lest bash will still be fetching the command at the wrong path. To check and clear this hash table you can use, respectively, the commands:

hash -r

Well, it is not the Seventh Circle of Heaven, but it is half-way through the Purgatory. Not bad for this Dante, who’s got only Google and StackOverflow for Virgile !