Withdrawal Syndrome

Ceremonial tea whiskAlthough I have my personal opinion, the effort of ranking things like Ubuntu Linux, Microsoft Windows and Apple OS X from ‘better to worse’ is clearly Lilliputian. Therefore I shall abstain.

But I’ll report that I was recently given a Windows 8 laptop, and an Android ‘Jelly Bean’ tablet. Much to my embarrassment, I am finding both barely operatable.

Whilst I’ve been under the pain and delight of OS X and iOS, Windows went through several transformations, and is now barely recognizable from the last time we’ve met (XP, if you believe). Meanwhile, Android is for me completely new (I went directly from Simbian to iOS) and equally alien.

As a wide-eyed Computer Science student, I was put off by the design philosophy of OS X and iOS : the ‘my way or the highway’ appearance, the monastic simplicity, the detachment between ends and means achieved through an extreme cloaking of the mechanisms. The system reeked of coercion and… insincerity.

Now, as an overworked Engineering professor, the same dreaded features became desirable. I no longer want to choose one from 50 color schemes for my window manager, or worse, design my own color scheme. I want, out of the box, one beautiful color scheme that works. I know : it is kinda sad, and totally against the hacker ethos. But nowadays, the investment of the scarce time I have for hacking is decided very judiciously.

Once upon a time, the Rube Goldberg design of Windows and Android would make my delight for weeks — no checkmark would go unexplored, no deep dialog box would remain unopened. It was a given that I would have to reinstall the system from scratch a few times, for I would “customize” (i.e., torture) it to the point of hopelessness.

I wouldn’t mind the updates. I wouldn’t mind the nagging confirmations. I wouldn’t mind having to reinstall the system from scratch to get rid of all ‘free !’ useless/noxious third-party bundled software. I wouldn’t mind the inconsistent GUI design. I wouldn’t mind that the system, fresh out of the box, would let a driver do a “fatal exception”, or an essential application “execute an illegal operation” — either demanding hours of troubleshooting.

All that mattered was that once everything was sorted out — days or weeks later — I would get a system customized and optimized to my exacting tastes. A system that kept no secrets from me. A system of my own.

Magic BulletNowadays, I want to easily unwrap a beautiful package — at each step feeling that my gestures follow a rhythm etched in the very fabric of the continuum — and have the hardware satisfyingly fall in the void of my hands, heavily filling the negative space between my fingers, caressing my skin with the touch of finely crafted rich materials. The software comes alive with a barely conscious command. A few discrete questions and the device and I are one, its assimilation into my collective accomplished, my diffusion into its anatomy complete and unnoticed.

For this privilege, I am willing to go as far (the horror !) as actually buying an Apple product.

There must be a 12-steps program for that.

EDIT 17/01 : I was complaining about my Windows 8 experience on twitter, and this dialog happened. I have to recognize they are charming.

IT Industry and Users — a lesson in advanced BDSM

Case study :  you have an hour-long education video that you want to upload to your YouTube channel. The video needs minor editing : (1) trimming and eliminating a few portions; (2) splicing-in another short video; (3) maybe normalizing the sound.

On a 2011 MacBook Pro using OS X, your options are :

Use iMovie (bundled with OS X) — it will take 30′ importing the clips, then you’ll have editing capabilities much more advanced than you need. But it will take 10 hours for it to reencode the final video, and of course, there will be a compounded quality loss due to the double reencoding ;

Use Quicktime X Player (bundled with OS X) — contrarily to what “player” would suggest, Quicktime X gives you recording abilities, and very minor editing abilities, like trimming and gluing takes, without having to reencode the video. Sounds perfect, no ? The trimming feature is, however, practically useless, since it has very poor resolution (what is the use of  editing if you’ll end up having people being cut in the middle of a utterance ?). And you cannot remove inner portions of a video, just trim the edges : although you can go around this by trimming and gluing several times, for very large videos, it quickly becomes annoying.

Use Quicktime 7 Pro (US$ 30 on Apple Store US) — the old version of QuickTime has much finer trimming abilities, and it will allow you to remove inner portions of videos. Free trumps bought any day, but the price is not bad,  considering the time it could save. (What really ruffles my feathers is that Apple is basically selling you abandonware 30 bucks-a-piece, but Apple’s greediness shouldn’t really surprise anyone by now.) After you go through all the hoops to find out how to install QT7 on OS X Mountain Lion or Mavericks  (it does not work if you don’t get exactly the right version), you find the deal breaker : it is not available in Apple Store Brazil, and Apple Store US does not accept foreign credit cards.

(This is getting ridiculous.)

Use Avidemux (Free as in speech, and as in beer) — Jackpot ! This open-source project is meant to solve exactly your problem : make minor editing on videos without having to reencode the whole thing (although it has advanced reencoding abilities as well). Installing it is a bit of a hassle : you do all the clicks and have no sound on output. Some googling tells you that the problem is a preferences setting, but you can’t find the “Preferences” panel because a freaky bug has blanked out most GUI elements, including most of the menu items. One hour later, you finally have sound (having found out which blank menu item is “Preferences”, and which blank tab is “Sounds options”, and which blank field is “Sound device”). All that to find out that the preview window has a time delay on the play/pause/ff/rev buttons that makes it completely useless for precise editing purposes.

(If it weren’t four in the morning I would scream.)

Use ffmpeg (Free as in speech, and as in beer) — Video editing on the command-line : you may be an Apple user by choice, but even for you there is such a thing as too much masochism.

Use Final Cut Pro (US$ 300 in Apple Store US), Adobe Premiere CC (US$ 30 per month for Teachers and Students in US) — In order to cut and paste video pieces. Are you kidding ?

(Okay, you see where this is going —  I’ve tried and discarded many other solutions. For sake of curiosity my browser history shows :  “H.264 Cutter”, “Why am I doing this to myself”, “MPEG Streamclip”, “Mp4Split”, “Can a MacBook Survive a 4 story fall”, “Xilisoft Video cutter”, etc.)

And I didn’t find a solution ! In the end, I sucked it up and went with iMovies. Grudgingly.

Computer Users, we have to do something about this play : this is neither safe, nor sane. But Computer Industry, if you are hearing, this is definitely not consensual.

Update 13/dec : A typo in the link for my YouTube channel was misdirecting people to the fake tiniurl URL shortener, instead of the real tinyurl. My bad.

Update 18/dec : The story is, unfortunately, not over. Now iMovie refuses to generate an output with a mystifying “error 49” that, according to a Google Search, might mean everything and the kitchen sink. I’ve reverted to a rough editing with Quicktime X and got an output… without audio. I am arriving at the conclusion that I should give up IT entirely and move to the country to raise pigs. This whole “computing” idea should just be filed as “tried, did not work”.

Time Machine menu on notification areaUpdate 19/dec : apparently, iMovie and Time Machine don’t play nice with each other — which is one of the possible causes of the error 49. Unless you are rendering very short videos, it is better to disable Time Machine (click on the Time Machine icon on the notification bar — upper left corner — then “Open Time Machine Preferences”, then switch the big “On/Off” button. Remember to turn it on again once the Finalization or Export are over !). This worked for me, but it is no guarantee to make the pestering error disappear.

Sharing Mathematica with Yourself

Maybe I forgot to click on some half-hidden checkbox when I’ve installed it, but I’ve found out that my Mathematica 9 copy was working only for one of the users in my MacBook. That is annoying, because I keep separate users for my everyday usage,  and for giving presentations and classes (so there is zero risk that one of my friends suddenly appears on Skype telling a dirty joke in the middle of a presentation to the Schools’ president).

But I’ve found out the problem is easy to remediate. There are three folders where Mathematica 9 searches for the license files : $BaseDirectory/Licensing, $InstallationDirectory/Configuration/Licensing, and $UserBaseDirectory/Licensing (open a new Mathematica notebook and type the commands $BaseDirectory, etc. to find out exactly what the paths are in your system). Sure enough, mine was in $UserBaseDirectory/Licensing — meaning it was accessible by just that user.

This simple sequence of commands solved the problem :


$ sudo su
$ mkdir $BaseDirectory
$ mv $UserBaseDirectory/Licensing $BaseDirectory

Again, be sure to substitute the $variables above by the correct paths. I’ve double checked the permissions, and mine were already ok (all users had reading permissions). If it’s not the case for you, try this command :

$ chmod -R a+rX $BaseDirectory

And that is all. (I hope that doesn’t violate any terms, but I can’t see why it would : this kind of Mathematica license is per machine, and even if it were per user, well, both users are the same person, and they are never both “on” at once, isn’t it ?)

I don’t know if this Mathematica single-machine/multi-user license problem (or solution) applies for systems other than Mac OS X — if you find out, I’d be glad to know.

Apple Blues

I don’t know what is happening. Maybe the ghost of Steve Jobs is haunting my devices. Or maybe Apple devices only work as supposed for the fervent believers. The fact is that both my iPhone 4S and my MacBook Pro (13″, circa 2010) took a nasty turn South since the last software update.

Both are experiencing network issues. The iPhone is often on Edge, when everyone around in on 3G. Turning it off and on, or turning 3G off and on in Preferences sometimes solves the problem. The MacBook is having issues with WiFi access, the nasty “!” now appears often over the menu bar icon — again rebooting the device usually solves the problem.

But the MacBook Pro is being the most infuriating. The rainbow wheel of hell has decided to test my patience, appearing even after seemingly innocent actions (changing fields in a web form, for example). Only a month ago, I have reinstalled OS X in this machine from scratch because it had become too slow, so the reappearance of this problem after such a short interval is being particularly annoying.

The main problem, I guess, is psychological. Whenever my Apple devices start to act, I can’t avoid feeling cheated. I have payed a premium to have something that works smoothly, out of the box. If I am to Google for solutions every other Wednesday, and spend 3 hours hacking configuration files, I might as well switch to a cheaper — and more open — platform.

(Erratum 17/2: I have also blamed an autocorrect bug on the new iOS updated, but it seems that it only occurs in Safari, and then only in a few sites, so I have been hasty in my anger.)

Apple Mail, Migration Assistant and Sandboxing

I’ve discovered the hard way that OS X’s Migration Assistant does not play nice with sandboxed applications, even its own. After moving my user to another machine, everything seemed all right, but mail kept asking me for account details.

I didn’t want to reconfigure my email — and re-download 10 Gb worth of past history — so I decided to investigate what was the matter. The standard instructions haven’t worked and the reason is that Apple Mail runs sandboxed in its container. In order to make those instructions to work, the following adaptations are needed :

  1. In the source machine, the com.apple.mail.plist file will not be found in ~/Library/Preferences, but in ~/Library/Containers/com.apple.mail/Data/Library/Preferences/
  2. In the destination machine, check if ~/Library/Containers/com.apple.mail already exists. If it does, rename it to something else (like com.apple.mail.old)
  3. Follow all other instructions unchanged — they can be used to restore/repair Apple Mail after a migration with Migration Assistant, or to just transfer Apple Mail from a machine to another. I was in the former case, and omitted the Keychain transfer step, without any problems — I figured it was transferred by Migration Assistant.

When you run Mail on the first time in the destination machine, it will not find the sandboxed container and it will recreate it with the correct data and settings provided out of the sandbox.

I wonder if similar procedures — mutatis mutandis — apply for other sandboxed applications that encounter migration troubles. If you find out, please drop me a note.

Mac OS X, Word and the quest for the unbloated PDF

Hard-science scholars are strange people, who insist on using TeX because it “typesets beautifully”, but then forget to check badness warnings, letting the lines spill beyond the right margin. I have resisted TeX as much as I could, until I finally caved to peer pressure. Still, I only use it for cooperative work : when all by myself, I want something, let’s say frankly, less Jurassic.

Still, I am forced to envy my frozen-in-1975 colleagues, when I find out that saving to PDF, an operation that the industry should had gotten right by now, turns my 1 MB Microsoft Word file into an 80 MB PDF-zilla.

I’ve spent a good part of my morning solving that problem, considering both the official solution, and more independente initiatives. The official solution flunked when I’ve found out that Adobe had no trial of Acrobat for Mac (am I really willing to spent US$ 500 just to find out whether or not it’ll do what I want?). I tried a PDF compression solution, PDF Shrink, which reduced my PDF… from 89 MB to 87 MB, while mangling horribly all the images: not exactly worth the US$ 35. I’ve also tried recreating the PDF from scratch, but PDF Studio, at US$ 125, just refused to open the Word file with a cryptic ‘error reading’ message. I was glad both were in trial.

In despair, I continued searching the Web. Lots of users crying “Large PDF !”, “Word PDF too big !”, “Huge PDFs on Mac !”, but very few answers. Industry, why u no listen ?

They say that we should never attribute to malice that which can be explained by incompetence. But Hanlon’s razor notwithstanding,  I couldn’t avoid drifting into conspiracy theories. What if that horrible implementation of PDF conversion was not completely accidental ?

Conspiracy theories are unfalsifiable, of course, but I’ll tell you what finally solved the problem and you’ll tell me if it doesn’t make you itsy bitsy suspicious :

  1. On Word, instead of saving to PDF, save to PostScript (using File… Print…, and then, on the print dialog, the PDF button on the lower left corner. The Save to PostsScript is one of the options);
  2. Open the PostScript file (double-click its icon) and let Preview make the automatic conversion;
  3. On Preview, save the file as PDF (using the menu File… Export… — or on older OS X versions File… Save as…)

And that’s all. Now lets check the sizes :

  • Original PDF file (using Save as PDF or Print to PDF from Word) : 89 MB
  • PostScript file (Using Print to PostScript) : 94 MB
  • Final PDF file (using the steps above) : 5 MB

That is, using only tools already present in OS X, and three small steps, I’ve got an almost 18x smaller file. Risking joining the ranks of the ‘moon hoax’ lunatics, I smell something rotten in the current state of PDF conversion implemented by Word–OS X.

* * *

Incidentally, I’ve found something I also needed : how to password-protect PDFs. I was ready to buy a solution, but I’ve found that unnecessary.

When creating a new one, on OS X, you can click on the “Security Options” menu of the “Export as” (“Save as” in older versions) dialog — how come I’ve never remarked that one ?.

If the PDF exists already, you can open it with Preview, go to File… Export… (File… Save as… in older OS X versions), and check the box “Encrypt”. Two textboxes below let you put a password. Save the file and it will be only visible after the password is entered.

EDIT 28/02/12 : I am finding out that the above method is by no means foolproof, i.e., it doesn’t work for every kind of PDF. In particular, I tested it for PDFs generated by PowerPoint, and it backfired (PostScript conversion generated a file much bigger). For image loaded PDFs from PowerPoint, contrarily to the mainly textual ones from Word, I’m finding that the usual tip of using the Quartz Filter (open the PDF file with Preview, then File… Export… [File… Save As… in older OS X versions], then select “Reduce File Size” on the Quartz Filter field in the dialog) works quite well.

EDIT 23/07/13 : I’ve never dreamed this blog entry was to become my most popular one. (Apple and Microsoft, aren’t you listening ?) I’ve edited the procedure above to reflect the change in the Save As… logic introduced in OS X Lion, when it became Export…

Buy me ! Upgrade me ! Register me !

It had happened just once before, but now that Parallels (the virtual environment of choice for Mac users) has launched its version 7, trying to use it is a constant source of irritation. Every other time I open the application, I am greeted with a huge colorful popup ad prompting me to upgrade. The “do not show me this again” checkbox is basically useless. I asked it politely to refrain from soliciting, to no avail.

I remember when shareware was a novel concept, and I used to download it by the dozens : nagging screens were part of the deal, something to be expected until you paid for a registered version. Since when those “features” have become cricket for bought wares ?

After wondering if I should open a support ticket, I’ve just reported the pestering behavior as a bug. Lets see if the development team will agree.

No easy answers

Barely a few months (6, 7 ?) since using Mac OS X and my system has got basically unusable: MacPorts has died and gone to hell, root certificates get systematically rejected by Safari, the system is sssoooo ssssllllloooowwww that using it feels like moving in water. Firefox crashes every few hours. Parallels isn’t working anymore in Coherence mode.

The system is so unreliable that my only option is to reformat the machine and restart from scratch (basically what I had to do when I used XP — but XP usually lasted a good year and a half between reformats).

With that and all the software incompatibility problems, I am seriously considering switching to Windows 7 + Cygwin and thrashing this whole Mac OS experiment in the huge pile of “tried: didn’t work”.

I can haz SciPy !!1!1!

"The Dependency Hell", rightmost panel of Hieronymus Bosch's "The Garden of Earthly Delights".
Well, at least it looks like so (the self-test does not smoke anymore).

I will try to reconstruct a walkthrough, but bear in mind that I have installed, uninstalled and reinstalled so much stuff in my Mac that the reproducibility of this recipe should be taken with a grain of salt. If you try it and it works, I would be thankful if you’d leave a note describing exactly what you did.

Without further ado, how to install the trio NumPy, SciPy and Matplotlib in Mac OS X Snow Leopard, using MacPorts:

  1. Download Apple’s Xcode for Mac OS X, without which nothing else is possible. You’ll have to register on Apple, but there is a free inscription if you are not planning to sell anything;
  2. Download MacPorts. This is the package manager which will operate all the magic — it will do for your Mac the same thing yum or apt-get does for a Linux box, so if it doesn’t work properly nothing else will;
  3. Install Xcode and then MacPorts. If you want to be extra sure, follow the intructions on the Macports page — basically instructing you to install X11 support on Mac and MacPorts. I didn’t do this (because I only became aware of those instruction very late in the whole process), but that doesn’t mean you shouldn’t. Alternatively, if you already have MacPorts installed, ensure it is up-to-date by typing:

    sudo port selfupdate

  4. You can compile your dependencies with any compiler — in theory. I found out that SciPy smokes unless compiled with GCC 4.4, but now I have some dependencies compiled with their default choice and some (NumPy, SciPy) in which I forced compilation with GCC 4.4. If you want to try your luck, start by downloading GCC 4.4 and making it default:

    sudo port install gcc44
    sudo port install gcc_select
    sudo gcc_select mp-gcc44

    If instead, you want to reproduce exactly my crazy (but successful) sequence, install GCC 4.4 but do not make it default (omit the last two lines), and also install GCC 4.5 (I know, how many versions of GCC does one need ?!):

    sudo port install gcc45

  5. Install the “non-dependencies” of PIL. Those “non-dependencies” are libraries needed by other libraries in other to provided optional (but often important) functionality, which won’t be installed by the package manager otherwise (PIL itself is a “non-dependency” of SciPy):

    sudo port install jpeg libpng tiff lcms freefont-ttf

  6. Install PIL. This will install python as a dependency. (I have chosen to install 2.7, so I’ve consistenly chosen py27-* packages through the process):

    sudo port install py27-pil

  7. Install the binary dependencies of NumPy and SciPy:

    sudo port install arpack
    sudo port install SuiteSparse

  8. Finally, install the goodies:

    sudo port install py27-numpy
    sudo port install py27-scipy

    Again, if you want to do exactly what I did, the process is more convoluted. After typing exactly those commands above, I then uninstalled then (but not their dependencies):

    sudo port uninstall py27-scipy py27-matplotlib

    And reinstalled everything again (but the dependencies) with the compiler directive:

    sudo port install py27-numpy configure.compiler=macports-gcc-4.5
    sudo port install py27-numpy configure.compiler=macports-gcc-4.4

    I don’t remember why I chose GCC 4.5 for NumPy and 4.4 for SciPy — but my history file is a more faithful testimony than my memory (probably I just mistyped NumPy’s and meant 4.4 for both — give-me some slack, it was almost 4 a.m. by then !).

  9. Last, but not least, install Matplotlib:

    sudo port install py27-matplotlib

I don’t know if all this mix-and-match of compilers is a good thing — all I know is that SciPy smokes if not compiled with GCC 4.4, but maybe compiling everything with GCC 4.4 is the way to go.

To test the installation, call python and selftest the packages one by one:

python
import numpy
numpy.test(‘1′,’10’)

import scipy
scipy.test(‘1′,’10’)

import matplotlib
matplotlib.test()

I’ve got a pristine regression test for NumPy and SciPy, including the problematic C++ code weaving. My Matplotlib regression smoked, with several “ImageComparisonFailure: images not close” errors with RMS values slightly above acceptable. I compared the images by hand, however, and could not spot the difference — and dismissed the problem as a matter of different engines of font smoothing.

By the way, if you want to make the new python default, there is a python_select for this job:

sudo port install python_select
sudo python_select python27

Both gcc_select and python_select allow you to choose betwen a list of options which include the MacPorts and the Apple versions of GCC and python, to see the options available, type;

gcc_select -l
python_select -l

And a little something I found out at my expense. When using bash, sometimes updating the PATH is not enough: recently used commands will be at a hash table, which will have to be refreshed, lest bash will still be fetching the command at the wrong path. To check and clear this hash table you can use, respectively, the commands:

hash
hash -r

Well, it is not the Seventh Circle of Heaven, but it is half-way through the Purgatory. Not bad for this Dante, who’s got only Google and StackOverflow for Virgile !

Then you just have to add another dependence…

Okay, I am well in the third day of trying to install SciPy on the Mac OS X, and things do not seem they are improving:

  1. I have started by signing on the Apple Developer Program, downloading and installing Xcode (without which nothing else is possible);
  2. I’ve downloaded and installed MacPorts (without which nothing interesting is humanely feasible);
  3. As required, I’ve installed a fresh copy of Python 2.6 from the .dmg, available at python.org;
  4. I’ve installed gfortran from the binary, available at AT&T Research;
  5. Using MacPorts, I’ve installed the SuiteSparse;
  6. I’ve installed numpy from the .dmg available at SourceForge;
  7. I’ve installed the setuputils for python, downloading the egg and running it as a shell script, as indicated on its website;
  8. Using setuputils, I’ve installed nose, the testing framework, and run the smoke test for numpy (import numpy \n numpy.test('1', '10')). It passed perfectly;
  9. I’ve installed scipy from the .dmg available at SourceForge;
  10. I’ve run the smoke test for scipy (import numpy \n scipy.test('1', '10')). It smoked badly at all attempts of code weaving with C(++).

This short version of the story does not include all the comings and goings — installing Python 2.7 and discovering very late in the process its incompatibility with Ccipy for Mac, finding out that the concept of automatic uninstallation simply does not exist in Mac OS X, it has been lots of fun ! It feels just like UNIX, only worse — I have fallen in the rabbit hole and I am waiting for the floor to arrive.

I swear that I am this close to just installing Windows on this damn thing.

EDIT 4/feb: I have just found out this very useful script, Scipy Superpack for Mac, courtesy of Chris Fonnesbeck of the now defunct macinscience.org. My scipy.test() still smokes badly, but now on other, fresh new issues. Apparently, I am not alone: getting SciPy to work on Mac seems to be something short of heroic.