In Barcelona for NIPS’2016

I’m in Barcelona now for NIPS’2016 — or should I say, for NIPS’ Symposia and Workshops, since the main conference this year… sold out. That’s at once exciting and frightening: is machine learning the next dot.com?

Anyways — science ! We’re participating, with posters, in two workshops: Adversarial Training (on Friday, 9th) and Bayesian Deep Learning (on Saturday, 10th). Will you be there, let’s talk!

The papers:

Adversarial Images for Variational Autoencoders

Pedro Tabacof, Julia Tavares, Eduardo Valle

We investigate adversarial attacks for autoencoders. We propose a procedure that distorts the input image to mislead the autoencoder in reconstructing a completely different target image. We attack the internal latent representations, attempting to make the adversarial input produce an internal representation as similar as possible as the target’s. We find that autoencoders are much more robust to the attack than classifiers: while some examples have tolerably small input distortion, and reasonable similarity to the target image, there is a quasi-linear trade-off between those aims. We report results on MNIST and SVHN datasets, and also test regular deterministic autoencoders, reaching similar conclusions in all cases. Finally, we show that the usual adversarial attack for classifiers, while being much easier, also presents a direct proportion between distortion on the input, and misdirection on the output. That proportionality however is hidden by the normalization of the output, which maps a linear layer into non-linear probabilities.

The fulltext here:  https://arxiv.org/abs/1612.00155

🔔 The organizers of the workshop have opened a reddit thread for the public to ask questions. We have a subthread there — ask us anything! 🔔

Known Unknowns: Uncertainty Quality in Bayesian Neural Networks

Ramon Oliveira, Pedro Tabacof, Eduardo Valle

We evaluate the uncertainty quality in neural networks using anomaly detection. We extract uncertainty measures (e.g. entropy) from the predictions of candidate models, use those measures as features for an anomaly detector, and gauge how well the detector differentiates known from unknown classes. We assign higher uncertainty quality to candidate models that lead to better detectors. We also propose a novel method for sampling a variational approximation of a Bayesian neural network, called One-Sample Bayesian Approximation (OSBA). We experiment on two datasets, MNIST and CIFAR10. We compare the following candidate neural network models: Maximum Likelihood, Bayesian Dropout, OSBA, and — for MNIST — the standard variational approximation. We show that Bayesian Dropout and OSBA provide better uncertainty information than Maximum Likelihood, and are essentially equivalent to the standard variational approximation, but much faster.

The fulltext here:  https://arxiv.org/abs/1612.01251

My postgraduate offer for 2017/1: Designing IT Research for Impact

I’ll offer, next semester, the subject “IA376 — Topics in Computer Engineering VII — Class D: Designing IT Research for Impact” to the postgraduate students at FEEC/UNICAMP. That will be a 4 credits (60 hours) subject.

When: Tuesdays and Thursdays, from 19 to 21h

Where: FEEC/UNICAMP, Post-graduate Hall, Room PE-24.

We aspire our research to impact the world beyond h-indices and impact factors — we dream that our work, directly or indirectly, improves people’s lives. I start with a tease: do you have any evidence that this is the case for your research? My aim is for us to learn to ask that question, and then to give it a fruitful answer.

Let’s mine academic sources, government policies, NGO/Charities experiences, think-tank statements to answer the following questions:
(1) How can I design research in IT to maximize social impact?
(2) How can I work with the intended communities to to favor that impact?
(3) How can I measure (or at least inspect) that impact?

Our scope will be limited to Information Technology (i.e., “a GIS to improve access to clean water” is within scope, “a new polymer to clean water” not so much), and to disfavored communities (i.e., “real-time social networks to improve sanitization” is within scope, “real-time social networks to exchange designer watches” not so much).

This course will be nothing like your typical classroom experience. There will be no lectures. We will meet for the presencial sessions to discuss previous work, and plan our attack for the next week. I’ll expect you to continue working throughout the week. There will be no exams: I’ll grade your work based on participation during the sessions, progress between sessions, self assessment, and peer assessment. Active participation will be mandatory. This means (surprise!) talking in public. Everyone will be learning together, so all of us must accept the risk to be wrong. This course won’t work for those who always want to appear wise and knowledgeable. Group work will be highly encouraged. The course will be in English.

We’ll be a cozy small group: at most 18 students. I’ll select the candidates based on a letter of intentions, and on previous experience. Write a short e-mail to dovalle@dca.fee.unicamp.br with a short-résumé (American style) summarizing your research, professional, and extra-curricular activities, highlighting experience relevant for the course; add also a short statement (300~600 words) explaining your current research, or future research that you’d like to develop, highlighting the social impact you’d like to see. We will be particularly interested in IT research for Health and Education benefiting disfavored communities, but other themes are welcome.

Attention to the procedures and deadlines: Whether you are a regular or special-enrolled student, please send me your application by e-mail (as explained above) until 8 January 2017. You’ll have also to make your enrollment as usual. If you want to enroll as a “special-enrolled student” act now: pre-enrollment closes on December  7th! Regular students can follow the usual enrollment calendar.

More info at FEEC’s Postgraduation Page.

 

Commencement Speech for the Computer Scientists and Engineers

I had the honor and pleasure to give a commencement speech for the Computer Scientists and Engineers who graduated at UNICAMP yesterday. Prof. Dr. Ricardo Torres, Dean of the Institute of Computing, gave the main speech and addressed the graduates. As Director of Graduate Studies for Computer Engineering, I addressed the public, before him. Below is a translation of what I said ; the original, in Portuguese, is at the end.

My colleagues at the board,
Dearest graduates,
Dear parents,
Ladies and gentlemen.

It is always a joy to participate at this solemnity. Although UNICAMP has many endeavors, certainly Undergraduate Teaching is the most conspicuous, the one that brings us closer to the public, the one that, so to speak, puts us in people’s hearts. Thus, Graduation brings us all — students and professors — a feeling of mission accomplished. I’ll start, therefore, congratulating — but also thanking — the graduates for their achievement.

Congratulations also to the parents, the families ; we know that this victory is also yours.

But as Director of Undergraduate Studies, I think of all our graduates : not only today’s, also tomorrow’s. So I’ll ask for a few minutes of your patience to talk about the courses and professions of Computing.

Professions, I say, in the plural, since of all areas, Informatics is among the largest, most welcoming. I prefer that word, Informatics, because the term Computing puts too much emphasis on calculations, and even on the device itself, the computer. Let’s put the focus where it is due : on information.

Information that changed the world. Who can conceive life without the Internet ? Without Wikipedia ? Or even WhatsApp ? Remember that the availability of Internet to the general public has about 20 years ; Wikipedia has 15 ; WhatsApp has 6. Six years ! What else had such an impact in such a short time ?

British writer Arthur C. Clarke said that any sufficiently advanced technology is indistinguishable from magic. And the recent achievements of Informatics seem indeed a thing of magic.

We have today augmented reality translators, where we film a sign, a marquee, and the screen shows the same image with the text in another language ;

Applications that map our way from home to work, diverting from traffic, and one day warn us “leave out earlier today because there is a traffic jam” ;

And there is also this: [ speaker takes the phone and says ] Find commencement speeches. [ The phone answers ] “Okay, here’s what I found on Internet about commencement speech.’

Magic ! — Or is it ?

As educators, it’s our ambition to remove this veil of magic. Our Universe after all, is not the World of Harry Potter, it’s not the Middle-Earth of Hobbits, it’s not Narnia. No : our world is much more interesting than all those, because it’s a world structured by science. Science maybe doesn’t get the same glamor as magic in Hollywood movies, but its power is ultimately higher — a system to find truth, to understand, and eventually to manipulate reality.

A device like this [smartphone] does not arise by spontaneous generation, it does not grow on trees, and neither is the result of a spell. Each of the thousands — or probably millions — of components and processes, of software and of hardware, that made this object possible, was the result of the deliberate actions of a person. There is no magic : there is a complex network of knowledge, inspiration, and sweat.

That complexity is one strength of Informatics ; it’s what makes the profession so diverse, so full of roles. Some design the hardware, some write the software, some support to the user, some design the interface, some even think of the user experience, from the moment of taking the product out of the box, until the moment of giving it back for recycling. There are people in computing who spend months without writing ten lines of code, and others who produce thousands per month. Some spend their lives dealing with people, mediating, negotiating ; others hide behind their keyboards and, of human interaction, are content with “good morning”. There is room for all inclinations, and for all vocations.

Complexity is also a weakness of Informatics. It is a profession misunderstood by the public, full of myths, stereotypes. I’d like to dispel some of them.

The first myth is that Computing is bureaucratic, without room for creativity. On the contrary, Informatics is creative, expressive. A computer program is, quite literally, a piece of thought fixed on a support, which the programmer shapes, gives a breath of life, and it gains its own independent existence. If that is not creation, it is not expression, then what is ? — And I’m not even touching the more conventionally artistic areas of the profession, such as interface, interaction, and product design.

The second myth is that computing is reserved for nerds, or for the wunderkinder. Look, nothing against nerds — after all, I am one of them — but we have in no way a monopoly of Informatics. It also makes me sad, not to mention furious with our education system, when a 10-year old child, who is having some trouble with the multiplication tables for seven decides (or has it decided for her) that [1] she is not good at Math ; [2] she was not born with the gift of Mathematics ; [3] she should not aspire to a career in STEM. Out with this nonsense ! Math is like everything in life, it’s like jumping rope, it’s like dancing tango : you learn it with effort, you learn it with persistence — it is not a matter of genetics, it is a matter of toil. And the real Mathematics, creative, expressive, useful — the Mathematics that changes the world — has nothing to do with being very good at the seven times table.

The third myth is that the impact of computing is technical, distant from people. Given all that I said, this one should be already gone, but since it’s a very insidious one, I’ll make it explicit. It saddens me to watch young people abandoning a vocation for Computing, for Engineering, and seeking other professions — usually Law or Medicine — because they want to “help people”. Nothing against Law and Medicine, beautiful professions, but that is an awful reason to abandon Computing. Just think a little bit beyond the obvious to realize the enormous impact — social, cultural, ecological, economic, you can pick — a committed computing professional has to a large number of people. What kind of impact has a comprehensive, high quality, free encyclopedia ? A channel of expression where all citizens can post their videos and be watched worldwide, without passing the scrutiny, the censorship of governments, of broadcasters ? A funding social network where any project may come to life, without having to please a conservative committee of investors ? Think of Wikipedia, of YouTube, of Kickstarter, the next time you be tempted to think Computing does not help people.

And finally, I have to attack the old myth — that we still hear — that Informatics is a male profession, a profession reserved for boys. How many talented girls, young women, do not give up Computing because they believe that nonsense ? The result is that still today our course is more sought after by boys than by girls, but fortunately that is changing. We are committed to making our environment increasingly diverse, and to be welcoming to all genders. In that spirit, I pray — I beg, in fact — that parents show enthusiasm for their daughters who become interested in Computing, the same enthusiasm that they usually manifest when the interest comes from their sons. And now I address directly the sisters, cousins, nieces, not still in College age who are here today : yes, Computing is for you, we are counting on you, we are waiting for you when the time comes to enter UNICAMP.

Ladies, gentlemen, I have by now certainly took more of your time than civility allowed. I am greatly thankful for the opportunity to participate in that wonderful moment, and again, congratulating our graduates, I close my speech.

 

Colegas da mesa,
Queridas formandas, queridos formandos,
Senhores pais,
Senhoras e senhores.

É sempre com alegria que participamos desta Solenidade. Embora a UNICAMP desenvolva muitas atividades, certamente o Ensino de Graduação é a mais presente na lembrança, e é a que mais nos aproxima do público, a que, por assim dizer, nos coloca no coração das pessoas. Sendo assim, a Formatura é um momento que nos dá a todos — estudantes e professores — a sensação de dever cumprido. Dessa forma, eu começo, parabenizando — mas também agradecendo — aos formandos por essa conquista.

Parabenizo também aos pais, às famílias, pois sabemos que essa vitória também é de vocês.

Mas sendo Coordenador do Curso, eu devo me preocupar com todos os nossos formandos: não só os de hoje, também os do futuro. Por isso vou pedir alguns minutos da sua paciência para falar dos cursos e das profissões de Computação.

Profissões, digo eu, no plural, pois de todas as áreas, a Informática está entre as mais amplas, mais acolhedoras. Gosto dessa palavra, Informática, porque o termo Computação coloca ênfase demais nos cálculos, e até no próprio aparelho, o computador. Vamos colocar o foco onde é devido, na Informação.

Informação que transformou o mundo. Quem de nós concebe a vida sem Internet ? Sem Wikipedia ? Sem WhatsApp ? Ora, eu quero lembrar que a Internet para o grande público tem cerca de 20 anos ; a Wikipedia, 15 ; o WhatsApp, 6. Seis anos ! Que outras mudanças tiveram tanto impacto em tão pouco tempo ?

O escritor britânico Arthur C. Clarke dizia que qualquer tecnologia suficientemente avançada é indistinguível de magia. E as conquistas recentes da Informática parecem mesmo coisa de mágica :

Temos hoje tradutores de realidade aumentada, em que filmamos uma placa, um letreiro, e a tela nos mostra a mesma imagem com o texto em outra língua ;

Aplicativos que traçam nosso caminho da casa para o trabalho, desviando do trânsito, e um belo dia nos avisam “saia mais cedo hoje, porque há um engarrafamento” ;

E há também isso : [ orador toma o telefone e diz ] Encontre discursos de formatura.   [ o telefone responde ] “Okay, aqui está o que eu encontrei na Internet sobre ‘discurso de formatura’.”

Mágica ! — Ou será mesmo ?

É nossa ambição como educadores remover esse véu da magia. Nosso Universo afinal, não é o Mundo de Harry Potter, não é a Terra Média dos Hobbits, não é Narnia. Não : nosso mundo é muito mais interessante que todos esses, pois é um mundo estruturado pela ciência. A ciência pode não ter o mesmo glamour da magia nos filmes de Hollywood, mas o poder dela é maior — um sistema de encontrar a verdade, que nos permite entender, e finalmente manipular a realidade.

Um aparelho como este [smartphone] não surge por geração espontânea, não cresce em árvores, e nem é resultado de um feitiço. Cada um dos milhares — ou provavelmente milhões — de componentes e processos, de hardware e de software que tornaram este objetivo possível, foi resultado da ação deliberada de uma pessoa. Não existe mágica : existe uma complexa rede de conhecimento, inspiração, e suor.

Essa complexidade é uma das forças da Informática, é ela que torna a profissão tão diversa, tão cheia de papéis. Há quem projete o hardware, há quem escreva o software, há quem dê suporte ao usuário, há quem desenhe a interface, há até quem pense a experiência do usuário, desde o momento de tirar o produto da caixa, até o devolve-lo para reciclagem. Há computeiros que passam meses sem escrever dez linhas sequer de programa, e outras que produzem milhares por mês. Alguns passam a vida lidando com pessoas, mediando, negociando ; outras se escondem atrás do teclado e, de interação humana, já estão felizes com “bom dia”. Há espaço para todos os perfis, e para todas as vocações.

A complexidade é também uma fraqueza da Informática. É uma profissão mal compreendida pelo público, cheia de mitos, de estereótipos. Eu gostaria de desfazer alguns deles.

O primeiro mito é que a Computação é burocrática, sem espaço para a criatividade. Muito ao contrário, a Informática é criativa, é expressiva. Um programa de computador é, quase literalmente, um pedaço de pensamento fixado em um suporte, que o programador molda, dá um sopro da vida, e ele cria uma existência própria, independente. Se isso não é criação, não é expressão, então o que é ? — E eu nem estou mencionando as áreas mais convencionalmente artísticas da profissão, como o design de interface, de interação, e de produto.

O segundo mito é que a Computação é reservada aos nerds, ou então aos prodígios. Olha, nada contra os nerds — afinal eu sou um deles — mas nós não temos nenhum monopólio da Informática. Além disso, me deixa tristíssimo, para não dizer furioso com nosso sistema de ensino, quando uma criança de 10 anos que está com dificuldade na tabuada do sete decide (ou então é decidido por ela) que [1] ela não é boa em matemática ; [2] ela não nasceu com o dom da matemática ; [3] ela não deve seguir uma profissão de exatas. Vamos abandonar essas bobagens ! Matemática é como tudo na vida, é como pular corda, é como dançar tango : se aprende com esforço, se aprende com persistência — não é questão de genética, é questão de labuta. E a verdadeira Matemática, criativa, expressiva, útil — a Matemática que muda o mundo — não tem nada a ver com ser muito bom na tabuada do sete.

O terceiro mito é que o impacto da Computação é técnico, distante do humano. Dado tudo o que eu falei, era para esse já estar desfeito, mas ele é muito insidioso, então eu vou deixar explícito. Me entristece ver jovens abandonarem a vocação para a Computação, para a Engenharia, e procurarem outras profissões — normalmente o Direito e a Medicina — porque querem “ajudar as pessoas”. Nada contra Direito e Medicina, profissões belíssimas, mas essa é uma péssima razão para abandonar a Informática. Basta pensar um mínimo além do imediato para perceber o enorme impacto — social, cultural, ecológico, econômico : podem escolher — que um profissional da informática engajado tem para um grande número de pessoas. Que impacto traz criar uma enciclopédia abrangente, de alta qualidade, gratuita ? Um canal de expressão onde qualquer cidadão pode postar o seu vídeo e ser assistido no mundo inteiro, sem passar pelo crivo ou censura de governos, de emissoras ? Uma rede social de financiamento onde qualquer projeto pode ganhar vida, sem ter que agradar um comitê conservador de investidores ? Pensem na Wikipedia, no YouTube, no Kickstarter a próxima vez em que ficarem tentados a pensar que a informática não ajuda as pessoas.

E finalmente, tenho que atacar o velho mito, que a gente ainda ouve, de que a Informática é uma profissão masculina, uma profissão reservada aos meninos. Quantas meninas, jovens, mulheres talentosas não deixam de vir para a Computação porque acreditaram nessa bobagem ? O resultado é termos um curso ainda muito mais procurado pelos meninos do que pelas meninas, mas felizmente isso já está mudando. Nós estamos engajados em tornar o curso cada vez mais diverso, e acolhedor para todos os gêneros. Nesse sentido é que eu rogo, aliás imploro, aos pais que manifestem entusiasmo pelas filhas que se interessarem pela Informática, o mesmo entusiasmo que normalmente manifestam quando o interesse vem dos filhos homens. E deixo aqui minha mensagem para as irmãs, para as primas, para as sobrinhas que ainda não ingressaram na Universidade. A Computação é para vocês sim, e nós estamos contando com vocês, esperando por vocês quando chegar a hora de vocês virem para a UNICAMP !

Senhoras, senhores, eu já me alonguei certamente mais do que o bom tom permitia. Agradeço imensamente a oportunidade de participar desse momento maravilhoso, e parabenizando novamente nossos formandos encerro aqui a minha fala.

Deep networks can be seen as hierarchical generalized linear models.

My postgraduate offer for 2016/1 : Deep Learning From a Statistician’s Viewpoint

With few exceptions, my postgraduate offers follow a pattern. On the second semester, I offer my “101” Multimedia Information Retrieval course, which introduces multimedia representations, machine learning, computer vision, and… information retrieval. On the first semester, I offer a topics course, usually following a book : so far we have explored Bishop’s PRML, Hofstadter’s GEB, and Jaynes’ “Probability Theory”.

For 2016/1, I’m risking something different :

“Artificial Intelligence is trending again, and much of the buzz is due to Deep Neural Networks. For long considered untrainable, Deep Networks were boosted by a leap in computing power, and in data availability.

Deep Networks stunned the world by classifying images into thousands of categories with accuracy, by writing fake wikipedia articles with panache, and by playing difficult videogames with competence.

My aim here is a less “neural” path to deep models. Let us take the biological metaphors with a healthy dose of cynicism and seek explanations instead in statistics, in information theory, in probability theory. Remember linear regression ? Deep models are multi-layered generalized linear models whose parameters are learned by maximum likelihood. Let us start from there and then explore the most promising avenues leading to the current state of the art.

This course will be nothing like your typical classroom experience. There will be no lectures. We will meet once a week for a presencial session to discuss previous work, and plan our attack for the next week. I’ll expect you to continue working throughout the week. There will be no exams. I’ll grade your work based on participation during the sessions, progress between sessions, self assessment, and peer assessment.

Active participation will be mandatory. This means (surprise !) talking in public. Everyone will be learning together, so all of us must accept the risk to be wrong. This course won’t work for those who always want to appear wise and knowledgeable. The course will be in English.

Deep networks can be seen as hierarchical generalized linear models.

Deep networks can be seen as hierarchical generalized linear models.

We’ll be a cozy small group : at most 12 students. I’ll select the candidates based on a letter of intentions, and on previous experience. Write a short e-mail to dovalle@dca.fee.unicamp.br. No need to be fancy : just state your reasons for participating, and any previous experience (academic, professional, and extra-curricular) with Machine Learning, Statistics, Probability, or Information Theory.

This course is not for beginners, nor for the faint of heart. We are jumping in head first at the deep (tee hee !) end. After all, we will delve into one of the most engaging intellectual frontier of our time. I dare you to join us !”

Very important ! If you want to enroll at this course without being enrolled at the program (what UNICAMP awfully calls “special students”), you have to do you pre-enrollment until 7/Dec/2015 (hard deadline !). Even if you are enrolled at the program (“regular student”) send me your application at most until 31/Dec/2015, because I’ll select regular and special (urgh !) students at the same time.

EDIT 20/01 : I have sent the acceptance notices — looking forward to work with a swell group of very motivated students !

What : Post-graduate course for the Master or Doctorate in Electrical Engineering program of UNICAMP (4 credits)

When : 2016/1st semester — mandatory presencial meetings Tuesdays from 19 to 21h ; support meetings same day from 16 to 18h

Image credit : composite from Ramón y Cajal 1st publication showing a cerebellum cut, and scatterplots from Fisher’s iris dataset drawn by Indon~commonswiki, wikimediacommons.

Linux brew won't turn your dependency hell into a package management heaven, but it might turn it into a tolerable purgatory.

Installing software on Linux without root : managing packages in user space

First step : don’t. Reconsider. There really isn’t an alternative ? Using something else that is already installed ? Sweet-talking your sysadmin into doing the installation ? Giving up that particular task ? Giving up Computer Sciences altogether and moving to the country to raise pigs ?

Ok, so you really don’t have an alternative. May the gods have mercy on your soul, because Linux won’t. By necessity, this won’t be a step-by-step guide, because each system has its quirks. I’m not promising you heaven, just a tolerable purgatory instead of a living hell.

Take a deep breath.

(And as always : follow those instructions at your own risk. If you turn your Linux box into a Linux brick, or into Skynet, I’m not liable.)

The problem : dependency hell

There’s a reason why installing software from sources is so painful : dependencies. Sure, you just want to install Caffe, or Torch7, or Theano. But Theano needs python, python needs openssl, openssl needs… it’s endless.

High-level package managers like apt-get and yum are so popular because they deal with those. When installing from source, you’re on your own.

But here’s the catch : when installing from sources, you can almost always relocate the software to your ~home, bypassing the need for root access. High-level package managers, at least the current generation, can’t relocate.

Except for Homebrew.

The strategy : Linuxbrew

Homebrew was created as “the missing package manager for OS X”, and is required to do anything interesting on a Mac. It was designed around two principles : installation at the user home, and installation from sources.

Say that again ? Installation at the user home, without need for root. From sources. Wow. If only there was a version for Linux ! Enter Linuxbrew. Homebrew concept was so successful that, in an ironic turn, it’s now becoming “the missing package manager for Linux”.

So, case closed ? Hardly. To start, Linuxbrew has dependencies of its own, and you have to take care of those by hand. Then, the software you want to install has to be available as a brew “formula” (but the list is quite comprehensive, and growing). Finally, it doesn’t always goes smoothly. Linuxbrew is a much bumpier ride than Homebrew/OS X, at least for now. Most formulas will install without issue, but a good 20% will require tweaking, googling, and deciphering forum posts.

The strategy is most definitely not user-friendly. But contrarily to installing each and every package by hand it is just user-bearable enough to be feasible. If you really need the software. (Are you sure you don’t prefer the pig farm ?)

Okay, you are sure.

Our strategy will be :

  1. Ensuring Linuxbrew dependencies ;
  2. Installing and configuring Linuxbrew ;
  3. Using Linuxbrew to install the desired software… ;
  4. …or if you’re unlucky, using Linuxbrew to install the desired software dependencies, and then installing the desired software by hand.

The tatics

Installing Linuxbrew dependencies

Linuxbrew has, fortunately, few dependencies : Ruby, GCC, Git, and… Linux (duh !). It runs on x86 or 32-bit ARM platforms. If you’re running Linux in other architectures this is your cue to break down sobbing.

Most systems will, fortunately have those dependencies already installed. You can check the minimal versions currently required by Linuxbrew, and then check the versions installed at your system (and whether they are installed at all) calling the commands with --version :

$ ruby --version
ruby 1.8.7 (2011-12-28 patchlevel 357) [x86_64-linux]

$ gcc --version
gcc (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973]
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ git --version
If 'git' is not a typo you can run the following command to lookup the package that contains the binary:
command-not-found git
-bash: git: command not found

Linuxbrew won’t turn your dependency hell into a package management heaven, but it might turn it into a tolerable purgatory.

Linuxbrew won’t turn your dependency hell into a package management heaven, but it might turn it into a tolerable purgatory.

As you see, I got almost lucky. Ruby and GCC are good to go, but I’ll have to install Git (and its dependencies).

The trick is being minimalist. Many packages have optional features : extra commands, documentation, GUI interfaces, etc., that will fire zillions of extra dependencies. Get rid of those optionals as much as possible.

I recommend a top-down plan of attack. Start with the package you want to install and try to configure–make–install it. If it breaks (and it will, it will), find out which dependency/configuration is missing and correct it. Do this recursively, depth-first, until you get everything installed.

Resist perfectionism. You might spend a lot of time smoothing out every wrinkle of package-1.73.03 just to find a bit later that it breaks your installation and has to be removed to make room for package-1.73.04. This is war, kid : not a time for bells and whistles. Once you get a dependency working, move on to the next.

In more detail, each cycle will consist of :

  1.  Finding, downloading, and unpacking the source package ;
  2.  Configuring the package to work with a $HOME prefix ;
  3. Building the package ;
  4. Installing the package.

Step 1 is usually trivial after bit of googling. If your Linux distribution is Debian-based, you might be able to use a single command-line operation :

apt-get source git

There are similar facilities for other high-level package managers.

Otherwise, you might download either a compressed source file, or even the bleeding edge version from the source repository (sourceforge, github, etc.) In the case of Git, this would be at https://github.com/git/git. (Be careful, because those hyper-up-to-date versions might be unstable.)

Step 2 varies a bit from package to package, but usually consists in calling a ./configure script. Sometimes pre-configuration is involved : a call to make configure or make config, or another script, e.g., ./buildconf. Sometimes it involves cmake (cross your fingers for having autoconf/automake already installed). Sometimes there’s no step 2, all options being passed directly to make during step 3. It varies.

How will you know ? Try to locate a INSTALL.* or README.* file. Usually the instructions are there. No luck ? Try browsing the official website of the package for installations instructions. Googling <package> installation instructions usually will point you to the right direction.

For git, this will work :

cd git-2.1.4/
./configure --prefix=$HOME

Well, sort of. It will probably break, because one or more dependencies will be missing. Install those (and their recursive dependencies) and try again.

Step 3 is almost always :

make

or sometimes :

make all

Sometimes this is the moment when things break down for lack of dependencies (or wrong versions, or wrong settings, or the universe showing its lack of care). Sometimes the --prefix=$HOME option comes here instead of Step 2.

Step 4 is almost always :

make install

If you set the prefixes right, that will automagically put everything in the right place, under your ~home directory. And you won’t need root permissions.

Got it ? Good. I hope you enjoy typing command-line commands : you’ll be doing it all day. For extra enjoyment, get a second monitor and close the shades.

Installing Linuxbrew

Once you have all dependencies working, installing Linuxbrew itself is a breeze :

git clone https://github.com/Homebrew/linuxbrew.git ~/.linuxbrew

Aaand… that’s it. It won’t work immediately because you have to set the paths (see below). After you do it you can simply type :

brew install $WHATEVER_YOU_WANT

And it should take care of everything.

Before you do it, however it is a good idea to call

brew doctor

and check if everything is ok. Again, be minimalist : you don’t have to correct every tiny issue. Take a good look and make the smallest needed intervention.

Linuxbrew comes ready with a lot of recipes for installing packages, or as they call, formulas. You can keep them up do date by typing

brew update

Depending on what you want to install, however, you’ll need extra formulas. In Homebrew/Linuxbrew parlance this is called tapping. For example :

brew tap homebrew/science

will install a lot of new formulas related to science, data analysis, etc.

PATH configurations

Both phases (manual dependency installations; Linuxbrew operation) won’t do you much good if your paths aren’t configured. There are at least four important paths, maybe more depending on your setup : executables (PATH), static libraries (LIBRARY_PATH), dynamic libraries (LD_LIBRARY_PATH), and include files (CPATH).

The usual place to set up those is your your shell configuration file. The examples below assume you’re using bash. If that’s your case, decide whether .bashrc or .bash_profile is better for you (usually it’s the former).

During the manual installation of dependencies add the following lines :

# Manually installed packages
export PATH="$HOME/bin:$PATH"
export LIBRARY_PATH="$HOME/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="$HOME/lib:$LD_LIBRARY_PATH"
export CPATH="$HOME/include:$CPATH"

During Linuxbrew operation put those additional lines :

# HomeBrew / LinuxBrew
export HOMEBREW_PREFIX="$HOME/.linuxbrew"
export PATH="$HOMEBREW_PREFIX/bin:$PATH"
export LIBRARY_PATH="$HOMEBREW_PREFIX/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="$HOMEBREW_PREFIX/lib:$LD_LIBRARY_PATH"
export CPATH="$HOMEBREW_PREFIX/include:$CPATH"

Remember that shell configurations are not effective immediately, only on the next start. You don’t have to reboot the system : simple closing and reopening the terminal, or logging out and back in suffices.

An ugly ugly ugly workaround

During my installations, I faced an issue with CA certificates that I could not bypass. Many formulas would refuse to proceed, stopping during download with the error : “cURL error 60: SSL certificate problem: unable to get local issuer certificate”.

Yes : I tried downloading updated certificates from Mozilla Corp. Yes : I checked my curl-config --ca. Yes : I tried reinstalling cURL. And Git. And OpenSSL. I spent, litteraly, hours trying to solve the problem in an elegant way.

I concede defeat. Here’s the very inelegant solution. Be aware that it opens your package manager to man-in-the-middle attacks. That is more than a theoretical risk : it has been done. This is a huge security hole. If you decide to apply it, don’t do it preemptively, wait to see if you’ll actually get the SSL certificate problem.

So you got the error, and you’re willing to expose your neck to the wolves ? Sucks to be you. Open the file download_strategy.rb at ~/.linuxbrew/Library/Homebrew and find the lines below

# Curl options to be always passed to curl,
# with raw head calls (`curl -I`) or with actual `fetch`.
def _curl_opts
  copts = []
  copts << "--user" << meta.fetch(:user) if meta.key?(:user)
  copts
end

Change line four to

# Curl options to be always passed to curl,
# with raw head calls (`curl -I`) or with actual `fetch`.
def _curl_opts
  copts = ["-k"] # Disable certificate verification
  copts << "--user" << meta.fetch(:user) if meta.key?(:user)
  copts
end

And that’s it. You’re ready to proceed installing source packages. And to be a victim of cyber mafias, and of terrorists, and of tyrannical governments.

(Note to security people : if your watertight security solution makes a system unusable, guess what will happen ?)

Extra tips

First and foremost, source code relocation is not a panacea. Some things require root access, for example, driver installations, kernel recompilations, boot sector modifications, etc. You might want to check if your software require one of those before you start this whole adventure.

You can learn a lot about a formula with the info option

$ brew info python
python: stable 2.7.10 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org
Not installed
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python.rb
==> Dependencies
Build: pkg-config ✘
Required: openssl ✘
Recommended: readline ✘, sqlite ✘, gdbm ✘
Optional: homebrew/dupes/tcl-tk ✘, berkeley-db4 ✘
==> Options
--universal
  Build a universal binary
--with-berkeley-db4
  Build with berkeley-db4 support
--with-poll
  Enable select.poll, which is not fully implemented on OS X (https://bugs.python.org/issue5154)
--with-quicktest
  Run `make quicktest` after the build (for devs; may fail)
--with-tcl-tk
  Use Homebrew's Tk instead of OS X Tk (has optional Cocoa and threads support)
--without-gdbm
  Build without gdbm support
--without-readline
  Build without readline support
--without-sqlite
  Build without sqlite support
--HEAD
  Install HEAD version

Take a good look at the --without-* options because they are sometimes a lifesaver. Some packages have optional GUI extras. They might fire hundreds of useless extra dependencies — especially if you are installing on a headless server.

Sometimes Linuxbrew breaks down for the lack of a dependency, refusing to install it, but will gladly do it if you explicitly ask for it. For example : brew install package1 breaks for the lack of package2, and all it takes is typing brew install package2 and retrying package1. Mysteries.

Installation time is highly unpredictable. Sometimes a small innocent little package will require a precise version of GCC… that Linuxbrew will then have to install from the sources. Time for a tea.

If your installation becomes so corrupted with conflicting packages that you have to restart from scratch (nooooooo !), it can be — small consolation — accomplished easily :

rem ~/.linuxbrew -rf

For extra OCD cred, clean-up the download cache as well :

rm ~/.cache/Homebrew/ -rf

If the whole thing becomes so messed up that you have to scratch even the manual dependencies (two words : pig farm), it is also easily done :

rm ~/bin/ ~/lib/ -rf

You might also consider :

rm ~/include ~/etc -rf

but be careful because that might erase innocent third parties.

You might be forced to install multiple versions of the same package. That adds another nightmare layer to the ongoing nightmare, but it’s doable. Linuxbrew will usually be friendly enough to tell you what to do.

For example, when I had to install both opencv2 and opencv3 I got this :

opencv3 and opencv install many of the same files.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/home/valle/.linuxbrew/opt/opencv3/lib
    CPPFLAGS: -I/home/valle/.linuxbrew/opt/opencv3/include

Those little displays of care are the reason why I like Homebrew/Linuxbrew so much. Love breeds love : a truism even for human–computer interaction.

web-server-rapberry-pi

A lightweight web server with Lighttpd, PHP and SQLite3 on a Raspberry Pi

Many consumer devices (set top boxes, routers, media centers) use lightweight embedded web servers as user interface. In this walkthrough, I’ll show you how to install a complete web server in a Raspberry Pi, which is a cheap ARM-based computer that is perfect for prototyping such appliances.

In embedded systems, we aim at minimizing computational expenditure. Therefore, we’ll use Lighttpd as the web server, and SQLite as the DBMS. For server-side scripting, we’ll use PHP, a nice framework for protyping : easy to learn, and very popular.

Installing a fresh copy of Raspbian in an SD card

I’ll start from scratch, with a fresh Raspbian system. This is not strictly necessary : you can probably succeed from any working Raspbian installation.

You can start by downloading the Raspbian image, and installing it into the micro SD card. There are different instructions for Linux, for Windows, and for OS X.

For OS X, there are two sets of instructions. In my setup (MacBook Pro 13″, early 2011, Yosemite), only the second set worked, and I’ll explain it here. (If it doesn’t work for you, try the other one.)

Attention : we will be doing low-level disk transfers here. One single distraction and you may lose all your data. Your motivational PowerPoints. Those pictures of the 70th birthday of Auntie Joaquina. That spreadsheet you were almost finishing with all continuity mistakes in Bervely Hills, 90210. Gone ! Forever ! (No, seriously : backup your stuff. Really !)

(And my lawyer insists that I remind you : as always, proceed at your own risk. If you follow those instructions and your computer turns to a brick, or your cat divorces you, I’m not liable.)

First let’s unzip the disk image. Open a terminal in the folder where the image file is located, and type :

unzip 2015-05-05-raspbian-wheezy.zip

Now, let’s find out the device path to the SD card. First ensure that the SD card slot is empty, and type df -h in the shell. You should get something like :

$ df -h
Filesystem                          Size   Used  Avail Capacity   iused     ifree %iused  Mounted on
/dev/disk1                         930Gi  486Gi  444Gi    53% 127405636 116484474   52%   /
devfs                              183Ki  183Ki    0Bi   100%       636         0  100%   /dev
map -hosts                           0Bi    0Bi    0Bi   100%         0         0  100%   /net
map auto_home                        0Bi    0Bi    0Bi   100%         0         0  100%   /home
localhost:/p9Q694fzz20lCp5sv3CZyj  930Gi  930Gi    0Bi   100%         0         0  100%   /Volumes/MobileBackups

Now, insert the micro SD card in the slot (you might need an adapter), count five seconds, and type df-h again. You should get something like :

$ df -h
Filesystem                          Size   Used  Avail Capacity   iused     ifree %iused  Mounted on
/dev/disk1                         930Gi  486Gi  444Gi    53% 127405653 116484457   52%   /
devfs                              185Ki  185Ki    0Bi   100%       642         0  100%   /dev
map -hosts                           0Bi    0Bi    0Bi   100%         0         0  100%   /net
map auto_home                        0Bi    0Bi    0Bi   100%         0         0  100%   /home
localhost:/p9Q694fzz20lCp5sv3CZyj  930Gi  930Gi    0Bi   100%         0         0  100%   /Volumes/MobileBackups
/dev/disk2s1                        56Mi   20Mi   36Mi    36%       512         0  100%   /Volumes/boot

Got it ? There’s a new device /dev/disk2s1 for the SD card. You might get several new lines if the card currently has more than one partition (e.g. /dev/disk2s1, /dev/disk2s2, etc.)

We need to unmount all those partitions, but without ejecting the device (if you eject it, you’ll have to restart the whole process) :

sudo diskutil unmount /dev/disk2s1

Repeat that command for each partition in the device.

Now for the dangerous command. We will use the low-level copy command dd, using the disk image as input, and the SD card device as output. Take a deep breath and check each character thrice before hitting Enter, because overwriting the wrong device will be a nightmare.

Check out the partition names above. If you got /dev/disk2s1, you’ll write to /dev/rdisk2. If you got /dev/disk3s2, you’ll write to /dev/rdisk3, etc. Got it ? Just throw away the s<number> suffix and add an r to the beginning :

sudo dd bs=1m if=2015-05-05-raspbian-wheezy.img of=<device path to write>

This might take a while — in my system, 5 minutes or so. You may type ctrl+T to send the SIGINFO signal, and get an status update. Also, you might need to write bs=1m as bs=1M, depending on the version of dd.

This is it ! Eject the the card. Time to move to the Pi.

Configuring Raspbian

Insert the newly formatted SD card into the Pi, plug the needed devices. We will need at least : an ethernet connection to the Internet, a keyboard, and a monitor. Plug the power last.

With some luck, the Raspberry Pi Software Configuration Tool will appear during the first boot. (If it doesn’t appear, or if you are not using a fresh system, you can launch it by typing sudo raspi-config in the command shell.)

Be sure to at least activate option 1 (“Expand Filesystem”) so Raspbian will use the entire SD card. Otherwise the upgrade commands below risk running out of space. I will also use option 2 (“Change User Password”) to make access to the system secure ; option 3 (“Enable Boot to Desktop/Scratch“) to ensure that the system boots to Console Text (no need for graphical desktop in this small web server) ; option 4 (“Internationalization Options“) to configure my keyboard to US International; and option 8 (“Advanced Options“) to set the hostname, and to ensure that SSH access is enabled.

Updating and upgrading the system

Once the system reboots, log in (the default user is pi, and the defaul password, if you didn’t change it with option 2 above, is raspberry).

We’ll start by bringing the system up to speed :

sudo apt-get -y update
sudo apt-get -y dist-upgrade

Installing Lighttpd, SQLite, and PHP

Time to install the components of the web server. The instructions are adapted from smching’s Instructable.

sudo apt-get -y install lighttpd
sudo apt-get -y install sqlite3
sudo apt-get -y install php5 php5-common php5-cgi php5-sqlite 

Now for a bit of configuration. We’ll want to enable fastcgi to handle the PHP pages :

sudo lighty-enable-mod fastcgi
sudo lighty-enable-mod fastcgi-php
sudo service lighttpd force-reload # restart the Lighttpd service

Setting permissions

Finally, we need to set up the permissions of /var/www (the root of the web server), and include the default user on the group that can read/write that directory.

sudo chown -R www-data:www-data /var/www
sudo chmod -R 775 /var/www
sudo usermod -a -G www-data pi

Time for a reboot !

sudo reboot

Testing

If you have a second computer attached to your network, you can use it to access the recently installed web server. Before leaving the Pi, type ifconfig to get the IP. You should get something like…

$ ifconfig
eth0      Link encap:Ethernet  HWaddr b8:27:eb:7e:b2:3c  
          inet addr:10.0.1.7  Bcast:10.0.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:218 errors:0 dropped:0 overruns:0 frame:0
          TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:47677 (46.5 KiB)  TX bytes:18722 (18.2 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1104 (1.0 KiB)  TX bytes:1104 (1.0 KiB)

…showing that the IP attributed to the Pi on ethernet is 10.0.1.7.

You can type http://10.0.1.7/ on the browser of another computer to check the web server. You should see the placeholder page for Lighttpd showing that the web server is correctly installed.

Also — if you haven’t disabled SSH — you can type ssh pi@10.0.1.7 to access the Pi remotely. This is useful if you have a single set of monitor/keyboard/mouse.

If the Pi is your only computer, you can start the graphical desktop with the command startx, and then navigate to the newly installed web server. Typing the IP address of the machine, as before, will work, but so does using the loopback interface : http://127.0.0.1/ or http://localhost/.

Next, let’s test if PHP is working. Paste this simple test page…

<?PHP
phpinfo()
?>

…into /var/www/test.php.

Then, type in your browser http://10.0.1.7/test.php (change 10.0.1.7 for the actual IP of your Pi, or use http://localhost/test.php if browsing from inside the Pi). You should see the PHP info page.

Finally, let's run a small test on PHP + SQLite. I adapted this code from Veit Osiander's post on Scandio. Paste the following...

<?php
try {
    // Create file "scandio_test.db" as database
    $db = new PDO('sqlite:scandio_test.db');
    // Throw exceptions on error
    $db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
 
    $sql = <<<SQL
CREATE TABLE IF NOT EXISTS posts (
    id INTEGER PRIMARY KEY,
    message TEXT,
    created_at INTEGER
)
SQL;
    $db->exec($sql);
 
    $data = array(
        'Test '.rand(0, 10),
        'Data: '.uniqid(),
        'Date: '.date('d.m.Y H:i:s')
    );
 
    $sql = <<<SQL
INSERT INTO posts (message, created_at)
VALUES (:message, :created_at)
SQL;
 
    $stmt = $db->prepare($sql);
    foreach ($data as $message) {
        $stmt->bindParam(':message', $message, SQLITE3_TEXT);
        $stmt->bindParam(':created_at', time());
 
        $stmt->execute();
    }
 
    $result = $db->query('SELECT * FROM posts');
 
    foreach($result as $row) {
        list($id, $message, $createdAt) = $row;
        $output  = "Id: $id<br/>\n";
        $output .= "Message: $message<br/>\n";
        $output .= "Created at: ".date('d.m.Y H:i:s', $createdAt)."<br/>\n";
 
        echo $output;
    }
 
    $db->exec("DROP TABLE posts");
} catch(PDOException $e) {
    echo $e->getMessage();
    echo $e->getTraceAsString();
}
?>

...into /var/www/testdb.php.

Navigate to http://10.0.1.7/testdb.php (or http://localhost/testdb.php), and if everything goes well, you should get something like :

Id: 1
Message: Test 1
Created at: 30.09.2015 05:34:33
Id: 2
Message: Data: 560b746950a70
Created at: 30.09.2015 05:34:33
Id: 3
Message: Date: 30.09.2015 05:34:33
Created at: 30.09.2015 05:34:33
The raspberry pi is perfect to prototype a small appliance with a web server as interface.

The raspberry pi is perfect to prototype a small appliance with a web server as interface.

If you get an ugly error, like "General error: 14 unable to open database file", double check the permissions of /var/www (and everything inside it). The user www-data must have reading and writing permission on everything (see the section "Setting permissions" above).

Otherwise, you are ready to start working on your project !

The devil in the details : feeding images to Theano

Another lesson from the Kaggle competition : pushing your code to ultimate speed is still labour intensive.

High-level libraries like Numpy and Theano really help, but when you are training complex deep-learning models, rushing through the epochs takes more than typing a couple of imports, enabling the GPU, and hoping for the best.

Seemingly trivial tasks — input, output, copying buffers, rearranging arrays — may easily become the critical bottleneck. Those treacherous operations always raise flags for practioners of high-performance computing. The rest of us, as we fixate on numerical computations, tend to overlook that moving data around easily dominates the costs.

Sometimes the bottleneck is a bolt out of the blue, a straightforward task that ends up being painfully slow. Case in point, the pipeline of importing an image from PIL to Theano through a Numpy array.

Opening an image using PIL is certainly very simple :

>>> from PIL import Image
>>> image = Image.open('color.png', 'r')
>>> image
<PIL.PngImagePlugin.PngImageFile image mode=RGB size=12x8 at 0x1D3E518>

PIL will be so charming as even to deduce by itself the necessary decoder from the file extension or headers.

From the returned object, you can fetch metadata :

>>> image.format
'PNG'
>>> image.getbbox()
(0, 0, 12, 8)
>>> image.getbands()
('R', 'G', 'B')

Or you can get an object that gives access to the actual pixels with :

>>> image.getdata()
<ImagingCore object at 0x7f21e5bb2350>

If you want to use the image as input for a model in Theano, you’ll need first to convert it to a Numpy array. This can very easily be done with :

>>> import numpy
>>> array1 = numpy.asarray(image)
>>> array1 
array([[[255,   0,   0],
        [255,   0,   0],
        [128,   0,   0],
        [128,   0,   0],
        [  0,   0,   0],
        [ 64,  64,  64],
        [128, 128, 128],
        [255, 255, 255],
        [  0,   0, 255],
        [  0,   0, 255],
        [  0,   0, 128],
        [  0,   0, 128]],

       [[255,   0,   0],
        [255,   0,   0],
        [128,   0,   0],
        [128,   0,   0],
        [  0,   0,   0],
        [ 64,  64,  64],
        [128, 128, 128],
        [255, 255, 255],
        [  0,   0, 255],
        [  0,   0, 255],
        [  0,   0, 128],
        [  0,   0, 128]],

       [[255,   0,   0],
        [255,   0,   0],
        [128,   0,   0],
        [128,   0,   0],
        [  0,   0,   0],
        [ 64,  64,  64],
        [128, 128, 128],
        [255, 255, 255],
        [  0,   0, 255],
        [  0,   0, 255],
        [  0,   0, 128],
        [  0,   0, 128]],

       [[255,   0,   0],
        [255,   0,   0],
        [128,   0,   0],
        [128,   0,   0],
        [  0,   0,   0],
        [ 64,  64,  64],
        [128, 128, 128],
        [255, 255, 255],
        [  0,   0, 255],
        [  0,   0, 255],
        [  0,   0, 128],
        [  0,   0, 128]],

       [[255, 255,   0],
        [255, 255,   0],
        [  0, 255, 255],
        [  0, 255, 255],
        [  0, 255,   0],
        [  0, 255,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        [128, 128,   0],
        [128, 128,   0],
        [  0, 128, 128],
        [  0, 128, 128]],

       [[255, 255,   0],
        [255, 255,   0],
        [  0, 255, 255],
        [  0, 255, 255],
        [  0, 255,   0],
        [  0, 255,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        [128, 128,   0],
        [128, 128,   0],
        [  0, 128, 128],
        [  0, 128, 128]],

       [[255,   0, 255],
        [255,   0, 255],
        [  0,   0,   0],
        [  0,   0,   0],
        [  0, 255,   0],
        [  0, 255,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        [128,   0, 128],
        [128,   0, 128],
        [255, 255, 255],
        [255, 255, 255]],

       [[255,   0, 255],
        [255,   0, 255],
        [  0,   0,   0],
        [  0,   0,   0],
        [  0, 255,   0],
        [  0, 255,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        [128,   0, 128],
        [128,   0, 128],
        [255, 255, 255],
        [255, 255, 255]]], dtype=uint8)

You may, at this point, specify a data type. For injecting images into Theano, we’ll often want to convert them to float32 to allow GPU processing :

array1 = numpy.asarray(image, dtype='float32')

Or for enhanced portability, you can use Theano’s default float datatype (which will be float32 if the code is intended to run in the current generation of GPUs) :

import theano
array1 = numpy.asarray(image, dtype=theano.config.floatX)

Alternatively, the array can be converted from the ImagingCore object :

>>> array2 = numpy.asarray(image.getdata())
>>> array2
array([[255,   0,   0],
       [255,   0,   0],
       [128,   0,   0],
       [128,   0,   0],
       [  0,   0,   0],
       [ 64,  64,  64],
       [128, 128, 128],
       [255, 255, 255],
       [  0,   0, 255],
       [  0,   0, 255],
       [  0,   0, 128],
       [  0,   0, 128],
       [255,   0,   0],
       [255,   0,   0],
       [128,   0,   0],
       [128,   0,   0],
       [  0,   0,   0],
       [ 64,  64,  64],
       [128, 128, 128],
       [255, 255, 255],
       [  0,   0, 255],
       [  0,   0, 255],
       [  0,   0, 128],
       [  0,   0, 128],
       [255,   0,   0],
       [255,   0,   0],
       [128,   0,   0],
       [128,   0,   0],
       [  0,   0,   0],
       [ 64,  64,  64],
       [128, 128, 128],
       [255, 255, 255],
       [  0,   0, 255],
       [  0,   0, 255],
       [  0,   0, 128],
       [  0,   0, 128],
       [255,   0,   0],
       [255,   0,   0],
       [128,   0,   0],
       [128,   0,   0],
       [  0,   0,   0],
       [ 64,  64,  64],
       [128, 128, 128],
       [255, 255, 255],
       [  0,   0, 255],
       [  0,   0, 255],
       [  0,   0, 128],
       [  0,   0, 128],
       [255, 255,   0],
       [255, 255,   0],
       [  0, 255, 255],
       [  0, 255, 255],
       [  0, 255,   0],
       [  0, 255,   0],
       [  0, 128,   0],
       [  0, 128,   0],
       [128, 128,   0],
       [128, 128,   0],
       [  0, 128, 128],
       [  0, 128, 128],
       [255, 255,   0],
       [255, 255,   0],
       [  0, 255, 255],
       [  0, 255, 255],
       [  0, 255,   0],
       [  0, 255,   0],
       [  0, 128,   0],
       [  0, 128,   0],
       [128, 128,   0],
       [128, 128,   0],
       [  0, 128, 128],
       [  0, 128, 128],
       [255,   0, 255],
       [255,   0, 255],
       [  0,   0,   0],
       [  0,   0,   0],
       [  0, 255,   0],
       [  0, 255,   0],
       [  0, 128,   0],
       [  0, 128,   0],
       [128,   0, 128],
       [128,   0, 128],
       [255, 255, 255],
       [255, 255, 255],
       [255,   0, 255],
       [255,   0, 255],
       [  0,   0,   0],
       [  0,   0,   0],
       [  0, 255,   0],
       [  0, 255,   0],
       [  0, 128,   0],
       [  0, 128,   0],
       [128,   0, 128],
       [128,   0, 128],
       [255, 255, 255],
       [255, 255, 255]])

Note that the techniques are not exactly interchangeable. Converting directly from the Image object preserves the shape, while converting from .getdata() creates a flatter array :

>>> array1.shape
(8, 12, 3)
>>> array2.shape
(96, 3)

In both cases the image channels are “interleaved”, i.e., for each row and column of the image, the values of the channels (in this example: red, green and blue) appear in sequence. This organization is helpful if you are doing local transformations in the pixels (say, changing colorspaces) as it keeps referential locality. For deep learning, however, that arrangement is a nuisance.

Even with 2D input images, we will typically want “3D” convolutional filters in order to reach through — and mix and match — the entire stack of input channels. We need the input channels separated by planes, i.e., all red data, then all green data, etc. That is a breeze with rollaxis and reshape :

>>> array1 = numpy.rollaxis(array1, 2, 0)
>>> array2 = array2.T.reshape(3,8,12)

Some neural nets go one step further and stack together all the images from a given batch into a single tensor. The LeNet/MNIST sample from deeplearning.net does exactly that. That strategy can increase GPU utilization by giving it bigger chunks to chew at once. If you decide to adopt it, it’s not difficult to assemble the tensor :

ImageSize = (512, 512)
NChannelsPerImage = 3
imagesData = [ Image.open(f, 'r').getdata() for f in batch ]
for i in imagesData :
    assert i.size == ImageSize
    assert i.bands == NChannelsPerImage

allImages = numpy.asarray(imagesData)
nImages = len(batch)
allImages = numpy.rollaxis(allImages, 2, 1).reshape(nImages, NChannelsPerImage, ImageSize[0], ImageSize[1])
print allImages.shape

The code above checks that all images conform to a given shape (essential to convolutional networks, which are very rigid about input sizes). And it works… but it will rather walk than run.

As you try to discover what is holding back the speed, it is easy to suspect the array reshaping operations, but the real culprit is the innocent-loking image conversion to array. For some reason importing images to Numpy either from Image objects or from ImagingCore objects — as we have been trying so far — takes an absurd amount of time.

The solution is not exactly elegant but it makes the conversion so much faster, you might want to consider it. You have to bridge the conversion with a pair of .tostring / .fromstring operations :

ImageSize = (512, 512)
NChannelsPerImage = 3
images = [ Image.open(f, 'r') for f in batch ]
for i in images :
    assert i.size == ImageSize
    assert len(i.getbands()) == NChannelsPerImage

ImageShape =  (1,) + ImageSize + (NChannelsPerImage,)
allImages = [ numpy.fromstring(i.tostring(), dtype='uint8', count=-1, sep='') for i in images ]
allImages = [ numpy.rollaxis(a.reshape(ImageShape), 3, 1) for a in allImages ]
allImages = numpy.concatenate(allImages)

The snippet above has exactly the same effect than the previous one, but it will run up to 20 times faster. In both cases, the array will be ready to be fed to the network.

* * *

TL;DR ?

If speed is important to you, do not convert an image from PIL to Numpy like this…

from PIL import Image
import numpy
image = Image.open('color.png', 'r')
array = numpy.asarray(image)

…nor like this…

from PIL import Image
import numpy
imageData = Image.open('color.png', 'r').getdata()
imageArray = numpy.asarray(imageData).reshape(imageData.shape + (imageData.bands,))

…because although both methods work, they will be very very slow. Do it like this :

from PIL import Image
import numpy
image = Image.open('color.png', 'r').getdata()
imageArray = numpy.fromstring(image.tostring(), dtype='uint8', count=-1, sep='').reshape(image.shape + (len(image.getbands()),))

It will be up to 20⨉ faster.

(Also, you’ll probably have to work on the shape of the array before you feed it to a convolutional network, but for that, I’m afraid, you’ll have to read the piece from the top.)

Deep Neural Network

From instance launch to model accuracy: an AWS/Theano walkthrough

My team has recently participated at Kaggle’s Diabetic Retinopathy challenge, and we won… experience. It was our first Kaggle challenge and we found ourselves unprepared for the workload.

But it was fun — and it was the opportunity to learn new skills, and to sharpen old ones. As the deadline approached, I used Amazon Web Services a lot, and got more familiar with it. Although we have our GPU infrastructure at RECOD, the extra boost provided by AWS allowed exploring extra possibilities.

But it was in the weekend just before the challenge deadline that AWS proved invaluable. Our in-house cluster went AWOL. What are the chances of having a power outage bringing down your servers and a pest control blocking your physical access to them in the weekend before a major deadline ? Murphy knows. Well, AWS allowed us to go down fighting, instead of throwing in the towel.

In this post, I’m compiling Markus Beissinger’s how-to and deeplearning.net tutorials into a single hyper-condensed walkthrough to get you as fast as possible from launching an AWS instance until running a simple convolutional deep neural net. If you are anything like me, I know that you are aching to see some code running — but after you scratch that itch, I strongly suggest you to go back to those sources and study them at leisure.

I’ll assume that you already know :

  1. How to create an AWS account ;
  2. How to manage AWS users and permissions ;
  3. How to launch an AWS instance.

Those preparations out of way, let’s get started ?

Step 1: Launch an instance at AWS, picking :

  • AMI (Amazon Machine Image) : Ubuntu Server 14.04 LTS (HVM), SSD Volume Type – 64-bit
  • Instance type : GPU instances / g2.2xlarge

For the other settings, you can use the defaults, but be careful with the security group and access key to not lock yourself out of the instance.

Step 2 : Open a terminal window, and log into your instance. In my Mac I type :

ssh -i private.pem ubuntu@xxxxxxxx.amazonaws.com

Where private.pem is the private key file of the key pair used when creating the instance, and xxxxxxxx.amazonaws.com is the public DNS of the instance. You might get an angry message from SSH, complaining that your .pem file is too open. If that happens, change its permissions with :

chmod go-rxw private.pem

Step 3 : Install Theano.

Once you’re inside the machine, this is not complicated. Start by making the machine up-to date :

sudo apt-get update
sudo apt-get -y dist-upgrade

Install Theano’s dependencies :

sudo apt-get install -y gcc g++ gfortran build-essential git wget linux-image-generic libopenblas-dev python-dev python-pip python-nose python-numpy python-scipy

Get the package for CUDA and install it :

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.0-28_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1404_7.0-28_amd64.deb
sudo apt-get update
sudo apt-get install -y cuda

This last command is the only one that takes some time — you might want to go brew a cuppa while you wait. Once it is over, put CUDA on the path and reboot the machine :

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> .bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> .bashrc
sudo reboot

Log into the instance again and query for the GPU :

nvidia-smi -q

This should spit a lengthy list of details about the installed GPU.

Now you just have to install Theano. The one-liner below installs the latest version, and after the wait for the CUDA driver, runs anticlimactically fast :

sudo pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git

And that’s it ! You have Theano on your system.

Step 4 : Run an example.
Let’s take Theano for a run. The simplest sample from deeplearning.net that’s already interesting is the convolutional/MNIST digits tutorial. The sample depends on code written in the previous tutorials, MLP and Logistic Regression, so you have to download those too. You also have to download the data. The commands below do all that:

mkdir theano
mkdir theano/data
mkdir theano/lenet
cd theano/data
wget http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz
cd ../lenet
wget http://deeplearning.net/tutorial/code/logistic_sgd.py
wget http://deeplearning.net/tutorial/code/mlp.py
wget http://deeplearning.net/tutorial/code/convolutional_mlp.py

Finally, hit the magical command :

python convolutional_mlp.py

What ? All this work and the epochs will go by as fast as molasses on a winter’s day. What gives ?

You have to tell Theano to run on the GPU, otherwise it will crawl on the CPU. You can paste the lines below into your ~/.theanorc file :

[global]
floatX=float32
device=gpu

[lib]
cnmem=0.9

[nvcc]
fastmath=True

…or you can use the one-liner below to create it:

echo -e '[global]\nfloatX=float32\ndevice=gpu\n\n[lib]\ncnmem=0.9\n\n[nvcc]\nfastmath=True' > ~/.theanorc

Try running the example again.

python convolutional_mlp.py

With some luck, you’ll note two differences: first, Theano will announce the use of the GPU…

Using gpu device 0: GRID K520 (CNMeM is enabled)

…and second, the epochs will run much, much faster !

(Image credit : networkworld.com).

Keeping your iPhone alive in France — Part III

I’m back in France, this time for a leisure trip (my first real vacation in Europe since I graduated here as a Ph.D.) Yes, it’s Winter, but promenading through the streets of Paris without having to worry about the next meeting / deadline / academic obligation is a nice change of pace, cold and rain notwithstanding.

I’m skeptical about the concept of unconnected vacations — in my leisure time I still want to have access to the hive mind, lest my IQ drop a full 30 points. But having a decent data plan in France without breaking the bank is not necessarily obvious, as I’ve been exploring for some time.

mobicarte-holidayOrange Telecom, the main cell-phone company of France, has finally waken-up to the reality that 80M tourists come to France every year — 20% more people than the country own population of 66M. They now propose the Mobicarte Holiday, a pre-paid SIM card loaded with 2h of calls, 1000 SMS, and 1GB of cell data, and unlimited access to Orange Wifi Hotspots — at a price of 40€. If you already have a working Mobicarte you can buy a “Holiday recharge” for 30€. After you use all your credit (or after your credit expires) you can reload the Mobicarte with either the “Holiday” or a normal recharge.

That package is not exactly the cheapest, but it is the most convenient I’ve experienced so far : an offer completely adapted to the needs of the traveller.

Well, almost…

The first limitation is that you’ll probably won’t be able to buy the Mobicarte Holiday from Orange Online store — unless you have a French credit card. This prevents having it delivered directly to your hotel. To compensate that inconvenience, several Orange physical stores are open from Monday to Saturday, until 19h30. I had no difficulty in buying it at the physical store at boulevard Haussmann on a Saturday afternoon.

The second limitation is much more irritating : you won’t be able to connect to the Orange Wifi Hotspots — unless you have a French credit card ! In order to get access to the Wifi hotspots, Orange forces you to install an iOS app — “Mon Réseau” (My Network) — but that app is only available at the French App Store ! Here the synergy of Orange’s nearsightedness and Apple’s greediness creates the perfect storm, as you won’t be able to create an Apple ID for the French Apple store unless you enter a credit card valid in France. (My love–hate relationship with Apple has such a healthy dose of hate because of those things.)

Finally, the kick on the shins : the Holiday credit is valid for meager 14 days, so for longer trips you’ll have to keep buying recharges.

Is there any silver linings ? Well, the SIM card itself will remain valid for 6 months after the last recharge. The price of 20€ a week is still 3 times cheaper than the data roaming offer of my Brazilian operator (Vivo Telecom). You can get a Mobicarte in any one of the mini-, micro-, and nano-SIM formats : you won’t have to deal with SIM clippers (or worse : a sharp kitchen knife and a steady hand). In addition, the 3G Internet offer takes effect immediately (some previous Internet options of Orange took up to 3 days to kick in.)

The Mobicart Holiday is far from perfect, but it’s still the most traveller-friendly offer by Orange France I’ve experienced so far.

Printing Multiple Copies of a Single Page on a Sheet in OS X

This is something that was making me crazy : getting multiple copies of a page into a single sheet in OS X — think of small fliers or business cards. The problem was particularly annoying in Adobe Creative Suite (Illustrator, Photoshop, InDesign), where I hoped (in vain) to find an option to do it easily. The straightforward solution (asking on the system Print dialog for multiple pages per sheet, and then asking for multiple copies) doesn’t really work.

The answer is as simple as Columbus’ egg : convert the document to PDF, duplicate the pages manually, and then ask for multiple pages per sheet. It works like a charm !

Need more guidance ? You’re in luck, for I did a video tutorial (my very first — be forgiving) to show the process step by step :

I’m demonstrating the solution on Illustrator CS5, but it works for any page that can be rendered on a PDF, so not only for Illustrator or Photoshop, but also for Microsoft Word and PowerPoint, or Apple Pages and Keynote.

Edit 8/dec : there’s a simpler procedure than the one explained above. Once you open the PDF in Preview, don’t duplicate the pages, and choose the number of Copies per page on the Preview tab (not the Layout tab) on the Print system dialog. I would have completely overlooked this if it weren’t for a helpful YouTube commenter, who also suggests that you can avoid the intermediate PDF step in Microsoft Word by, for example, putting  “1, 1, 1, 1” in Page Range (Copies & Pages tab), and then selecting 4 Pages per Sheet (Layout tab).