Razão & Sensibilidade — Episódio 4: Mudar de Idéia

Razão e Sensibilidade é o podcast onde transformamos incertezas em decisões corretas.

No quarto episódio, discuto a importância de mudar de idéia, e como reconhecer a verdade, embora desafiador do ponto de vista emocional, nos deixa mais fortes.

 

RSS Feed Apple Android Spotify

Razão & Sensibilidade — Episódio 3: Emoção

Razão e Sensibilidade é o podcast onde transformamos incertezas em decisões corretas.

Neste terceiro episódio, descobrimos que a irracionalidade dos comentários da internet é culpa dos tigres de dente de sabre. Exploramos nossa história evolutiva e falamos de inteligência emocional.

 

RSS Feed Apple Android Spotify

Razão & Sensibilidade — Episódio 2: Verdade

Razão e Sensibilidade é o podcast onde transformamos incertezas em decisões corretas.

Neste segundo episódio: dizem que a verdade nunca é pura e raramente é simples, mas nestes tempos de pós verdade, vou defender justamente uma ideia bem simples de verdade. Quero dizer: simples de entender, mas difícil de de querer. Querer a verdade é um trabalho que todos temos que desenvolver aos poucos. Vamos começar?

 

RSS Feed Apple Android Spotify

Razão & Sensibilidade — Episódio 1: Introdução

Razão e Sensibilidade é o podcast onde transformamos incertezas em decisões corretas.

Neste primeiro episódio: nuvens cinzas e sapatos novos. Torço para não chover, ou calço minhas galochas?

 

RSS Feed Apple Android Spotify

In Barcelona for NIPS’2016

I’m in Barcelona now for NIPS’2016 — or should I say, for NIPS’ Symposia and Workshops, since the main conference this year… sold out. That’s at once exciting and frightening: is machine learning the next dot.com?

Anyways — science ! We’re participating, with posters, in two workshops: Adversarial Training (on Friday, 9th) and Bayesian Deep Learning (on Saturday, 10th). Will you be there, let’s talk!

The papers:

Adversarial Images for Variational Autoencoders

Pedro Tabacof, Julia Tavares, Eduardo Valle

We investigate adversarial attacks for autoencoders. We propose a procedure that distorts the input image to mislead the autoencoder in reconstructing a completely different target image. We attack the internal latent representations, attempting to make the adversarial input produce an internal representation as similar as possible as the target’s. We find that autoencoders are much more robust to the attack than classifiers: while some examples have tolerably small input distortion, and reasonable similarity to the target image, there is a quasi-linear trade-off between those aims. We report results on MNIST and SVHN datasets, and also test regular deterministic autoencoders, reaching similar conclusions in all cases. Finally, we show that the usual adversarial attack for classifiers, while being much easier, also presents a direct proportion between distortion on the input, and misdirection on the output. That proportionality however is hidden by the normalization of the output, which maps a linear layer into non-linear probabilities.

The fulltext here:  https://arxiv.org/abs/1612.00155

🔔 The organizers of the workshop have opened a reddit thread for the public to ask questions. We have a subthread there — ask us anything! 🔔

Known Unknowns: Uncertainty Quality in Bayesian Neural Networks

Ramon Oliveira, Pedro Tabacof, Eduardo Valle

We evaluate the uncertainty quality in neural networks using anomaly detection. We extract uncertainty measures (e.g. entropy) from the predictions of candidate models, use those measures as features for an anomaly detector, and gauge how well the detector differentiates known from unknown classes. We assign higher uncertainty quality to candidate models that lead to better detectors. We also propose a novel method for sampling a variational approximation of a Bayesian neural network, called One-Sample Bayesian Approximation (OSBA). We experiment on two datasets, MNIST and CIFAR10. We compare the following candidate neural network models: Maximum Likelihood, Bayesian Dropout, OSBA, and — for MNIST — the standard variational approximation. We show that Bayesian Dropout and OSBA provide better uncertainty information than Maximum Likelihood, and are essentially equivalent to the standard variational approximation, but much faster.

The fulltext here:  https://arxiv.org/abs/1612.01251

My postgraduate offer for 2017/1: Designing IT Research for Impact

I’ll offer, next semester, the subject “IA376 — Topics in Computer Engineering VII — Class D: Designing IT Research for Impact” to the postgraduate students at FEEC/UNICAMP. That will be a 4 credits (60 hours) subject.

When: Tuesdays and Thursdays, from 19 to 21h

Where: FEEC/UNICAMP, Post-graduate Hall, Room PE-24.

We aspire our research to impact the world beyond h-indices and impact factors — we dream that our work, directly or indirectly, improves people’s lives. I start with a tease: do you have any evidence that this is the case for your research? My aim is for us to learn to ask that question, and then to give it a fruitful answer.

Let’s mine academic sources, government policies, NGO/Charities experiences, think-tank statements to answer the following questions:
(1) How can I design research in IT to maximize social impact?
(2) How can I work with the intended communities to to favor that impact?
(3) How can I measure (or at least inspect) that impact?

Our scope will be limited to Information Technology (i.e., “a GIS to improve access to clean water” is within scope, “a new polymer to clean water” not so much), and to disfavored communities (i.e., “real-time social networks to improve sanitization” is within scope, “real-time social networks to exchange designer watches” not so much).

This course will be nothing like your typical classroom experience. There will be no lectures. We will meet for the presencial sessions to discuss previous work, and plan our attack for the next week. I’ll expect you to continue working throughout the week. There will be no exams: I’ll grade your work based on participation during the sessions, progress between sessions, self assessment, and peer assessment. Active participation will be mandatory. This means (surprise!) talking in public. Everyone will be learning together, so all of us must accept the risk to be wrong. This course won’t work for those who always want to appear wise and knowledgeable. Group work will be highly encouraged. The course will be in English.

We’ll be a cozy small group: at most 18 students. I’ll select the candidates based on a letter of intentions, and on previous experience. Write a short e-mail to dovalle@dca.fee.unicamp.br with a short-résumé (American style) summarizing your research, professional, and extra-curricular activities, highlighting experience relevant for the course; add also a short statement (300~600 words) explaining your current research, or future research that you’d like to develop, highlighting the social impact you’d like to see. We will be particularly interested in IT research for Health and Education benefiting disfavored communities, but other themes are welcome.

Attention to the procedures and deadlines: Whether you are a regular or special-enrolled student, please send me your application by e-mail (as explained above) until 8 January 2017. You’ll have also to make your enrollment as usual. If you want to enroll as a “special-enrolled student” act now: pre-enrollment closes on December  7th! Regular students can follow the usual enrollment calendar.

More info at FEEC’s Postgraduation Page.

 

Commencement Speech for the Computer Scientists and Engineers

I had the honor and pleasure to give a commencement speech for the Computer Scientists and Engineers who graduated at UNICAMP yesterday. Prof. Dr. Ricardo Torres, Dean of the Institute of Computing, gave the main speech and addressed the graduates. As Director of Graduate Studies for Computer Engineering, I addressed the public, before him. Below is a translation of what I said ; the original, in Portuguese, is at the end.

My colleagues at the board,
Dearest graduates,
Dear parents,
Ladies and gentlemen.

It is always a joy to participate at this solemnity. Although UNICAMP has many endeavors, certainly Undergraduate Teaching is the most conspicuous, the one that brings us closer to the public, the one that, so to speak, puts us in people’s hearts. Thus, Graduation brings us all — students and professors — a feeling of mission accomplished. I’ll start, therefore, congratulating — but also thanking — the graduates for their achievement.

Congratulations also to the parents, the families ; we know that this victory is also yours.

But as Director of Undergraduate Studies, I think of all our graduates : not only today’s, also tomorrow’s. So I’ll ask for a few minutes of your patience to talk about the courses and professions of Computing.

Professions, I say, in the plural, since of all areas, Informatics is among the largest, most welcoming. I prefer that word, Informatics, because the term Computing puts too much emphasis on calculations, and even on the device itself, the computer. Let’s put the focus where it is due : on information.

Information that changed the world. Who can conceive life without the Internet ? Without Wikipedia ? Or even WhatsApp ? Remember that the availability of Internet to the general public has about 20 years ; Wikipedia has 15 ; WhatsApp has 6. Six years ! What else had such an impact in such a short time ?

British writer Arthur C. Clarke said that any sufficiently advanced technology is indistinguishable from magic. And the recent achievements of Informatics seem indeed a thing of magic.

We have today augmented reality translators, where we film a sign, a marquee, and the screen shows the same image with the text in another language ;

Applications that map our way from home to work, diverting from traffic, and one day warn us “leave out earlier today because there is a traffic jam” ;

And there is also this: [ speaker takes the phone and says ] Find commencement speeches. [ The phone answers ] “Okay, here’s what I found on Internet about commencement speech.’

Magic ! — Or is it ?

As educators, it’s our ambition to remove this veil of magic. Our Universe after all, is not the World of Harry Potter, it’s not the Middle-Earth of Hobbits, it’s not Narnia. No : our world is much more interesting than all those, because it’s a world structured by science. Science maybe doesn’t get the same glamor as magic in Hollywood movies, but its power is ultimately higher — a system to find truth, to understand, and eventually to manipulate reality.

A device like this [smartphone] does not arise by spontaneous generation, it does not grow on trees, and neither is the result of a spell. Each of the thousands — or probably millions — of components and processes, of software and of hardware, that made this object possible, was the result of the deliberate actions of a person. There is no magic : there is a complex network of knowledge, inspiration, and sweat.

That complexity is one strength of Informatics ; it’s what makes the profession so diverse, so full of roles. Some design the hardware, some write the software, some support to the user, some design the interface, some even think of the user experience, from the moment of taking the product out of the box, until the moment of giving it back for recycling. There are people in computing who spend months without writing ten lines of code, and others who produce thousands per month. Some spend their lives dealing with people, mediating, negotiating ; others hide behind their keyboards and, of human interaction, are content with “good morning”. There is room for all inclinations, and for all vocations.

Complexity is also a weakness of Informatics. It is a profession misunderstood by the public, full of myths, stereotypes. I’d like to dispel some of them.

The first myth is that Computing is bureaucratic, without room for creativity. On the contrary, Informatics is creative, expressive. A computer program is, quite literally, a piece of thought fixed on a support, which the programmer shapes, gives a breath of life, and it gains its own independent existence. If that is not creation, it is not expression, then what is ? — And I’m not even touching the more conventionally artistic areas of the profession, such as interface, interaction, and product design.

The second myth is that computing is reserved for nerds, or for the wunderkinder. Look, nothing against nerds — after all, I am one of them — but we have in no way a monopoly of Informatics. It also makes me sad, not to mention furious with our education system, when a 10-year old child, who is having some trouble with the multiplication tables for seven decides (or has it decided for her) that [1] she is not good at Math ; [2] she was not born with the gift of Mathematics ; [3] she should not aspire to a career in STEM. Out with this nonsense ! Math is like everything in life, it’s like jumping rope, it’s like dancing tango : you learn it with effort, you learn it with persistence — it is not a matter of genetics, it is a matter of toil. And the real Mathematics, creative, expressive, useful — the Mathematics that changes the world — has nothing to do with being very good at the seven times table.

The third myth is that the impact of computing is technical, distant from people. Given all that I said, this one should be already gone, but since it’s a very insidious one, I’ll make it explicit. It saddens me to watch young people abandoning a vocation for Computing, for Engineering, and seeking other professions — usually Law or Medicine — because they want to “help people”. Nothing against Law and Medicine, beautiful professions, but that is an awful reason to abandon Computing. Just think a little bit beyond the obvious to realize the enormous impact — social, cultural, ecological, economic, you can pick — a committed computing professional has to a large number of people. What kind of impact has a comprehensive, high quality, free encyclopedia ? A channel of expression where all citizens can post their videos and be watched worldwide, without passing the scrutiny, the censorship of governments, of broadcasters ? A funding social network where any project may come to life, without having to please a conservative committee of investors ? Think of Wikipedia, of YouTube, of Kickstarter, the next time you be tempted to think Computing does not help people.

And finally, I have to attack the old myth — that we still hear — that Informatics is a male profession, a profession reserved for boys. How many talented girls, young women, do not give up Computing because they believe that nonsense ? The result is that still today our course is more sought after by boys than by girls, but fortunately that is changing. We are committed to making our environment increasingly diverse, and to be welcoming to all genders. In that spirit, I pray — I beg, in fact — that parents show enthusiasm for their daughters who become interested in Computing, the same enthusiasm that they usually manifest when the interest comes from their sons. And now I address directly the sisters, cousins, nieces, not still in College age who are here today : yes, Computing is for you, we are counting on you, we are waiting for you when the time comes to enter UNICAMP.

Ladies, gentlemen, I have by now certainly took more of your time than civility allowed. I am greatly thankful for the opportunity to participate in that wonderful moment, and again, congratulating our graduates, I close my speech.

 

Colegas da mesa,
Queridas formandas, queridos formandos,
Senhores pais,
Senhoras e senhores.

É sempre com alegria que participamos desta Solenidade. Embora a UNICAMP desenvolva muitas atividades, certamente o Ensino de Graduação é a mais presente na lembrança, e é a que mais nos aproxima do público, a que, por assim dizer, nos coloca no coração das pessoas. Sendo assim, a Formatura é um momento que nos dá a todos — estudantes e professores — a sensação de dever cumprido. Dessa forma, eu começo, parabenizando — mas também agradecendo — aos formandos por essa conquista.

Parabenizo também aos pais, às famílias, pois sabemos que essa vitória também é de vocês.

Mas sendo Coordenador do Curso, eu devo me preocupar com todos os nossos formandos: não só os de hoje, também os do futuro. Por isso vou pedir alguns minutos da sua paciência para falar dos cursos e das profissões de Computação.

Profissões, digo eu, no plural, pois de todas as áreas, a Informática está entre as mais amplas, mais acolhedoras. Gosto dessa palavra, Informática, porque o termo Computação coloca ênfase demais nos cálculos, e até no próprio aparelho, o computador. Vamos colocar o foco onde é devido, na Informação.

Informação que transformou o mundo. Quem de nós concebe a vida sem Internet ? Sem Wikipedia ? Sem WhatsApp ? Ora, eu quero lembrar que a Internet para o grande público tem cerca de 20 anos ; a Wikipedia, 15 ; o WhatsApp, 6. Seis anos ! Que outras mudanças tiveram tanto impacto em tão pouco tempo ?

O escritor britânico Arthur C. Clarke dizia que qualquer tecnologia suficientemente avançada é indistinguível de magia. E as conquistas recentes da Informática parecem mesmo coisa de mágica :

Temos hoje tradutores de realidade aumentada, em que filmamos uma placa, um letreiro, e a tela nos mostra a mesma imagem com o texto em outra língua ;

Aplicativos que traçam nosso caminho da casa para o trabalho, desviando do trânsito, e um belo dia nos avisam “saia mais cedo hoje, porque há um engarrafamento” ;

E há também isso : [ orador toma o telefone e diz ] Encontre discursos de formatura.   [ o telefone responde ] “Okay, aqui está o que eu encontrei na Internet sobre ‘discurso de formatura’.”

Mágica ! — Ou será mesmo ?

É nossa ambição como educadores remover esse véu da magia. Nosso Universo afinal, não é o Mundo de Harry Potter, não é a Terra Média dos Hobbits, não é Narnia. Não : nosso mundo é muito mais interessante que todos esses, pois é um mundo estruturado pela ciência. A ciência pode não ter o mesmo glamour da magia nos filmes de Hollywood, mas o poder dela é maior — um sistema de encontrar a verdade, que nos permite entender, e finalmente manipular a realidade.

Um aparelho como este [smartphone] não surge por geração espontânea, não cresce em árvores, e nem é resultado de um feitiço. Cada um dos milhares — ou provavelmente milhões — de componentes e processos, de hardware e de software que tornaram este objetivo possível, foi resultado da ação deliberada de uma pessoa. Não existe mágica : existe uma complexa rede de conhecimento, inspiração, e suor.

Essa complexidade é uma das forças da Informática, é ela que torna a profissão tão diversa, tão cheia de papéis. Há quem projete o hardware, há quem escreva o software, há quem dê suporte ao usuário, há quem desenhe a interface, há até quem pense a experiência do usuário, desde o momento de tirar o produto da caixa, até o devolve-lo para reciclagem. Há computeiros que passam meses sem escrever dez linhas sequer de programa, e outras que produzem milhares por mês. Alguns passam a vida lidando com pessoas, mediando, negociando ; outras se escondem atrás do teclado e, de interação humana, já estão felizes com “bom dia”. Há espaço para todos os perfis, e para todas as vocações.

A complexidade é também uma fraqueza da Informática. É uma profissão mal compreendida pelo público, cheia de mitos, de estereótipos. Eu gostaria de desfazer alguns deles.

O primeiro mito é que a Computação é burocrática, sem espaço para a criatividade. Muito ao contrário, a Informática é criativa, é expressiva. Um programa de computador é, quase literalmente, um pedaço de pensamento fixado em um suporte, que o programador molda, dá um sopro da vida, e ele cria uma existência própria, independente. Se isso não é criação, não é expressão, então o que é ? — E eu nem estou mencionando as áreas mais convencionalmente artísticas da profissão, como o design de interface, de interação, e de produto.

O segundo mito é que a Computação é reservada aos nerds, ou então aos prodígios. Olha, nada contra os nerds — afinal eu sou um deles — mas nós não temos nenhum monopólio da Informática. Além disso, me deixa tristíssimo, para não dizer furioso com nosso sistema de ensino, quando uma criança de 10 anos que está com dificuldade na tabuada do sete decide (ou então é decidido por ela) que [1] ela não é boa em matemática ; [2] ela não nasceu com o dom da matemática ; [3] ela não deve seguir uma profissão de exatas. Vamos abandonar essas bobagens ! Matemática é como tudo na vida, é como pular corda, é como dançar tango : se aprende com esforço, se aprende com persistência — não é questão de genética, é questão de labuta. E a verdadeira Matemática, criativa, expressiva, útil — a Matemática que muda o mundo — não tem nada a ver com ser muito bom na tabuada do sete.

O terceiro mito é que o impacto da Computação é técnico, distante do humano. Dado tudo o que eu falei, era para esse já estar desfeito, mas ele é muito insidioso, então eu vou deixar explícito. Me entristece ver jovens abandonarem a vocação para a Computação, para a Engenharia, e procurarem outras profissões — normalmente o Direito e a Medicina — porque querem “ajudar as pessoas”. Nada contra Direito e Medicina, profissões belíssimas, mas essa é uma péssima razão para abandonar a Informática. Basta pensar um mínimo além do imediato para perceber o enorme impacto — social, cultural, ecológico, econômico : podem escolher — que um profissional da informática engajado tem para um grande número de pessoas. Que impacto traz criar uma enciclopédia abrangente, de alta qualidade, gratuita ? Um canal de expressão onde qualquer cidadão pode postar o seu vídeo e ser assistido no mundo inteiro, sem passar pelo crivo ou censura de governos, de emissoras ? Uma rede social de financiamento onde qualquer projeto pode ganhar vida, sem ter que agradar um comitê conservador de investidores ? Pensem na Wikipedia, no YouTube, no Kickstarter a próxima vez em que ficarem tentados a pensar que a informática não ajuda as pessoas.

E finalmente, tenho que atacar o velho mito, que a gente ainda ouve, de que a Informática é uma profissão masculina, uma profissão reservada aos meninos. Quantas meninas, jovens, mulheres talentosas não deixam de vir para a Computação porque acreditaram nessa bobagem ? O resultado é termos um curso ainda muito mais procurado pelos meninos do que pelas meninas, mas felizmente isso já está mudando. Nós estamos engajados em tornar o curso cada vez mais diverso, e acolhedor para todos os gêneros. Nesse sentido é que eu rogo, aliás imploro, aos pais que manifestem entusiasmo pelas filhas que se interessarem pela Informática, o mesmo entusiasmo que normalmente manifestam quando o interesse vem dos filhos homens. E deixo aqui minha mensagem para as irmãs, para as primas, para as sobrinhas que ainda não ingressaram na Universidade. A Computação é para vocês sim, e nós estamos contando com vocês, esperando por vocês quando chegar a hora de vocês virem para a UNICAMP !

Senhoras, senhores, eu já me alonguei certamente mais do que o bom tom permitia. Agradeço imensamente a oportunidade de participar desse momento maravilhoso, e parabenizando novamente nossos formandos encerro aqui a minha fala.

My postgraduate offer for 2016/1 : Deep Learning From a Statistician’s Viewpoint

With few exceptions, my postgraduate offers follow a pattern. On the second semester, I offer my “101” Multimedia Information Retrieval course, which introduces multimedia representations, machine learning, computer vision, and… information retrieval. On the first semester, I offer a topics course, usually following a book : so far we have explored Bishop’s PRML, Hofstadter’s GEB, and Jaynes’ “Probability Theory”.

For 2016/1, I’m risking something different :

“Artificial Intelligence is trending again, and much of the buzz is due to Deep Neural Networks. For long considered untrainable, Deep Networks were boosted by a leap in computing power, and in data availability.

Deep Networks stunned the world by classifying images into thousands of categories with accuracy, by writing fake wikipedia articles with panache, and by playing difficult videogames with competence.

My aim here is a less “neural” path to deep models. Let us take the biological metaphors with a healthy dose of cynicism and seek explanations instead in statistics, in information theory, in probability theory. Remember linear regression ? Deep models are multi-layered generalized linear models whose parameters are learned by maximum likelihood. Let us start from there and then explore the most promising avenues leading to the current state of the art.

This course will be nothing like your typical classroom experience. There will be no lectures. We will meet once a week for a presencial session to discuss previous work, and plan our attack for the next week. I’ll expect you to continue working throughout the week. There will be no exams. I’ll grade your work based on participation during the sessions, progress between sessions, self assessment, and peer assessment.

Active participation will be mandatory. This means (surprise !) talking in public. Everyone will be learning together, so all of us must accept the risk to be wrong. This course won’t work for those who always want to appear wise and knowledgeable. The course will be in English.

Deep networks can be seen as hierarchical generalized linear models.

Deep networks can be seen as hierarchical generalized linear models.

We’ll be a cozy small group : at most 12 students. I’ll select the candidates based on a letter of intentions, and on previous experience. Write a short e-mail to dovalle@dca.fee.unicamp.br. No need to be fancy : just state your reasons for participating, and any previous experience (academic, professional, and extra-curricular) with Machine Learning, Statistics, Probability, or Information Theory.

This course is not for beginners, nor for the faint of heart. We are jumping in head first at the deep (tee hee !) end. After all, we will delve into one of the most engaging intellectual frontier of our time. I dare you to join us !”

Very important ! If you want to enroll at this course without being enrolled at the program (what UNICAMP awfully calls “special students”), you have to do you pre-enrollment until 7/Dec/2015 (hard deadline !). Even if you are enrolled at the program (“regular student”) send me your application at most until 31/Dec/2015, because I’ll select regular and special (urgh !) students at the same time.

EDIT 20/01 : I have sent the acceptance notices — looking forward to work with a swell group of very motivated students !

What : Post-graduate course for the Master or Doctorate in Electrical Engineering program of UNICAMP (4 credits)

When : 2016/1st semester — mandatory presencial meetings Tuesdays from 19 to 21h ; support meetings same day from 16 to 18h

Image credit : composite from Ramón y Cajal 1st publication showing a cerebellum cut, and scatterplots from Fisher’s iris dataset drawn by Indon~commonswiki, wikimediacommons.

Installing software on Linux without root : managing packages in user space

First step : don’t. Reconsider. There really isn’t an alternative ? Using something else that is already installed ? Sweet-talking your sysadmin into doing the installation ? Giving up that particular task ? Giving up Computer Sciences altogether and moving to the country to raise pigs ?

Ok, so you really don’t have an alternative. May the gods have mercy on your soul, because Linux won’t. By necessity, this won’t be a step-by-step guide, because each system has its quirks. I’m not promising you heaven, just a tolerable purgatory instead of a living hell.

Take a deep breath.

(And as always : follow those instructions at your own risk. If you turn your Linux box into a Linux brick, or into Skynet, I’m not liable.)

The problem : dependency hell

There’s a reason why installing software from sources is so painful : dependencies. Sure, you just want to install Caffe, or Torch7, or Theano. But Theano needs python, python needs openssl, openssl needs… it’s endless.

High-level package managers like apt-get and yum are so popular because they deal with those. When installing from source, you’re on your own.

But here’s the catch : when installing from sources, you can almost always relocate the software to your ~home, bypassing the need for root access. High-level package managers, at least the current generation, can’t relocate.

Except for Homebrew.

The strategy : Linuxbrew

Homebrew was created as “the missing package manager for OS X”, and is required to do anything interesting on a Mac. It was designed around two principles : installation at the user home, and installation from sources.

Say that again ? Installation at the user home, without need for root. From sources. Wow. If only there was a version for Linux ! Enter Linuxbrew. Homebrew concept was so successful that, in an ironic turn, it’s now becoming “the missing package manager for Linux”.

So, case closed ? Hardly. To start, Linuxbrew has dependencies of its own, and you have to take care of those by hand. Then, the software you want to install has to be available as a brew “formula” (but the list is quite comprehensive, and growing). Finally, it doesn’t always goes smoothly. Linuxbrew is a much bumpier ride than Homebrew/OS X, at least for now. Most formulas will install without issue, but a good 20% will require tweaking, googling, and deciphering forum posts.

The strategy is most definitely not user-friendly. But contrarily to installing each and every package by hand it is just user-bearable enough to be feasible. If you really need the software. (Are you sure you don’t prefer the pig farm ?)

Okay, you are sure.

Our strategy will be :

  1. Ensuring Linuxbrew dependencies ;
  2. Installing and configuring Linuxbrew ;
  3. Using Linuxbrew to install the desired software… ;
  4. …or if you’re unlucky, using Linuxbrew to install the desired software dependencies, and then installing the desired software by hand.

The tatics

Installing Linuxbrew dependencies

Linuxbrew has, fortunately, few dependencies : Ruby, GCC, Git, and… Linux (duh !). It runs on x86 or 32-bit ARM platforms. If you’re running Linux in other architectures this is your cue to break down sobbing.

Most systems will, fortunately have those dependencies already installed. You can check the minimal versions currently required by Linuxbrew, and then check the versions installed at your system (and whether they are installed at all) calling the commands with --version :

$ ruby --version
ruby 1.8.7 (2011-12-28 patchlevel 357) [x86_64-linux]

$ gcc --version
gcc (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973]
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ git --version
If 'git' is not a typo you can run the following command to lookup the package that contains the binary:
command-not-found git
-bash: git: command not found

Linuxbrew won’t turn your dependency hell into a package management heaven, but it might turn it into a tolerable purgatory.

Linuxbrew won’t turn your dependency hell into a package management heaven, but it might turn it into a tolerable purgatory.

As you see, I got almost lucky. Ruby and GCC are good to go, but I’ll have to install Git (and its dependencies).

The trick is being minimalist. Many packages have optional features : extra commands, documentation, GUI interfaces, etc., that will fire zillions of extra dependencies. Get rid of those optionals as much as possible.

I recommend a top-down plan of attack. Start with the package you want to install and try to configure–make–install it. If it breaks (and it will, it will), find out which dependency/configuration is missing and correct it. Do this recursively, depth-first, until you get everything installed.

Resist perfectionism. You might spend a lot of time smoothing out every wrinkle of package-1.73.03 just to find a bit later that it breaks your installation and has to be removed to make room for package-1.73.04. This is war, kid : not a time for bells and whistles. Once you get a dependency working, move on to the next.

In more detail, each cycle will consist of :

  1.  Finding, downloading, and unpacking the source package ;
  2.  Configuring the package to work with a $HOME prefix ;
  3. Building the package ;
  4. Installing the package.

Step 1 is usually trivial after bit of googling. If your Linux distribution is Debian-based, you might be able to use a single command-line operation :

apt-get source git

There are similar facilities for other high-level package managers.

Otherwise, you might download either a compressed source file, or even the bleeding edge version from the source repository (sourceforge, github, etc.) In the case of Git, this would be at https://github.com/git/git. (Be careful, because those hyper-up-to-date versions might be unstable.)

Step 2 varies a bit from package to package, but usually consists in calling a ./configure script. Sometimes pre-configuration is involved : a call to make configure or make config, or another script, e.g., ./buildconf. Sometimes it involves cmake (cross your fingers for having autoconf/automake already installed). Sometimes there’s no step 2, all options being passed directly to make during step 3. It varies.

How will you know ? Try to locate a INSTALL.* or README.* file. Usually the instructions are there. No luck ? Try browsing the official website of the package for installations instructions. Googling <package> installation instructions usually will point you to the right direction.

For git, this will work :

cd git-2.1.4/
./configure --prefix=$HOME

Well, sort of. It will probably break, because one or more dependencies will be missing. Install those (and their recursive dependencies) and try again.

Step 3 is almost always :

make

or sometimes :

make all

Sometimes this is the moment when things break down for lack of dependencies (or wrong versions, or wrong settings, or the universe showing its lack of care). Sometimes the --prefix=$HOME option comes here instead of Step 2.

Step 4 is almost always :

make install

If you set the prefixes right, that will automagically put everything in the right place, under your ~home directory. And you won’t need root permissions.

Got it ? Good. I hope you enjoy typing command-line commands : you’ll be doing it all day. For extra enjoyment, get a second monitor and close the shades.

Installing Linuxbrew

Once you have all dependencies working, installing Linuxbrew itself is a breeze :

git clone https://github.com/Homebrew/linuxbrew.git ~/.linuxbrew

Aaand… that’s it. It won’t work immediately because you have to set the paths (see below). After you do it you can simply type :

brew install $WHATEVER_YOU_WANT

And it should take care of everything.

Before you do it, however it is a good idea to call

brew doctor

and check if everything is ok. Again, be minimalist : you don’t have to correct every tiny issue. Take a good look and make the smallest needed intervention.

Linuxbrew comes ready with a lot of recipes for installing packages, or as they call, formulas. You can keep them up do date by typing

brew update

Depending on what you want to install, however, you’ll need extra formulas. In Homebrew/Linuxbrew parlance this is called tapping. For example :

brew tap homebrew/science

will install a lot of new formulas related to science, data analysis, etc.

PATH configurations

Both phases (manual dependency installations; Linuxbrew operation) won’t do you much good if your paths aren’t configured. There are at least four important paths, maybe more depending on your setup : executables (PATH), static libraries (LIBRARY_PATH), dynamic libraries (LD_LIBRARY_PATH), and include files (CPATH).

The usual place to set up those is your your shell configuration file. The examples below assume you’re using bash. If that’s your case, decide whether .bashrc or .bash_profile is better for you (usually it’s the former).

During the manual installation of dependencies add the following lines :

# Manually installed packages
export PATH="$HOME/bin:$PATH"
export LIBRARY_PATH="$HOME/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="$HOME/lib:$LD_LIBRARY_PATH"
export CPATH="$HOME/include:$CPATH"

During Linuxbrew operation put those additional lines :

# HomeBrew / LinuxBrew
export HOMEBREW_PREFIX="$HOME/.linuxbrew"
export PATH="$HOMEBREW_PREFIX/bin:$PATH"
export LIBRARY_PATH="$HOMEBREW_PREFIX/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="$HOMEBREW_PREFIX/lib:$LD_LIBRARY_PATH"
export CPATH="$HOMEBREW_PREFIX/include:$CPATH"

Remember that shell configurations are not effective immediately, only on the next start. You don’t have to reboot the system : simple closing and reopening the terminal, or logging out and back in suffices.

An ugly ugly ugly workaround

During my installations, I faced an issue with CA certificates that I could not bypass. Many formulas would refuse to proceed, stopping during download with the error : “cURL error 60: SSL certificate problem: unable to get local issuer certificate”.

Yes : I tried downloading updated certificates from Mozilla Corp. Yes : I checked my curl-config --ca. Yes : I tried reinstalling cURL. And Git. And OpenSSL. I spent, litteraly, hours trying to solve the problem in an elegant way.

I concede defeat. Here’s the very inelegant solution. Be aware that it opens your package manager to man-in-the-middle attacks. That is more than a theoretical risk : it has been done. This is a huge security hole. If you decide to apply it, don’t do it preemptively, wait to see if you’ll actually get the SSL certificate problem.

So you got the error, and you’re willing to expose your neck to the wolves ? Sucks to be you. Open the file download_strategy.rb at ~/.linuxbrew/Library/Homebrew and find the lines below

# Curl options to be always passed to curl,
# with raw head calls (`curl -I`) or with actual `fetch`.
def _curl_opts
  copts = []
  copts << "--user" << meta.fetch(:user) if meta.key?(:user)
  copts
end

Change line four to

# Curl options to be always passed to curl,
# with raw head calls (`curl -I`) or with actual `fetch`.
def _curl_opts
  copts = ["-k"] # Disable certificate verification
  copts << "--user" << meta.fetch(:user) if meta.key?(:user)
  copts
end

And that’s it. You’re ready to proceed installing source packages. And to be a victim of cyber mafias, and of terrorists, and of tyrannical governments.

(Note to security people : if your watertight security solution makes a system unusable, guess what will happen ?)

Extra tips

First and foremost, source code relocation is not a panacea. Some things require root access, for example, driver installations, kernel recompilations, boot sector modifications, etc. You might want to check if your software require one of those before you start this whole adventure.

You can learn a lot about a formula with the info option

$ brew info python
python: stable 2.7.10 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org
Not installed
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python.rb
==> Dependencies
Build: pkg-config ✘
Required: openssl ✘
Recommended: readline ✘, sqlite ✘, gdbm ✘
Optional: homebrew/dupes/tcl-tk ✘, berkeley-db4 ✘
==> Options
--universal
  Build a universal binary
--with-berkeley-db4
  Build with berkeley-db4 support
--with-poll
  Enable select.poll, which is not fully implemented on OS X (https://bugs.python.org/issue5154)
--with-quicktest
  Run `make quicktest` after the build (for devs; may fail)
--with-tcl-tk
  Use Homebrew's Tk instead of OS X Tk (has optional Cocoa and threads support)
--without-gdbm
  Build without gdbm support
--without-readline
  Build without readline support
--without-sqlite
  Build without sqlite support
--HEAD
  Install HEAD version

Take a good look at the --without-* options because they are sometimes a lifesaver. Some packages have optional GUI extras. They might fire hundreds of useless extra dependencies — especially if you are installing on a headless server.

Sometimes Linuxbrew breaks down for the lack of a dependency, refusing to install it, but will gladly do it if you explicitly ask for it. For example : brew install package1 breaks for the lack of package2, and all it takes is typing brew install package2 and retrying package1. Mysteries.

Installation time is highly unpredictable. Sometimes a small innocent little package will require a precise version of GCC… that Linuxbrew will then have to install from the sources. Time for a tea.

If your installation becomes so corrupted with conflicting packages that you have to restart from scratch (nooooooo !), it can be — small consolation — accomplished easily :

rem ~/.linuxbrew -rf

For extra OCD cred, clean-up the download cache as well :

rm ~/.cache/Homebrew/ -rf

If the whole thing becomes so messed up that you have to scratch even the manual dependencies (two words : pig farm), it is also easily done :

rm ~/bin/ ~/lib/ -rf

You might also consider :

rm ~/include ~/etc -rf

but be careful because that might erase innocent third parties.

You might be forced to install multiple versions of the same package. That adds another nightmare layer to the ongoing nightmare, but it’s doable. Linuxbrew will usually be friendly enough to tell you what to do.

For example, when I had to install both opencv2 and opencv3 I got this :

opencv3 and opencv install many of the same files.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/home/valle/.linuxbrew/opt/opencv3/lib
    CPPFLAGS: -I/home/valle/.linuxbrew/opt/opencv3/include

Those little displays of care are the reason why I like Homebrew/Linuxbrew so much. Love breeds love : a truism even for human–computer interaction.

A lightweight web server with Lighttpd, PHP and SQLite3 on a Raspberry Pi

Many consumer devices (set top boxes, routers, media centers) use lightweight embedded web servers as user interface. In this walkthrough, I’ll show you how to install a complete web server in a Raspberry Pi, which is a cheap ARM-based computer that is perfect for prototyping such appliances.

In embedded systems, we aim at minimizing computational expenditure. Therefore, we’ll use Lighttpd as the web server, and SQLite as the DBMS. For server-side scripting, we’ll use PHP, a nice framework for protyping : easy to learn, and very popular.

Installing a fresh copy of Raspbian in an SD card

I’ll start from scratch, with a fresh Raspbian system. This is not strictly necessary : you can probably succeed from any working Raspbian installation.

You can start by downloading the Raspbian image, and installing it into the micro SD card. There are different instructions for Linux, for Windows, and for OS X.

For OS X, there are two sets of instructions. In my setup (MacBook Pro 13″, early 2011, Yosemite), only the second set worked, and I’ll explain it here. (If it doesn’t work for you, try the other one.)

Attention : we will be doing low-level disk transfers here. One single distraction and you may lose all your data. Your motivational PowerPoints. Those pictures of the 70th birthday of Auntie Joaquina. That spreadsheet you were almost finishing with all continuity mistakes in Bervely Hills, 90210. Gone ! Forever ! (No, seriously : backup your stuff. Really !)

(And my lawyer insists that I remind you : as always, proceed at your own risk. If you follow those instructions and your computer turns to a brick, or your cat divorces you, I’m not liable.)

First let’s unzip the disk image. Open a terminal in the folder where the image file is located, and type :

unzip 2015-05-05-raspbian-wheezy.zip

Now, let’s find out the device path to the SD card. First ensure that the SD card slot is empty, and type df -h in the shell. You should get something like :

$ df -h
Filesystem                          Size   Used  Avail Capacity   iused     ifree %iused  Mounted on
/dev/disk1                         930Gi  486Gi  444Gi    53% 127405636 116484474   52%   /
devfs                              183Ki  183Ki    0Bi   100%       636         0  100%   /dev
map -hosts                           0Bi    0Bi    0Bi   100%         0         0  100%   /net
map auto_home                        0Bi    0Bi    0Bi   100%         0         0  100%   /home
localhost:/p9Q694fzz20lCp5sv3CZyj  930Gi  930Gi    0Bi   100%         0         0  100%   /Volumes/MobileBackups

Now, insert the micro SD card in the slot (you might need an adapter), count five seconds, and type df-h again. You should get something like :

$ df -h
Filesystem                          Size   Used  Avail Capacity   iused     ifree %iused  Mounted on
/dev/disk1                         930Gi  486Gi  444Gi    53% 127405653 116484457   52%   /
devfs                              185Ki  185Ki    0Bi   100%       642         0  100%   /dev
map -hosts                           0Bi    0Bi    0Bi   100%         0         0  100%   /net
map auto_home                        0Bi    0Bi    0Bi   100%         0         0  100%   /home
localhost:/p9Q694fzz20lCp5sv3CZyj  930Gi  930Gi    0Bi   100%         0         0  100%   /Volumes/MobileBackups
/dev/disk2s1                        56Mi   20Mi   36Mi    36%       512         0  100%   /Volumes/boot

Got it ? There’s a new device /dev/disk2s1 for the SD card. You might get several new lines if the card currently has more than one partition (e.g. /dev/disk2s1, /dev/disk2s2, etc.)

We need to unmount all those partitions, but without ejecting the device (if you eject it, you’ll have to restart the whole process) :

sudo diskutil unmount /dev/disk2s1

Repeat that command for each partition in the device.

Now for the dangerous command. We will use the low-level copy command dd, using the disk image as input, and the SD card device as output. Take a deep breath and check each character thrice before hitting Enter, because overwriting the wrong device will be a nightmare.

Check out the partition names above. If you got /dev/disk2s1, you’ll write to /dev/rdisk2. If you got /dev/disk3s2, you’ll write to /dev/rdisk3, etc. Got it ? Just throw away the s<number> suffix and add an r to the beginning :

sudo dd bs=1m if=2015-05-05-raspbian-wheezy.img of=<device path to write>

This might take a while — in my system, 5 minutes or so. You may type ctrl+T to send the SIGINFO signal, and get an status update. Also, you might need to write bs=1m as bs=1M, depending on the version of dd.

This is it ! Eject the the card. Time to move to the Pi.

Configuring Raspbian

Insert the newly formatted SD card into the Pi, plug the needed devices. We will need at least : an ethernet connection to the Internet, a keyboard, and a monitor. Plug the power last.

With some luck, the Raspberry Pi Software Configuration Tool will appear during the first boot. (If it doesn’t appear, or if you are not using a fresh system, you can launch it by typing sudo raspi-config in the command shell.)

Be sure to at least activate option 1 (“Expand Filesystem”) so Raspbian will use the entire SD card. Otherwise the upgrade commands below risk running out of space. I will also use option 2 (“Change User Password”) to make access to the system secure ; option 3 (“Enable Boot to Desktop/Scratch“) to ensure that the system boots to Console Text (no need for graphical desktop in this small web server) ; option 4 (“Internationalization Options“) to configure my keyboard to US International; and option 8 (“Advanced Options“) to set the hostname, and to ensure that SSH access is enabled.

Updating and upgrading the system

Once the system reboots, log in (the default user is pi, and the defaul password, if you didn’t change it with option 2 above, is raspberry).

We’ll start by bringing the system up to speed :

sudo apt-get -y update
sudo apt-get -y dist-upgrade

Installing Lighttpd, SQLite, and PHP

Time to install the components of the web server. The instructions are adapted from smching’s Instructable.

sudo apt-get -y install lighttpd
sudo apt-get -y install sqlite3
sudo apt-get -y install php5 php5-common php5-cgi php5-sqlite 

Now for a bit of configuration. We’ll want to enable fastcgi to handle the PHP pages :

sudo lighty-enable-mod fastcgi
sudo lighty-enable-mod fastcgi-php
sudo service lighttpd force-reload # restart the Lighttpd service

Setting permissions

Finally, we need to set up the permissions of /var/www (the root of the web server), and include the default user on the group that can read/write that directory.

sudo chown -R www-data:www-data /var/www
sudo chmod -R 775 /var/www
sudo usermod -a -G www-data pi

Time for a reboot !

sudo reboot

Testing

If you have a second computer attached to your network, you can use it to access the recently installed web server. Before leaving the Pi, type ifconfig to get the IP. You should get something like…

$ ifconfig
eth0      Link encap:Ethernet  HWaddr b8:27:eb:7e:b2:3c  
          inet addr:10.0.1.7  Bcast:10.0.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:218 errors:0 dropped:0 overruns:0 frame:0
          TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:47677 (46.5 KiB)  TX bytes:18722 (18.2 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1104 (1.0 KiB)  TX bytes:1104 (1.0 KiB)

…showing that the IP attributed to the Pi on ethernet is 10.0.1.7.

You can type http://10.0.1.7/ on the browser of another computer to check the web server. You should see the placeholder page for Lighttpd showing that the web server is correctly installed.

Also — if you haven’t disabled SSH — you can type ssh pi@10.0.1.7 to access the Pi remotely. This is useful if you have a single set of monitor/keyboard/mouse.

If the Pi is your only computer, you can start the graphical desktop with the command startx, and then navigate to the newly installed web server. Typing the IP address of the machine, as before, will work, but so does using the loopback interface : http://127.0.0.1/ or http://localhost/.

Next, let’s test if PHP is working. Paste this simple test page…

<?PHP
phpinfo()
?>

…into /var/www/test.php.

Then, type in your browser http://10.0.1.7/test.php (change 10.0.1.7 for the actual IP of your Pi, or use http://localhost/test.php if browsing from inside the Pi). You should see the PHP info page.

Finally, let's run a small test on PHP + SQLite. I adapted this code from Veit Osiander's post on Scandio. Paste the following...

<?php
try {
    // Create file "scandio_test.db" as database
    $db = new PDO('sqlite:scandio_test.db');
    // Throw exceptions on error
    $db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
 
    $sql = <<<SQL
CREATE TABLE IF NOT EXISTS posts (
    id INTEGER PRIMARY KEY,
    message TEXT,
    created_at INTEGER
)
SQL;
    $db->exec($sql);
 
    $data = array(
        'Test '.rand(0, 10),
        'Data: '.uniqid(),
        'Date: '.date('d.m.Y H:i:s')
    );
 
    $sql = <<<SQL
INSERT INTO posts (message, created_at)
VALUES (:message, :created_at)
SQL;
 
    $stmt = $db->prepare($sql);
    foreach ($data as $message) {
        $stmt->bindParam(':message', $message, SQLITE3_TEXT);
        $stmt->bindParam(':created_at', time());
 
        $stmt->execute();
    }
 
    $result = $db->query('SELECT * FROM posts');
 
    foreach($result as $row) {
        list($id, $message, $createdAt) = $row;
        $output  = "Id: $id<br/>\n";
        $output .= "Message: $message<br/>\n";
        $output .= "Created at: ".date('d.m.Y H:i:s', $createdAt)."<br/>\n";
 
        echo $output;
    }
 
    $db->exec("DROP TABLE posts");
} catch(PDOException $e) {
    echo $e->getMessage();
    echo $e->getTraceAsString();
}
?>

...into /var/www/testdb.php.

Navigate to http://10.0.1.7/testdb.php (or http://localhost/testdb.php), and if everything goes well, you should get something like :

Id: 1
Message: Test 1
Created at: 30.09.2015 05:34:33
Id: 2
Message: Data: 560b746950a70
Created at: 30.09.2015 05:34:33
Id: 3
Message: Date: 30.09.2015 05:34:33
Created at: 30.09.2015 05:34:33
The raspberry pi is perfect to prototype a small appliance with a web server as interface.

The raspberry pi is perfect to prototype a small appliance with a web server as interface.

If you get an ugly error, like "General error: 14 unable to open database file", double check the permissions of /var/www (and everything inside it). The user www-data must have reading and writing permission on everything (see the section "Setting permissions" above).

Otherwise, you are ready to start working on your project !