Feliz Navidad

Saturday night

 While you are reading this, somebody, somewhere, is learning something...

Change

A month ago I went out for dinner with an old friend. We finally met in Chamartín, after a rather serendipitous choice of meeting point due to an unexpected change in train timetables. Looking at the buildings above, I was amazed at how much Madrid has changed in the time I have been away. But you could say the same about us, we have changed a lot as well. But our friendship didn´t. Time literally flew while talking as if we had just met a few weeks before.

This is the second person I have met this year after sending a letter last Christmas. Maybe it is a coincidence. Maybe not.

Gejagt

Before I go on holidays, I always make sure I have a few books added to my kindle, either programming books or novels. In this case, I was reading one from Cory Doctorow´s novels. Obviously I did not take this photo myself. It was my daughter who did, and true to the serenity project, I was deep into the book and never noticed it.

Biblioteca del Colegio Trinitarios

This library is in what used to be a church, and I like the idea of the altar being replaced by a desk and the retable by shelves with books. Although it was never one of my favourite libraries, I have fond memories of this place because this is one of the places where I prepared the very last exam before I got my degree. I had already finished my final degree thesis in Cranfield University as an exchange student and I got back home a few days in advance to prepare the exam for Helicopters, basically reviewing this book, which I still have in my shelf (!).

I have to say that, in my memory, this library was a lot bigger than it actually is.

Biblioteca de la Facultad de Derecho, UAH

I was recently back in Alcalá de Henares and decided to pay a visit to a couple of libraries in which I spent countless hours many years ago.

Not alone...

I just discovered https://projecteuler.net and solved problem #1. I liked the message
"You are the 602534th person to have solved this problem."

I feel that the real gold mine is the discussion threads.

Sometimes you win, sometimes you learn...

I recently participated in a competition (marketing stunt?) hosted by GE, called GEeks Go for #CC9900. There were three challenges, each one only opening once you had cracked the previous one. As it was to be expected, the fist two were reasonably easy. The third was not. There was only one prize for the first one to crack all three challenges. But the real prize was trying to solve the puzzles, having some fun in the way and learning some bits. Coding helped, but as far as I can see, it was not strictly required, definitely not for the first two.

Since the winner has already been announced, I guess that was it.

First challenge: Talking Machines

A number of twitter accounts were set up, impersonating Things which were having conversations. The only twist was that they were talking in binary, which in twitter only allows for very short conversations. It doesn't take long to notice that this must be some sort of ASCII code. Now, there were ten of these accounts, so although the instructions said that the tip would be in a conversation between two of them, I thought it would be best to have a script to automatically translate all of them, and not one character at a time. I have done this some other times while scraping webs for some content, like in this example.

The problem I had was that conversations are added to the page by a script, so I could not save the html directly and use beautifulsoup to extract the text.Something like this wouldn´t work
import requests
res = requests.get('https://www.geeksgoforgold.com')
print res.text()

Same for urllib.urlopen(url)

Maybe because the page does not allow robots? No idea. But if you know why, please leave a message with the explanation.

Eventually, I just pressed F12 to see the html, navigated to the class with the list of tweets and manually saved them in a text file. From there, extracting each tweet was simple, since there was a timeline-Tweet-text class which was easy to pick up with the code, and then translating the binary tweets to something readable was a breeze. 

But of course, it couldn´t be so simple. One of the tweets did not seem to have ASCII characters. So what do you next? Well, you try to see if they are unicode characters. Which they were and so the first code was cracked.

filein = open(r'.\GE_BULB.txt','r')

myhtml = filein.read()

GEsoup = bs4.BeautifulSoup(myhtml)

GE_Tweets = GEsoup.select('p[class="timeline-Tweet-text"]')

for tweet in reversed(GE_Tweets):

    s=''.join(unichr(int(letters,2)) for letters in tweet.getText().split() if letters.isdigit())

    print s

filein.close()

Funny thing is that by the time I joined the challenge, GE had already posted a clue in the form of an ascii converter! So I could have saved all the coding and just spent a few minutes copy pasting tweets. OK, I think it was more fun like this.

Second challenge: Hidden word

The second challenge did not seem to require much coding skills. The task was to find a hidden message somewhere in the image. However the image was actually like a Google map, with several levels of zooming, and of course the more you zoom in, the more tiles you have to inspect. At the highest zoom level, there were some 16 thousand tiles to be inspected, which with loading times and what not, starts to become a really tedious task. So web scraping abilities again to the rescue, I wrote a script that would download all the tiles in one go so that I could inspect them. Actually I was even thinking of running an OCR tool on each of them. Now, in one go here means in a long, long go. It takes quite some hours to download all the tiles (there must be better ways to do it). In the mean time I got to talk with one of the guys doing the challenge and he kindly gave me a dump of all the tiles. I still let the script run and keep downloading the images, just for the sake of it. He even put together all of them into two huge halves.

Because the image was using a google map class, I ended up clicking in the google logo as well, which led me to https://beyondthemap.withgoogle.com/en-us/ This has nothing to do with the challenge, but it is interesting in itself.

.

import requests
for n in range(128):
    for m in range(128):
        res = requests.get('https://www.geeksgoforgold.com/version/src/inc/image/tiles/7/'+str(n)+'/'+str(m)+'.jpg')
        imageFile = open(str(n)+'x'+str(m)+'.jpg','wb')
        imageFile.write(res.content)
        imageFile.close()
        print n, m

OCR is reasonably straight forward, but like everything, theory and practice are two separate things. Luckily I realized that I am a reasonably good scanning device myself. I loaded all the photos to Picasa and had them displayed in sufficient size to scan quickly through them. Half way through my eye caught a discrepancy. And bingo, two letters in ascii code. On to the third challenge!

Third challenge: WTF!

The third challenge was the final one, so this had to be the one separating the wannabees and script kiddies like myself and the pros. But since I was there and had made it by my own means, I may as well give it a go. First problem, not a coding puzzle, or not that I could see at first sight but a riddle instead of instructions and one image to download, but no submission button. What am I supposed to do with this? As it happens, quite a lot of things. They didn't work (like in I didn't win), but it was fun trying them.

Gimping the image to death with filters.

Nothing obvious

Unwarping the image.

You notice that huge vertical line from the center, do you? So did I.
I started by writing my own script to transform a round image to a rectangular one, and finally used a variation of this

I also tried other things like flattening the unwarped image, but nothing obvious there. Hold on, is that not a kind of @ in there?. The first challenge had to do with twitter accounts. So this could be the hiddend clue. Nope.

Translate image name to unicode

The image to be downloaded had a somewhat interesting name: "dc046bc021a7d3b818958e3a524cec68.png" I thought that a clue could be in the name, so I translated the dc04 6bc0 21a7 d3b8 1895 8e3a 524c ec68 to unicode. When I realized that this resulted in some chinese characters I thought, wow, I may win this one! But when I translated those I thought, ok, either an easter egg form a disgruntled programmer, or a really strange coincidence. No idea, but I could not really start anything with this.

Text in PNG file

Next idea, what about reading the binary file? I had never poked into a png file, but this lead me to this article, which clearly says that it is possible to embed text into a png file. And there was indeed some text and metadata, but nothing that seemed like a clue.

IPv6 address

What if the name of the file is a hidden IPv6 address? That got me tinkering with my router to see how IPv6 addresses can be enabled. Unfortunately http://[dc04:6bc0:21a7:d3b8:1895:8e3a:524c:ec68] did not work. In the mean time, I have read a couple of interesting articles on IPv6, so I guess, time well spent.

Long-Lat coordinates

What if the name of the file contains some hidden longitude and latitude coordinates? I ended up in weird places in Google Maps. In one case, this was pretty much in the border of Russia, not really what I was expecting. But this led to another idea. There were a couple of addresses mentioned in the terms and conditions of the challenge. One in Connecticut and one in Australia. I went straight into the streetmap view to see if there were some clues posted in or around those locations. No luck.

GE Clues

I have to say that I didn´t find the clues very enlightening. One of them was about the realizing that I was the point of view was in the center. After thoroughly checking the center of the image, I decided to check some of the latest 360 videos posted by GE in youtube. Again, interesting but no obvious clues there.

Slice the ripples and checking histogram

I remembered reading once about hidden messages in histograms. This could be the one! Not quite.

And I guess that was it. Congratulations to the winner Zoltán Szabó! 

Update:
There was another cue later on that had something to do with listening to the crowds. This made me think of signal processing (admittedly not my forte) and I ended up here, I already knew the Think Python book, but finding a whole series of free books is even better. Unfortunately I ran out of time to try new things due to private issues. All in all, a thoroughly enjoyable experience.

Stadtbücherei Bad Sachsa

This library is simply amazing, a joy for the senses and it was just delicious to wander around the small rooms and corridors and snap a couple of photos here and there. By way of comparison, check this one other out. Not quite the same.

Another amazing thing is how much time and effort I spent over the years creating panorama photos and how easy it is now to do it with a smartphone.

Enigma

A couple of weeks ago I visited the Military History Museum in Gatow. Among many interesting things, I saw this, an Enigma machine from WWII. Quite appropriate, as these days I am participating in the GEeks Go for #CC9900. I have cracked the first two challenges, but the third and final one is proving to be quite hard.

Introducing the PiMoodBox

In this post I´m going to give some details of a small project using the Raspberry Pi. Maybe not a step by step tutorial but detailed enough for anybody to replicate with very little effort, including a link to a github repository with the source code.

Some airports have had, for quite a while, feedback systems that allow them to collect information on how a particular process is perceived by the public. I have seen this in the UK for example after the security check. As you walk into the waiting area, you have several buttons to vote and rate the "security experience". At the beginning of the year I was approached at work to see what it would take to recreate the system to try and measure the mood of the department. Since I had been playing around with the Raspberry Pi for a while doing timelapse videos and playing pong with my children using buttons made with pegs and alufoil, this was clearly within easy reach.

The concept is extremely simple: two buttons and one LED to give feedback to the user, all connected to a raspberry pi running a python script inside a shoe box. I used Fritzing for the schematics below.

Unfortunately I am not allowed to hook up the Raspberry Pi to the wifi on our campus, which would have paved the way for a nice mini IoT project. Instead, I found a bash script written by Andrea Fabrizi that uploads data into Dropbox. The nice thing is that it connects to dropbox using the API and limits access to a given app folder so, even if somebody takes the unattended Raspi and cracks the password, I don´t have to worry about people accessing my account (not that there is a lot in there anyway, I only created it a while ago because it was a required for a MOOC). The first time you use the Dropbox-Uploader, you will have to connect it to your app by entering the app key and secret and later entering an authorization token into the browser. Quite straightforward.

My script runs when the Pi boots up and records the time stamp every time a button is pressed. If both buttons are pressed then the Pi looks for my phone hotspot and, if connected, uploads all existing files to dropbox (note that I actually use the skip function so only new files are uploaded).

If I ever have to do any maintenance or troubleshooting, I can connect to the Pi through ConnectBot (by the way, you may also want to get the Hackers keyboard app). While there is room for further automation and improvement, it was interesting to see how easy it is to go from idea to realization with existing blocks. It sounds very topical, but this was a weekend hack and a lot of the time was getting the holes into the shoe box and soldering the two buttons. I appreciate this is not very sophisticated, but for the amount of effort that it took to put together, it is more than enough. Truth be told, I´m really pleased by how well this has worked for the last seven months, despite a couple of hiccups that have made me realize how even a simple project with the simplest of interfaces still has plenty of issues to iron out before scaling up.

I participated recently in a meetup organised by a colleague and prepared the snapshots above, so I thought I may as well share this with the raspberry pi community.

Erwin Schrödinger Zentrum

I would have loved to have this library close by in the mid and late nineties.

Españoles en Berlín entre 2007 y 2014

El servicio estadístico de Berlin-Brandenburg tiene datos de varios años y otras muchas variables. Este gráfico muestra los datos de diciembre de 2007, 2009, 2011 y 2014.

Si no puedes interactuar con el gráfico, prueba aquí

Españoles en Alemania: Berlín, barrio a barrio

El servicio estadístico de Berlin-Brandenburg tiene datos muy detallados de los residentes, por ejemplo los españoles residentes en cada barrio e, incluso, casi calle a calle (los datos están divididos en unas 450 zonas, incluso más detallado que por el código postal. Creo que estos datos no incluyen a los que tienen la doble nacionalidad, que posiblemente aparecen como alemanes.

Datos de diciembre de 2014	Españoles
Charlottenburg-Wilmersdorf	1690
Friedrichshain-Kreuzberg	2869
Lichtenberg	356
Marzahn-Hellersdorf	101
Mitte	2694
Neukölln	1658
Pankow	1866
Reinickendorf	304
Spandau	254
Steglitz-Zehlendorf	713
Tempelhof-Schöneberg	1018
Treptow-Köpenick	244
Grand Total	13767

He creado un mapa interactivo, así que pinchando en cada círculo aparecen los cuatro niveles organizativos en los que están disponibles los datos y el número de españoles inscritos (este número no es el número de inscritos en el consulado, sino en el registro alemán)

Sugar 3:0 Diet

Lange Nacht der Wissenschaften 2016

La semana pasada estuve varias horas en la Freie Universität Berlin con motivo de la LNDW2016.  Llegamos a casa cerca de las once y media de la noche, y la verdad es que no me habría importado estar un rato más porque nos quedaron un montón de cosas por ver.
Por otra parte, quería ver la biblioteca de filología desde hace tiempo y nunca sacaba un rato para ir a echar un vistazo. Durante la carrera pasé muchísimas horas en diversas bibliotecas y me encantan. Ésta, de Norman Foster, realmente es atractiva por fuera y por dentro.

View from the PanAm Lounge

Gothic Festival 2016

This is the second time that I happen to be in Leipzig during the Gothic Festival, always a feast for the eyes. Kudos to all participants for their tremendous efforts, as the costumes are really stunning. Many people were taking photos like mad, some times flocking in front of couples or groups as if it was a photo call. I decided to do it in a slightly different way, so I entered a McPaper and bought a set of cards. Besides asking for permission to take a photo, I asked if they would like to have a copy. A few took me on my offer and wrote their e-mails on a card for me to send a link with the photos, which I have already done. I hope they like them. They put a lot of effort on their appearance, so it felt appropriate to give something back in return.

The weather could have been definitely better, with scattered rain throughout the weekend and some impressive fast moving clouds. But of course, this is a gothic festical, you don´t get to choose the weather, just your clothing...

Playground

Any place is a good place to read a book...

Españoles en Alemania: Muchos más datos (2015)

El servicio estadístico alemán ha publicado recientemente el informe sobre población extranjera en Alemania con los datos de diciembre de 2015. Siguiendo con previas entradas para los años 2013 y 2014, aquí va la entrada analizando los datos actuales de los españoles.

La tendencia sigue igual, es decir, el número total de españoles en Alemania sigue incrementándose, aunque el ritmo ha decrecido un poco. En los doce meses de 2015 la población española en Alemania creció en nueve mil personas hasta las 156,000, frente al incremento de once mil en 2014. El número de españoles ha aumentado en todos los estados federados, pero los mayores incrementos se han dado en Nordrhein-Westfalen (2,500), Berlín y Baviera (aproximadamente 1,300 en cada uno). Éstas son sólo las cifras absolutas ya que hay muchas altas y bajas, como demuestra que el número de nuevos españoles inscritos en 2015 es cerca de 22.000 (incluyendo casi 900 nacimientos), mientras que el número de bajas ascienden a 13,000 (incluyendo 650 muertes).

Por género, la diferencia entre hombres y mujeres ha aumentado respecto al año anterior y el 2015 acabó con 5,500 mujeres residentes  menos que hombres (frente a una diferencia de 4,500 un año antes).

Por edades, ya hay casi trece mil niños menores de quince años, lo que significa un aumento importante  respecto al año anterior cuando sólo eran un poco más de diez mil y nos llegaban a ocho mil a finales de 2013. ¿Significa esto que las familias se están reagrupando en Alemania? ¿Se está tomando alguna medida para apoyar este segmento, tales como potenciar las Aulas de Lengua y Cultura Españolas? Si alguien tiene información al respecto, que la comparta en los comentarios. Por cierto, de estos trece mil niños, unos tres mil han nacido en Alemania.

La distribución por estados:

	2015			Delta
	Total	Hombres	Mujeres	Total	Hombres	Mujeres
Baden-Württemberg	24817	12789	12028	1041	588	453
Bayern	21289	10885	10404	1262	611	651
Berlin	8442	4142	4300	1314	706	608
Brandenburg	872	481	391	117	92	25
Bremen	2096	1116	980	51	37	14
Hamburg	5746	2910	2836	316	137	179
Hessen	24261	12710	11551	959	566	393
Mecklenburg-Vorpommern	773	423	350	103	53	50
Niedersachsen	11532	6074	5458	590	322	268
Nordrhein-Westfalen	43898	22754	21144	2518	1440	1078
Rheinland-Pfalz	5347	2769	2578	248	189	59
Saarland	962	443	519	65	35	30
Sachsen	1797	964	833	192	114	78
Sachsen-Anhalt	519	303	216	111	83	28
Schleswig-Holstein	2715	1437	1278	69	1	68
Thüringen	852	528	324	116	79	37

Y a continuación la serie de gráficos similares a los de las entradas anteriores

Españoles en Alemania por distrito (Landkreis)

Unos conocidos me comentaron la Open Knowledge Foundation y, curioseando, acabé en un página que facilita hacer peticiones de datos a las autoridades alemanas, FragdenStaat.de. Dado que hacía poco que había estado curioseando en el instituto alemán de estadística y sólo había encontrado datos agregados, decidí pedirles datos más detallados, como por ejemplo divididos edad y por código postal o, al menos, por distrito para ver donde están mayoritariamente los niños y donde se pueden crear nuevas Aulas de Lengua y Cultura Españolas.

Eso fue un domingo por la noche. Al día siguiente por la mañana recibí el acuse de recibo y un comentario muy amable dándome la referencia en caso de que tuviera que ponerme en contacto con ellos. Pero en menos de dos semanas el Statistisches Bundesamt se han vuelto a poner en contacto conmigo con los datos, que si bien no son tan detallados como yo pedí, al menos son gratis. Más la oferta de poder pedir más detalles si estoy dispuesto a pagar. Los datos tienen copyright pero se pueden reproducir citando la fuente (ver nota al final de la entrada)

El mapa muestra los distritos donde hay censados españoles, cuantos son mujeres y cuantos hombres. En realidad yo quería hacer un "cloropleth map" (que por cierto no sé como se dice en español), pero últimamente no tengo tiempo para florituras.

I recently got to know about the Open Knowledge Foundation through some friends and after spending some time in this page, I landed on FragdenStaat.de, which allows anyone to make a request for the authorities to provide pretty much any kind of data. Since I had already been looking at the stats department a few days ago, I decided to ask for more detailed data (I only got aggregated data, without geographical information). I asked for data split by age and post code or, at the very least, by district, so that I could figure out which areas have a better chance of establishing new Aulas de Lengua y Cultura Españolas and this is what I got.

I know this is a bit of a topic, but I made the request on a Sunday evening and got the initial reply message on that very same Monday morning. And then a couple of weeks later I got data by district and gender from the Statistisches Bundesamt (see copyright notice below) and a link to where I can find even more data. And all of this in the most polite way (apart from being free). Definitely a place to go back to in the future.

The map above shows where Spaniards live in Germany (as of Dec, 2014). I actually wanted to do a cloropleth map, but I don´t have much time lately and this was the lazy-man option. Unfortunately published heat maps are not interactive, so below is a screen snapshot.

Datenquelle: Ausländerzentralregister (C)opyright Statistisches Bundesamt, Wiesbaden 2016 Vervielfältigung und Verbreitung, auch auszugsweise, mit Quellenangabe gestattet.
Stand: 03.03.2016 / 10:03:33
Mapa: Elaboración propia

Serenity in a bus stop

For months I´ve had in my mind a very similar photo to this one, somebody reading a book on a deserted bus stop, preferably with heavy rain or snow. One thing I was not sure how to deal with was the big ads in the bus stops, which could end up being quite distracting for the final composition of the scene. This evening, as I was driving back from work, I spotted this lady reading a book while waiting for the bus. I parked and walked back a bit to take this shot from across the street with my phone. Everything happened in a few seconds and as I was driving away, the next bus arrived.

Aulas de Lengua y Cultura Españolas 2016

Las Aulas de Lengua y Cultura Españolas ofrecen clases complementarias de lengua materna a aquellos alumnos cuya nacionalidad o la de sus padres es o haya sido la española. Para el curso 2016-2017 el plazo de matrícula es del 25 de febrero al 8 de abril de 2016.

Toda la información se puede obtener aquí. El año pasado hice un mapa interactivo y este año lo he actualizado con la lista de escuelas. Algunas escuelas han cambiado de ubicación y bastantes parece que han desaparecido, como por ejemplo las de Braunschweig y Mönchengladbach en la Agrupación de Hamburgo y varias otras en las agrupaciones de Mannheim y Stuttgart.

Lo que decía el año pasado sobre el riesgo de cierre se ha cumplido, así que es importante difundir lo más posible los plazos de matrícula, y recordad que puede abrirse una nueva aula a partir de 14 inscripciones. Los detalles están en el Artículo 18 del BOE-A-2010-18555

• Hoja informativa 2016-2017
• Relación de aulas del presente curso
• Impreso de solicitud de continuidad
• Impreso de solicitud de nueva inscripción
• Resolución de 3 de marzo de 2015 de la Secretaría de Estado de Educación y Formación Profesional

Actualización 23 de febrero: El aula de Neuss pasa a Düsseldorf e inclusión del enlace a la resolución que regula ALCE.

Información adicional
Hamburgo: Mapa actualizado el 7 de marzo con días, horarios y ubicación en el edificio
Mannheim: Mapa no actualizado con días, horarios y niveles
Stuttgart: Mapa no actualizado con días, horarios y niveles

Españoles en Alemania: Muchos más datos (2014)

Escuchando un podcast de Onda Cero esta semana referido a una nueva serie de televesión, "Buscando el norte", sobre dos hermanos que se vienen a Berlín, escuché atónito como el periodista de turno decía tan tranquilo que a día de hoy en Berlín hay cuarenta mil españoles. Sin entrar a debatir las razones que llevan a alguien a decir en público semejante majadería, he pensado que estaría bien actualizar esta otra de hace un año. Está basada en el mismo informe del servicio estadístico alemán sobre los extranjeros en Alemania, actualizando los gráficos con los últimos datos disponibles (diciembre de 2014, creo que los datos de 2015 saldrán pronto).

La tendencia sigue igual, es decir el número de españoles en Alemania sigue incrementándose. En los doce meses de 2014 la población española creció en once mil personas. Por cierto que el número de españoles nacidos en Alemania también creció.

A 31 de diciembre de 2014, había 146846 españoles registrados, con unos 4500 hombres más que mujeres. El resto de los gráficos de esta entrada se refieren a los números de finales de 2014.

El gráfico por edades muestra una estructura similar, pero el número de niños hasta quince años ha crecido un 30% respecto a los datos del 2013 y ya eran casi diez mil quinientos. 

En cuanto a los datos del tiempo en Alemania, la estructura es similar a la del año pasado. La media de estancia cae un poco, de 22,6 años a 21,3 años, debido a los recién llegados.
 

En cuanto al estado civil, por una parte aumentan el número de solteros y desciende el de solteras y crece el de casados y casadas. Me pregunto si esto indica que las familias se están reagrupando, es decir, en algunos casos los hombre se vienen a Alemania y cuando ya tienen una situación estable se vienen las mujeres y los niños. O quizás, dado que el número de casados con alemanes permanece bastante estable, el significado es que los españoles tienden casarse entre sí includo cuando están en el extranjero.

Mirando las altas y las bajas, se ve el ya mencionado aumento de niños hasta quince años.

Y de los que se dieron de baja, un aumento de casi el cincuanta por ciento respecto de los que se fueron en 2013 para los que se van después de menos de cuatro años en Alemania. 

Historias de Alcalá

Para los que estén interesados en historias y anécdotas relacionadas con Alcalá de Henares, merece la pena una visita al blog de Historias de Alcalá de José Antonio Perálvarez Santaella.