libight: a new hope

This blog post is part of a series of three regarding ooniprobe-mobile.

On May 28th 2014, Arturo Filastò and I were sitting in a pub in Turin drinking Weißbier. He had been invited to present his Open Observatory of Network Interference (OONI) project at the 21st Nexa Center's lunch seminar, and he also gave a talk in our Rivoluzione Digitale course.


The pub in which we were, The Isle of Skye Pub, probably one of the best in Turin.

Two months earlier, I completed my PhD at Politecnico di Torino, and when I was discussing with Arturo, sadly my daily job was more focused on finalizing my doctoral thesis than on writing software.

I have to admit that the months during which I was editing my doctoral thesis were amongst the least pleasant months of my life, because I really do not like dealing with such large bloats of text (unless it's code, ça va sans dire).

Yet, as all the kids born in Liguria ought to know, from diamonds it stems nothing, from manure bloom flowers: while writing the doctoral thesis I started cheating and writing not about the Neubot that had been implemented, but about the one I'd have liked to see. And this was the beginning of libneubot.


Fabrizio De André, the late Italian singer and poet I cited above, is really one of my favorite ones.
Here's his cover of Bob Dylan's Desolation Row.

Libneubot? Well, even earlier than writing the thesis, in Autumn '13, the importance of working on a mobile version of Neubot was clear to me — and we also did receive specific funding, later in '15 used to cover my salary.

And, to address this mobile need, the idea I had was indeed libneubot. A C library meant to expose specific functionalities to Neubot, written in Python, and, in the future, to an Android app, written in Java.

Now, notwithstanding what some academic types will tell you, in the real world (TM), it does not suffice to have a neat idea to thrive, as Edison famously codified. Luckily, at the time I was assisted by a very brilliant student, Alessandro Quaranta, and together we were adding the missing 99% of transpiration to reach a workable product.

And, in that nice spring evening of '14, libneubot was indeed one of the topics I was discussing with Arturo, the other topic being MORFEO.


There are lots of known facts (and also some alt-facts) surrounding the different approach of Tesla and Edison. I don't feel qualified enough to enter into the debate, but, for what I have understood, I'd say I'm more, with all due respect, an Edison-like type that transpires a lot than a Tesla-like type that relies on theory.

MORFEO? Yes, MORFEO. Or, Morpheus, in English (even though it would be more fair to say that that is ancient Greek). Anyhow, in our case, the MORFEO we were talking about was not the ancient god of dreams, but rather a subproject of EINS for which we were awarded funding.

EINS. You seem confused at this point. But don't panic. EINS was a European project that the Nexa Center was participating in. Sometime around the spring of '13, Federico Morando and Raimondo Iemma (my then fellow researchers at Nexa Center) suggested that I write a proposal for an upcoming open call within the EINS project. They said it would've been nice to partner with someone else, and I proposed Arturo, who I had met in Autumn '11 during the EU hackathon, and with whom I subsequently had a number of technical discussions centered around our own projects, respectively Neubot and OONI, which shared several similar design traits and challenges, even though they were focusing on different domains (performance and censorship). Anyhow, to cut it short, MORFEO proposed to add censorship measurements (assisted by Arturo) to Neubot, but this was not awarded for a long time, to the point that I forgot it existed, and was finally surprisingly awarded in spring '14.


Lest we forget, the days of the EU hackathon were also very dramatic days for my country amidst the economic crisis. Berlusconi was stepping down and I recall spending the night hacking and constantly refreshing the browser for more news. This is the front page of a prominent Italian news paper, il Sole 24 ore, the day after the EU hackathon. Historians will tell us if the involved parties were acting in the national interest. Despite many hours reading and reflecting on the subject, I still feel like the whole matter was very complicated.

So, now we've reached the bottom of it, you have perfect information, and we can flash forward to that May '14 evening, where we were in fact discussing about what to actually do inside of MORFEO. And, it turned out that we were both interested in implementing measurements on mobile, but we were not able to do that so far. So, we were like: wait a minute, why don't we do this within the context of MORFEO?

The idea was promising, especially because it was clear to both of us that the effort of porting our measurement methodologies to mobile was complex enough that a partnership between two different teams was in order to guarantee increased chances of success.

Thus, we started discussing the pros and cons, and at a certain point I threw libneubot at the discussion, saying something like: well, a student and I have already a prototype, if you wish we can start looking into it, to move from discussion to real feasibility assessment.


Using libneubot was pretty much like realizing that the synthesizer was the sound of the future. When the discussion started, I did not really planned on proposing using libneubot, but it quickly become very obvious.

The legend says that we reached a point where we were either ordering another whisky or heading back to my home to hack.

Read more...

That Neubot update due one year ago (eh, ...)

That feeling when you discover you have not updated the blog at neubot.org for more than one year: you cleary feel very miserable.

Being a professional procrastinator, then, I then spent some more time asking myself why I did not update it in such a long time.


Who said procrastination?

I could now list all the reasons I found, including that I do not like working with Drupal, but I fear this may be a bad use of your and my time, as it would certainly be more interesting to write about what we did.

And so, here we are, Neubot update: October 2015 — January 2017!

During this period of time, and especially since we were awarded an unrestricted research grant from Google Research, I have mainly been working on measurement-kit. With (in alphabetical order) Davide Allavena, Carmine D'Amico, Arturo Filasto', Antonio Langiu, Lorenzo Primiterra, and others, we have been steadily improving the library.


measurement-kit logo

The most exciting features developed have been: the implementation of the network diagnostic tool (NDT) network performance measurement methodology and the work done to support the first implementation of OONI for mobile devices (a fact that is already hitting the news even though we've not yet pushed the publish-the-app-on-all-stores button).

Read more...

Received unrestricted grant to finance Measurement Kit!

Another short post to share epic news: we recently received an unrestricted grant of 50,000 USD from Google to support the development of Measurement Kit! Specifically, with "support" here we mean paying salary of persons involved (chiefly me, but also others to a smaller extent) and also travels to/from related hackathons.

We intend to use this grant to implement the testing methodology of the Network Diagnostic Tool for Measurement Kit. We are also considering what other network performance methodologies it would be optimal to implement and discussing on how to enable the Neubot software to run those tests.

Read more...

neubot-runtime 0.4.17.0 and neubot-server 0.5.0 released

Today I am pleased to announce that we have recently released two hopefully stable versions of neubot-runtime and neubot-server. These two releases come after many (perhaps too many) months of development aiming at moving forward with Neubot.

I already mentioned before these efforts that basically strive to split Neubot client and Neubot server to allow to develop them independently. In this context, neubot-runtime is the shared piece of software containing the basic classes used by both the client and the server of Neubot. The latest (and first) stable release of neubot-runtime, v0.4.17.0, contains code that was tested and known to run with neubot-server. Not all the code inside neubot-runtime is already compatible with python3, but we're working in this direction.

Instead, neubot-server contains (as the name implies) only Neubot server code. This first release, v0.5.0.0, is based on neubot-runtime 0.4.17.0. One notable feature that we introduced is that now all tests are independent modules loaded at runtime. Work in this direction was already initiated during the release process of Neubot 0.4.16.

These two new releases are the result of the efforts of me and Davide Allavena.

We will now start deploying the new neubot-server on mlab test machines and, when we will be satisfied, we will deploy it (or a subsequent version of it) on all mlab machines, for all clients to use it.

This piece of work (moving forward on the server side) allows us to concentrate now on the client side, for which we tentatively scheduled a new release (0.4.17.0) for end of March.

Read more...

MeasurementKit 0.1.0 and 0.1.1 released

A short post to say that we've recently made available the first and second stable releases of MeasurementKit, the network measurements library for mobile devices to which we've been working for more than two years now!

The first release (0.1.0 released on December 16, 2015) focused on providing an initial stable API for running OONI tests and iOS and Android support.

The second release (0.1.1 released on January 13, 2016) added some bug fixes and improvements.

Other related repositories are the iOS app repository, the Android app repository, and the Android JNI repository.

Read more...

[ITA] Slide Conferenza Nexa 2015

Slide che ho usato nella mia presentazione durante la 7a Conferenza Nexa su Internet e Società:

Versione PDF delle slide.

Read more...

"Network measurements from the edges"

As you probably have noticed, I changed the tagline of the website. From "the network neutrality bot" to "network measurements from the edges". Oh, and I also aggressively edited the content of the website. To the point that most content now sleeps with the fishes.

Such radical changes reflect profound transformations that have been happening in the last 1.5 years.

Basically, at a certain point the focus moved away from the Neubot software tool alone, and started to be measuring the internet from the edges with whatever mean, provided that it was adequate for the task.

This happened, I guess, during the MorFEO project (i.e. in 2014).

It's ironic that in the same post we have a project called Morfeo (i.e. Morpheus) and content sleeping with the fishes, don't you think? (Source: God Father The Game (Wikia)).

Today, preparing my talk for the (and my!) seventh Nexa conference, I reckoned that the time had come to finally attempt to sync up this website with reality.

So, what did I change? Well, basically I kept the original skeleton, but in many cases I've significantly reduced the bloat. And, most importantly, in all places I've clarified that Neubot is now the name of Nexa Center's project that R&D network measurement tools, one of which happens to be called neubot. (This was also clarified in Nexa's website and I also opened an helper GitHub repository to track TODOs.)

As said, the new content now reflects reality. In the last two years I have been working mostly on what was initially called libneubot, then libight, and finally MeasurementKit. This product is a portable network measurement library meant to support network-performance and internet-censorship measurements from mobile devices (iOS and Android). It is a new product because Neubot, as a tool written in Python, was not adequate to perform mobile measurements -- a thing that we're interested to do.

And yes this library, MeasurementKit, was significantly shaped by MorFEO. In fact MeasurementKit is jointly developed with the OONI project, which was partner of MorFEO.

Does this mean that Neubot (as a software) will die? No. I am currently working with Davide (another Neubot core developer) and Carmine to, let's say, extract the code that used to be Neubot from the ball of mud that Neubot become during the last two years of my doctorate. We're making a good job at that. We already have a quasi-stable implementation of the Neubot core classes (the "runtime"), we made this portable to Python3, we separated client code and server code, and now server code kinda works using the runtime.

While writing this blog post I was listening to this song. At a certain point it says:
"it shouldn't be impossible to use scissors if this is exactly what you want".

The plan for the next one year or so is to focus on MeasurementKit adding cool features and to maintain Neubot and engineer its code to make it simple again to do useful stuff with it. This seems to me at the moment the most rational way to move forward.

And possibly that's what I'll say tomorrow during my lightning talk at Nexa Conference. Now closing this log and focusing on the slides deck. I will likely have more to say in the near future.

Read more...

Neubot report: July - September 2015

Most hacking during these three months period concentrated on MeasurementKit (mainly pre-release cleanups) and NetworkMeter. This is a rather technical post. But, before entering into the details of the merged pull requests, here's some details for humans. I had a great summer vacation time, spanning three weeks all along August, in my home city (Bordighera) and nearby cities (such as Ventimiglia). Of course I never went to the sea, but spent lots of time with friends and dedicated the nights (after 2:00 AM) to hacking. The following photo shows the harbor of Bordighera from a small nearby hill (and reminds me of one famous picture by Monet also authored in Bordighera).

A view from Bordighera —Picture by me (CC BY 3.0)

Having said of my vacations, let's go back to business. The following MeasurementKit pull requests were merged:

  1. Teach connection to use custom poller
  2. Register test complete callback along with test
  3. Big rename from libight to measurement_kit
Read more...

Neubot report: April - June 2015

Greetings from the Hacker Standard Time timezone! The greatest change in the April - June 2015 period is that the library that I previously called libight was renamed and now is called measurement-kit. In addition the repository was moved from TheTorProject organization to another organization, called measurement-kit.

In addition to changing names, we did progresses towards releasing the first version of measurement-kit along with the sample applications. The iOS application, in particular, is nearly ready (thanks to Lorenzo Primiterra for helping with that). The Android application instead is not as ready; its code in fact has not been committed to GitHub yet, as we are still working offline.

Among all the measurement-kit changes a notable change was that I coded support for running traceroutes on Android by customizing code contributed by the Portolan project. This was done during a two days of hacking in Pisa, side by side with Valerio Luconi, one the authors of Portolan.

   

Lungarno in Pisa—Pictures by me (CC BY 3.0)

I have also started contributing to network-meter, a shell for running network measurement tools (e.g. Neubot, OONI) and for visualizing their results. This effort is spearheaded by Poly, a OONI volunteer. Also network-meter is on GitHub under the umbrella of the measurement-kit organization.

Filippo Michielon did a mockup of the new Neubot web interface written using Angular.js and D3.js. This could be a good basis for rewriting the web interface with more modern and less-custom tools.

Read more...

[ITA] Presentazione Neubot all'evento Digital Championship di Torino

Il 23 Aprile scorso ho presentato Neubot durante l'evento Digital Championship di Turin. In effetti io non sono un Digital Champions e originariamente infatti avevano invitato a parlare di Neubot Federico Morando che e' il Digital Champion di Novi Ligure. Tuttavia lui non c'era, cosi' alla fine sono andato io.

Durante la mia presentazione ho descritto la storia di Neubot (il focus sulla neutralita' della rete, gli obiettivi di ricerca) e ho parlato degli obiettivi futuri (misure su mobile, empowerment degli utenti via corsi e altri materiale online, scrittura di esperimenti che tengono conto di cio' che e' importante con gli utenti). Qua di seguito il video, che parte proprio nel momento in cui inizio a parlare:

Non ho preso un voto altissimo, ma a fine manifestazione un po' di persone mi hanno comunque fatto i complimenti per il progetto. E' stato anche interessante lo scambio che ho avuto con Marco Boglione (verso la fine del mio intervento) a proposito di open source e modelli di business.

Osservo come al solito con soddisfazione che il tempo che intercorre tra la decisione di postare sul blog di Neubot e il momento in cui posto e' sempre maggiore di due settimane. Alla prossima!

Read more...

Pages
{"cc_re__cc_re__cc_by_3_0": ["div>\n\t\t  </div> <!-- #sidebar-1 -->\n        <!-- /.section, /#sidebar-first -->\n      \n\t  <div class=\"clear\"></div>\n\n     </div> <!-- /.section, /#content -->\n\n\t <div class=\"clear\"></div>\n\n    </div></div> <!-- /#main, /#main-wrapper -->\n\t<hr />\n    <div id=\"footer\"><div class=\"section\">\n        <div class=\"region region-footer\">\n    <div id=\"block-block-6\" class=\"block block-block\">\n\n    \n  <div class=\"content\">\n    <p>Unless otherwise specified, the content of this site is available under the <a href=\"http://creativecommons.org/licenses/by/3.0/\">CC-BY 3.0 Unported</a> license.</p>\n<p>Please, check our <a href=\"/notice\">notice</a> page to find out what content is available under alternate licenses.</p>\n<p>This product includes GeoLite data created by MaxMind, available from <a href=\"http://www.maxmind.com/\">http://www.maxmind.com/</a>.</p>\n<p>Please, check <a href=\"http://www.neubot.org/privacy\">http://www.neubot.org/privacy</a> for the Privacy policy.<br />\nSite Powered by <a href=\"http://drupal.org/\">Drupal</a>. Webmaster: <a href=\"htt"]}