Neubot 0.4.15.5 released http://neubot.org/neubot-0.4.15.5 Version 0.4.15.5 is the latest generally-available stable release of Neubot, distributed for Ubuntu >= 11.04, MacOSX >= 10.6, Windows >= XP SP3, FreeBSD and ArchLinux, as well as in source format. Here's what changed in less than 140 characters: new "raw" test, support for mlab-ns (discovery service) and web100 (tcp stats), more robustness, cleanups and bug fixes. Install, upgrade instructions at <http://neubot.org/download>. Read on for more details. This release contains 122 commits by 1 author. The diffstat says: 180 changed files with 11,176 additions and 4,987 deletions. Here is the list of what changed and why: * new * "raw" transmission test (characterize you connection) This new test characterizes your connection. It connects to a random M-Lab server, estimates latency and measures download goodput at application level. In additions, the test saves snapshots during the test, including goodput information, and TCP variables (using web100, described below). As the name implies, this test does not use any protocol and just transmits "raw" random data. The plan is to analyze and use the results of this test in a paper we're working on. Moreover, we look forward to port innovative code added by this test (such as support for mlab-ns and web100) to other tests. * client for mlab-ns (M-Lab service for discovering servers) M-Lab is deploying its new service to discover servers, named mlab-ns (M-Lab name service). This Neubot release includes a client for such service, which is used to find a random active server for the raw test. Of course, the plan is to modify the other tests to use this service as well. (When mlab-ns will replace the rendezvous with the master server as a discovery mechanism, we won't decommission the master server. Instead, we have plans to use it to pass the client useful information, such as the estimated location, a list of known-up servers, information on what tests it should perform -in order of importance- given its geographic area, results of nearby Neubots, etc.) * web100 module (to collect TCP stats on M-Lab) Web100 is a Linux kernel patch, installed on all M-Lab servers, that allows to take snapshots of the internal state of TCP. This Neubot release includes code for taking Web100 snapshots, and, in turn, the code is used by the new raw test. It goes without saying that the plan is to modify the other tests and use Web100 for them as well. Since Web100 snapshots are quite big, in this release the raw client does not save them on the local sqlite3 database. This is just a temporary measure, until we decide what to do (it's likely that will add a knob that allows the user to specify what Web100 variables must be saved, along with a reasonable default, which saves just the most relevant ones). * more robustness * with respect to server discovery failure (by using a static table) Starting from this release, should the discovery procedure fail to select a server, Neubot will select one random server from the table of servers know to be up at release time. Of course, this strategy could be improved a lot: e.g., Neubot may periodically download an updated table of know-up servers. Still, the bottom line is that now, if the master server has a boo boo, Neubots around the world can still run useful tests (yes, with random servers, which is clearly not optimal but much better than nothing). (The term "discovery procedure" is new in Neubot jargon, and includes both the traditional "rendezvous" with the master server and the new possibility of querying mlab's name service (mlab-ns) for servers.) * in win32 updater (by emulating Twisted's deferreds) With this change in, the win32 updater should be more robust with respect to unexpected failures. This is done thanks to a piece of code that emulates a subset of Twisted's deferred: a feature that allows to register in advance callbacks to be invoked later when an event is fired. One nice property of deferreds is that, if the callback raises an exception, this is catched by the deferred machinery and routed to the callback assigned to exceptions, if any. Otherwise, an error message is printed. (The emulation is likely not complete, since I wrote it by just skimming through Twisted documentation and figuring out how to implement it. It probably does not behave same-as Twisted implementation, and is quite for sure less general and flexible. Anyway, this is not important, since I was just interested in emulating the bare minimum to make Neubot more robust.) * more generally in the code (by emulating Twisted's deferred) As explained above, deferreds allow the programmer to redirect unhandled exceptions in callbacks to other callbacks. So I have refactored certain portions of the code to make sure that, even if an unexpected error occurs, certain critical events, like the one that schedules the next test, are always scheduled. * better * release process (more agile, no more release candidates) Starting from this release, there are no more release candidate releases (the ones with "-rcX" appended). Instead, now a new version starts at, e.g. 0.4.15.0, and is tested. If there seem to be no bugs, it is released as-is. Otherwise bugs are fixed and 0.4.15.1 is prepared. (This explains why this version of Neubot is 0.4.15.5, I indeed needed to iterate five times before I was satisfied with the final result.) This is of course more agile, since: (a) you don't need to make a stable release once you find that, e.g. 0.4.15-rc5 is good enough; (b) if you find that a supposedly good version is not so good, you just need to increment the last number and iterate (whereas before you needed to "burn" that version number and initiate a new stable cycle with an incremented version number -- and this explains why 0.4.13 was never released :^). * python3 support I'm slowly making sure that all Neubot modules are compatible with the new major version of Python, so that we'll be ready when the 2.x line is decommissioned by major operating systems distributions (probably the same day we switch to IPv6). * more cleanups and bug fixes Full change log available here: https://raw.github.com/neubot/neubot/0.4.15.5/ChangeLog Diff since previous version here: https://github.com/neubot/neubot/compare/0.4.14...0.4.15.5 Commit history here: https://github.com/neubot/neubot/commits/0.4.15.5 SHA256sums here: https://github.com/neubot/neubot/wiki/sha256 Errata here: https://github.com/neubot/neubot/wiki/errata Data here: http://www.neubot.org/data http://data.neubot.org/ Roadmap here: https://github.com/neubot/neubot/wiki/roadmap -- Simone Basso http://nexa.polito.it/people/sbasso bassosimone@gmail.com simone.basso@polito.it