Montag, 3. Dezember 2012

PublicTransport 0.11 alpha available

Time for an update about the progress made in the PublicTransport project!
I just tagged version 0.11 alpha in GIT and updated the page on kde-look.org (with an installer script).
There are many changes and new features, in short:
Supported countries in Europe
(not shown: India, Japan, New Zealand, USA)
  • GTFS feeds are now supported along with GTFS-realtime
  • The Swiss public transport API is supported by a provider script
  • There is now a script API for provider scripts (eg. network requests can be started from scripts)
  • Stable data formats for HAFAS providers (with a shared script code base, removed most HTML layout dependency)
  • Scripts now run in their own threads using ThreadWeaver
  • Less network traffic, improved performance
  • More supported providers in more countries
  • Much improved TimetableMate (for creating and testing provider plugins)
  • A Plasma::Service for timetable data sources, can be used to request manual updates, additional data or earlier/later items
  • Another Plasma::Service to handle the GTFS database, eg. to import GTFS feeds
  • Popup maps for stop input fields (if supported by the provider)
  • Improved journey search and more details in journey view

Three new HAFAS providers have been added (since it is now very easy, see below): de_vbb, ie_eireann and no_dri (adding support for Norway and Ireland).

With the new GTFS providers the list of supported countries extends to (in alphabetical order):
Austria, Belgium, Denmark, France, Great Britain, Hungary, India, Ireland, Italy, Japan, New Zealand, Norway, Poland, Spain, Sweden, Switzerland, Ukraine, USA (some states).

Now some more details about what has changed, with quite a few screenshots:

Open Data - GTFS

GTFS (General Transit Feed Specification) is now supported. GTFS feeds can be downloaded and imported into an SQLite database using a new Plasma::Service.
GTFS providers can then be used offline, realtime data can be added while being online using GTFS-realtime. The database takes some disk space (around 50 - 500 MB depending on the area covered), but it works almost instantly.
Many new countries are now supported through GTFS: USA, Spain, France, Great Britain, Hungary, India, Japan, New Zealand, Ukraine, Poland. And there are many more GTFS feeds available that wait to be included by adding provider description files (*.pts) for them. TimetableMate can now also be used to create GTFS provider plugins.
GTFS feed import in progress
Provider information dialog showing a GTFS provider 
with it's current state

Open Data - Swiss public transport API

Beside the existing provider for switzerland (ch_sbb, uses the undocumented HAFAS API), there is now an open timetable data source for switzerland:  http://transport.opendata.ch (thanks to Mario Fux for the link). It works like the other scripted providers, but it has a simple API and it's documentation is publicly available. This of course makes it much easier to use such data. Hopefully more timetable data will be opened like this in the future.
The applet using transport.opendata.ch

New Provider Script API using QtScript

Kross is no longer used for script execution, instead QtScript gets now used. That made it easy to create an API for the scripts and to add debugging features to TimetableMate.
The new script API has been created to provide the scripts with some more possibilities. Scripts can now start network requests themselves using the network object, store some data using the storage object, publish found timetable items using the result object or use the helper object for some more functions, mostly for parsing. The helper object also includes functions to easily parse HTML without the need to think about the common pitfalls. A provider object includes the properties of the ServiceProviderData class, ie. data from the *.pts file describing the provider (name, author, version, homepage, etc.). Instead of using strings, scripts can now use enumerables eg. for vehicle types or provider features.
Decoding of documents downloaded using the network object is also done by the scripts using helper.decode(). The documents are available as QByteArray otherwise and can be read using a data stream: DataStream objects can be created by scripts, which wrap QDataStream and provide functions like readInt16(). QDataStream only provides operator >>, which cannot be used from QtScript.
Other script languages can still be used through Kross from inside QtScript, but this is untested.

Improvements for HAFAS Providers

For providers using HAFAS software (eg. de_db, ch_sbb, at_oebb, etc.), there is now a new flexible base script. It can be included using include() and is able to read some HAFAS formats like XML for departures/arrivals/route data and a binary format for journeys. Unsurprisingly the XML documents are much shorter than the HTML documents that were used before, parsing them is easier and more efficient and it is no longer dependent on the HTML layout of the provider web site. It also removes the need to write new parsers for each HTML layout of each supported HAFAS provider, because the XML format is always the same. With less data to be downloaded the engine is now also much faster for HAFAS providers. Stop suggestions are retrieved in JSON format, which now gets parsed using the builtin JavaScript function JSON.parse() for best performance.

A "HAFAS-XML" departure document of size 0.9 KB (uncompressed 4.2 KB) replaces an HTML document with the same information in it, but of size 22.6 KB (uncompressed 150 KB). This means that 25 times less data needs to be downloaded :)
The "HAFAS-binary" format for journeys is only 2 KB, the HTML version is 69 KB (both compressed), this means over 34 times less data to be downloaded.

Some providers also offer route data in an XML format, others in a Lynx text version or mobile/desktop HTML. Route data can now be downloaded later as additional timetable data using a new Plasma service for the departure data source. Unfortunately the URLs to the route data documents (traininfo.exe) are only available from HTML departure sources, therefore a mobile version of the HTML departure document gets parsed for the URLs and then cached for following additional data requests.
The applet got a new setting (in the "Advanced" tab), that controls when additional data should be requested. By default additional data gets requested when a departure gets unfolded for the first time. There is also the possibility to directly request all route data for all received departures. This makes sense when route data gets used in filters, ie. the "Via" or "Next Stop" filters. These filters cannot work correctly until the route data is available. Otherwise this setting should not be changed to not produce too much network traffic and waste CPU time.

The de_rmv script also uses the new HAFAS base script, but it uses another XML format (maybe only used by de_rmv). This XML format is somewhat longer but also includes route data. Actually the format used for de_rmv is the same that was already used in previous versions. But I ported the C++ code to QtScript (very easy with the qt.xml extension), it does no longer use an own provider type C++ class.

The it_cup2000 script is another special user of the HAFAS base script, it implements it's own parser functions for HTML departures/arrivals/stop suggestions.
Back to live is us_septa, also a provider using HAFAS software. Departures/arrivals are still only available as HTML, but journeys are available in the binary format.

Removed Providers

EFA also uses an undocumented API like HAFAS. EFA scripts parsing HTML that do not work any longer are removed for now. A base script class for EFA providers could be written just like it's done with HAFAS providers (base_hafas.js, base_hafas_timetable.js, etc.). TimetableMate can be used for this task. This would make it easy to add support for some more providers (mostly in germany).

The slovak provider sk_imhd was removed because of much changed HTML structure, fr_gares, sk_atlas were also removed because they no longer work, de_vvs is waiting for an EFA base script to be added again. If you miss one of these providers, you could use TimetableMate to fix the scripts, add yourself as author and send me the fixed script.

Improved Updates

The "Update Timetable" action of the applet works again. This is based on work in the engine and uses the "requestUpdate" operation of the timetable service.
The engine now stores two QDateTime values in timetable data sources: "nextAutomaticUpdate" and "minManualUpdateTime". Automatic updates are done by the engine at the given time. Before "minManualUpdateTime" has passed all (manual) update requests are rejected/blocked by the engine. The applet uses these values to disable the update action while the engine would reject the update and to show the next automatic update time to the user (in the tooltip of the bottom label).

The applet with disabled update action,
showing the next automatic update

Load Additional Timetable Data Later

If some timetable data is not available without more work (eg. downloading more documents) providers can now provide such data later as additional timetable data per departure/arrival. HAFAS providers use this for route data, which is not directly available in their departure XML-documents.
The applet by default requests additional data when a departure gets unfolded for the first time. It is also possible to never request additional data or to directly request additional data for all new departures.
The applet configuration showing additional data options

Marble Public Transport Stop Map

Marble gets now used to show public transport stops in a map (if Marble is available at build time). Such a map gets shown in a popup for StopLineEdit of libpublictransporthelper. Stops can be clicked in the map to use it's name in the StopLineEdit. Providers can now provide longitude/latitude values for each stop they read in getStopSuggestions(), these values are needed for the map to work. For providers that are able to get stops by geological position (eg. most HAFAS providers), the stops of the currently shown region of the map are automatically loaded.
The stop coordinates of the providers are now also used by the applet instead of using the osm engine. This makes the "Show in Map" action work reliably. The runner also uses this for the "run action" of stop suggestion results and shows found stops in Marble.

Applet configuration dialog showing a popup map

Improved Journey Search

Searching for journeys also got some new features. Favorite/recent journey searches with alias names can be used and quickly be executed using the "Quick Journey Search" button. To get earlier or later journeys one click is enough. And for HAFAS providers there are now more details, every single intermediate stop can be shown (hidden by default, showing only connecting stops).
New quick journey search button
with favorite/recent journey searches
Favorite/recent journey searches
can be edited in a dialog

Earlier and later journeys can easily be requested
More details for journeys of HAFAS providers,
showing all intermediate stops

 TimetableMate

Writing new provider scripts gets a lot easier with TimetableMate 0.3. It can now open multiple projects, debug scripts, got some docks (projects, variables, backtrace, breakpoints, console, output, web inspector, network monitor, test, documentation), there are multiple automatic tests, a project dashboard written in QML, a GTFS database viewer etc. There is also a new icon for it, a modified KDevelop-icon. TimetableMate now looks more like KDevelop.
The QScriptEngineDebugger class is not used, instead there is now a new script debugger to better integrate it into TimetableMate. It also allows other special handling for provider scripts.
The documentation shown inside TimetableMate gets generated at build time from some of the source documentation for the data engine.

TimetableMate with the new project dashboard
and the console/projects docks
TimetableMate while debugging,
 showing current variables and some produced output
The new testing features of TimetableMate


Versioning

All this will be available in the next version 0.11. If you wonder what happened to 0.10, it silently got a release tag in GIT, but the provider scripts were mostly no longer working (changed HTML layouts, etc.). I already had new working scripts, but they depended on the new script API and porting them back was not easily possible. But 0.11 will be more resistant to such errors.
Previously I planned to release GTFS in version 0.12, but since GTFS already works quite well and testers are needed, there is no reason to delay it further. Especially because provider types can be disabled at build time now. A 0.12 build without GTFS would have been the same as a 0.11 build.


Test it!


It will still have some bugs, but nothing too big. Most provider plugins should work as expected. All features should also work when supported by the used provider. Please help to make 0.11 stable :)

Kommentare:

  1. The program is looking really good, and I would like to test it. Unfortunately I get an error when I run install.sh: I get to the point where I can choose to install 0.11alpha_(unstable), and then I get:

    ./install.sh: line 1028: [: too many arguments
    >> Release branch "0.11", tag "unstable-0.11-alpha
    unstable-0.10-rc2
    unstable-0.10-rc1
    unstable-0.10-beta8
    unstable-0.10-beta7
    unstable-0.10-beta6
    unstable-0.10-beta5
    stable-0.9
    stable-0.10.1
    stable-0.10
    HEAD"
    Failed to checkout the selected version, exiting now (1).

    I didn't dig into this, because I hoped you would be easily able to tell me what the problem is. I'm running the 64-bit version Kubuntu 12.04.

    AntwortenLöschen
    Antworten
    1. Hi, I guess you haven't explicitly selected the version, eg. using the space key, so that the version gets "checked". It should be selected by default, but it isn't and fails if nothing is selected.

      Löschen
    2. The script is now fixed on kde-look :)

      Löschen
    3. Thank you for the quick answer. It's working nicely now, and I really like it very much. I didn't yet manage to use the runner, though. I can't seem to be able to make it recognize my searches, but that might be a mistake on my side.
      But the plasmaoid is great already. Thank you!

      Löschen
  2. Wow! This is just 100% Pure Awesome! :-) Thank you for such a fantastic and comprehensive application. It's contributions like yours that really make the KDE community such a great place to be :-)

    AntwortenLöschen
  3. Seems strange that the Netherlands is not yet supported, given that the data is most definitely available in the HAFAS system: http://hafas.bene-system.com/bin/query.exe/nn?L=profi (HAFAS for Belgium & Netherlands)

    AntwortenLöschen
    Antworten
    1. Hm.. is the provider for that URL called NMBS/SNCB? It thinks it only has data for Belgium. Some time ago I think it also used the hafas.bene-system.com URL, but there is no XML source available (with L=vs_java3). The new URL is http://hari.b-rail.be/hafas/bin/query.exe. Just try out that provider if it works for Netherlands, too.

      Löschen
  4. This is really incredibly advanced. I wish Plasma Mobile was ready and available on a phone - it'd have a perfect public transport widget ;-)

    AntwortenLöschen