Time for an update about the progress made in the PublicTransport project!
I just tagged version 0.11 alpha in
GIT and updated the page on
kde-look.org (with an installer script).
There are many changes and new features, in short:
 |
Supported countries in Europe
(not shown: India, Japan, New Zealand, USA) |
- GTFS feeds are now supported along with GTFS-realtime
- The Swiss public transport API is supported by a provider script
- There is now a script API for provider scripts (eg. network requests can be started from scripts)
- Stable data formats for HAFAS providers (with a shared script code base, removed most HTML layout dependency)
- Scripts now run in their own threads using ThreadWeaver
- Less network traffic, improved performance
- More supported providers in more countries
- Much improved TimetableMate (for creating and testing provider plugins)
- A Plasma::Service for timetable data sources, can be used to request manual updates, additional data or earlier/later items
- Another Plasma::Service to handle the GTFS database, eg. to import GTFS feeds
- Popup maps for stop input fields (if supported by the provider)
- Improved journey search and more details in journey view
Three new HAFAS providers have been added (since it is now very easy, see below): de_vbb, ie_eireann and no_dri (adding support for Norway and Ireland).
With the new GTFS providers the list of supported countries extends to (in alphabetical order):
Austria, Belgium, Denmark, France, Great Britain, Hungary, India, Ireland, Italy, Japan, New Zealand, Norway, Poland, Spain, Sweden, Switzerland, Ukraine, USA (some states).
Now some more details about what has changed, with quite a few screenshots:
Open Data - GTFS
GTFS (General Transit Feed Specification) is now supported. GTFS feeds can be downloaded and imported into an SQLite database using a new
Plasma::Service.
GTFS providers can then be used offline, realtime data can be added while being online using GTFS-realtime. The database takes some disk space (around 50 - 500 MB depending on the area covered), but it works almost instantly.
Many new countries are now supported through GTFS: USA, Spain, France, Great Britain, Hungary, India, Japan, New Zealand, Ukraine, Poland. And there are many more GTFS feeds available that wait to be included by adding provider description files (*.pts) for them. TimetableMate can now also be used to create GTFS provider plugins.
 |
| GTFS feed import in progress |
 |
Provider information dialog showing a GTFS provider
with it's current state |
Open Data - Swiss public transport API
Beside the existing provider for switzerland (ch_sbb, uses the undocumented HAFAS API), there is now an open timetable data source for switzerland:
http://transport.opendata.ch (thanks to Mario Fux for the link). It works like the other scripted providers, but it has a simple API and it's documentation is publicly available. This of course makes it much easier to use such data. Hopefully more timetable data will be opened like this in the future.
 |
| The applet using transport.opendata.ch |
New Provider Script API using QtScript
Kross is no longer used for script execution, instead QtScript gets now used. That made it easy to create an API for the scripts and to add debugging features to TimetableMate.
The new script API has been created to provide the scripts with some more possibilities. Scripts can now start network requests themselves using the network object, store some data using the storage object, publish found timetable items using the result object or use the helper object for some more functions, mostly for parsing. The helper object also includes functions to easily parse HTML without the need to think about the common pitfalls. A provider object includes the properties of the ServiceProviderData class, ie. data from the *.pts file describing the provider (name, author, version, homepage, etc.). Instead of using strings, scripts can now use enumerables eg. for vehicle types or provider features.
Decoding of documents downloaded using the network object is also done by the scripts using helper.decode(). The documents are available as QByteArray otherwise and can be read using a data stream: DataStream objects can be created by scripts, which wrap QDataStream and provide functions like readInt16(). QDataStream only provides operator >>, which cannot be used from QtScript.
Other script languages can still be used through Kross from inside QtScript, but this is untested.
Improvements for HAFAS Providers
For providers using HAFAS software (eg. de_db, ch_sbb, at_oebb, etc.), there is now a new flexible base script. It can be included using
include() and is able to read some HAFAS formats like XML for departures/arrivals/route data and a binary format for journeys. Unsurprisingly the XML documents are much shorter than the HTML documents that were used before, parsing them is easier and more efficient and it is no longer dependent on the HTML layout of the provider web site. It also removes the need to write new parsers for each HTML layout of each supported HAFAS provider, because the XML format is always the same. With less data to be downloaded the engine is now also much faster for HAFAS providers. Stop suggestions are retrieved in JSON format, which now gets parsed using the builtin JavaScript function
JSON.parse() for best performance.
A "HAFAS-XML" departure document of size 0.9 KB (uncompressed 4.2 KB) replaces an HTML document with the same information in it, but of size 22.6 KB (uncompressed 150 KB). This means that 25 times less data needs to be downloaded :)
The "HAFAS-binary" format for journeys is only 2 KB, the HTML version is 69 KB (both compressed), this means over 34 times less data to be downloaded.
Some providers also offer route data in an XML format, others in a Lynx text version or mobile/desktop HTML. Route data can now be downloaded later as additional timetable data using a new Plasma service for the departure data source. Unfortunately the URLs to the route data documents (traininfo.exe) are only available from HTML departure sources, therefore a mobile version of the HTML departure document gets parsed for the URLs and then cached for following additional data requests.
The applet got a new setting (in the "Advanced" tab), that controls when additional data should be requested. By default additional data gets requested when a departure gets unfolded for the first time. There is also the possibility to directly request all route data for all received departures. This makes sense when route data gets used in filters, ie. the "Via" or "Next Stop" filters. These filters cannot work correctly until the route data is available. Otherwise this setting should not be changed to not produce too much network traffic and waste CPU time.
The de_rmv script also uses the new HAFAS base script, but it uses another XML format (maybe only used by de_rmv). This XML format is somewhat longer but also includes route data. Actually the format used for de_rmv is the same that was already used in previous versions. But I ported the C++ code to QtScript (very easy with the qt.xml extension), it does no longer use an own provider type C++ class.
The it_cup2000 script is another special user of the HAFAS base script, it implements it's own parser functions for HTML departures/arrivals/stop suggestions.
Back to live is us_septa, also a provider using HAFAS software. Departures/arrivals are still only available as HTML, but journeys are available in the binary format.
Removed Providers
EFA also uses an undocumented API like HAFAS. EFA scripts parsing HTML that do not work any longer are removed for now. A base script class for EFA providers could be written just like it's done with HAFAS providers (base_hafas.js, base_hafas_timetable.js, etc.). TimetableMate can be used for this task. This would make it easy to add support for some more providers (mostly in germany).
The slovak provider sk_imhd was removed because of much changed HTML structure, fr_gares, sk_atlas were also removed because they no longer work, de_vvs is waiting for an EFA base script to be added again. If you miss one of these providers, you could use TimetableMate to fix the scripts, add yourself as author and send me the fixed script.
Improved Updates
The "Update Timetable" action of the applet works again. This is based on work in the engine and uses the "requestUpdate" operation of the timetable service.
The engine now stores two QDateTime values in timetable data sources: "nextAutomaticUpdate" and "minManualUpdateTime". Automatic updates are done by the engine at the given time. Before "minManualUpdateTime" has passed all (manual) update requests are rejected/blocked by the engine. The applet uses these values to disable the update action while the engine would reject the update and to show the next automatic update time to the user (in the tooltip of the bottom label).
 |
The applet with disabled update action,
showing the next automatic update
|
Load Additional Timetable Data Later
If some timetable data is not available without more work (eg. downloading more documents) providers can now provide such data later as additional timetable data per departure/arrival. HAFAS providers use this for route data, which is not directly available in their departure XML-documents.
The applet by default requests additional data when a departure gets unfolded for the first time. It is also possible to never request additional data or to directly request additional data for all new departures.
 |
| The applet configuration showing additional data options |
Marble Public Transport Stop Map
Marble gets now used to show public transport stops in a map (if Marble is available at build time). Such a map gets shown in a popup for StopLineEdit of libpublictransporthelper. Stops can be clicked in the map to use it's name in the StopLineEdit. Providers can now provide longitude/latitude values for each stop they read in getStopSuggestions(), these values are needed for the map to work. For providers that are able to get stops by geological position (eg. most HAFAS providers), the stops of the currently shown region of the map are automatically loaded.
The stop coordinates of the providers are now also used by the applet instead of using the osm engine. This makes the "Show in Map" action work reliably. The runner also uses this for the "run action" of stop suggestion results and shows found stops in Marble.
 |
| Applet configuration dialog showing a popup map |
Improved Journey Search
Searching for journeys also got some new features. Favorite/recent journey searches with alias names can be used and quickly be executed using the "Quick Journey Search" button. To get earlier or later journeys one click is enough. And for HAFAS providers there are now more details, every single intermediate stop can be shown (hidden by default, showing only connecting stops).
 |
New quick journey search button
with favorite/recent journey searches |
 |
Favorite/recent journey searches
can be edited in a dialog |
 |
| Earlier and later journeys can easily be requested |
 |
More details for journeys of HAFAS providers,
showing all intermediate stops |
TimetableMate
Writing new provider scripts gets a lot easier with TimetableMate 0.3. It can now open multiple projects, debug scripts, got some docks (projects, variables, backtrace, breakpoints, console, output, web inspector, network monitor, test, documentation), there are multiple automatic tests, a project dashboard written in QML, a GTFS database viewer etc. There is also a new icon for it, a modified KDevelop-icon. TimetableMate now looks more like KDevelop.
The
QScriptEngineDebugger class is not used, instead there is now a new script debugger to better integrate it into TimetableMate. It also allows other special handling for provider scripts.
The documentation shown inside TimetableMate gets generated at build time from some of the source documentation for the data engine.
 |
TimetableMate with the new project dashboard
and the console/projects docks |
 |
TimetableMate while debugging,
showing current variables and some produced output |
 |
| The new testing features of TimetableMate |
Versioning
All this will be available in the next version 0.11. If you wonder what happened to 0.10, it silently got a release tag in GIT, but the provider scripts were mostly no longer working (changed HTML layouts, etc.). I already had new working scripts, but they depended on the new script API and porting them back was not easily possible. But 0.11 will be more resistant to such errors.
Previously I planned to release GTFS in version 0.12, but since GTFS already works quite well and testers are needed, there is no reason to delay it further. Especially because provider types can be disabled at build time now. A 0.12 build without GTFS would have been the same as a 0.11 build.
Test it!
It will still have some bugs, but nothing too big. Most provider plugins should work as expected. All features should also work when supported by the used provider. Please help to make 0.11 stable :)
