Future import pipeline planning

[jbruechert]

I think it might be time to replace gtfsclean for a number of reasons:

• MOTIS is gaining NeTEx support. We could not do our usual checks and filtering from gtfsclean on netex
• No support for gtfs-flex. We currently skip gtfsclean for these feeds, but that again skips filtering.
• MOTIS itself is more forgiving in parsing than gtfsclean.
• our import pipeline is currently weirdly split between gtfsclean features and lua scripts.

IMO the only thing that makes sense is doing the filtering in MOTIS itself, to avoid implementing support for data formats multiple time. This could be done by adding the relevant features to MOTIS, or by automatically adding a lua script that does this everywhere. The lua scripting does not currently expose the needed features though.

Required features:

• null-coordinate filtering (null-coordinates still there, but don't lead to problems)
• speed-based filtering (miiiight not be needed if problems only stem from (0,0)-coordinates, not sure)
• agency-based filtering (doable in lua)
• finding gtfs in a subdirectory of the zip. Needs checking if this still occurs. The one feed I remember is meanwhile replaced.

It would still be nice to be able to re-publish our feeds as we currently do. I'd propose to:

• Try adding GTFS writing functionality to MOTIS. This would also give us a generic NeTEx to GTFS converter for free which as fas as I can tell the ecosystem is currently lacking. Particularly in the beginning, this feature might not be able to write all GTFS features.
• re-publish the unprocessed feeds as well. This is probably useful regardless of the rest. Sometimes people might need the files without filtering.

Steps

• Make sure required MOTIS features are ready
• Add option to opt out of gtfsclean per-feed
• Flip default to make gtfsclean opt-in

[felixguendling]

agency-based filtering

This is already implemented with Lua, but we can also add it as a config option if needed.

null-coordinate filtering
speed-based filtering

I'm not sure if we really need this. MOTIS stopped to connect stops around (0, 0) shortly after Transitous hit problems with it. So this does not lead to bugs or long preprocessing times anymore.

In general I don't see any issues in having stops around (0, 0) in the data. The stops still carry information (like the stop name) for the user in terms of "next stop: ..." or for counting "I need to alight in 3 stops". Of course, coordinates are nice for walk+transit connections, calculating transfers with OSM, "departures near me" function, etc. But if they are missing, the stop isn't useless because it still works as a start and destination stop (for station-station queries). So I would prefer to keep them.

This is mostly a graphical/display issue that it doesn't look nice on the map. But if that's the only issue, I would propose to add filtering to the UI code.

finding gtfs in a subdirectory of the zip

Should not be hard to add to MOTIS. Instead of expecting files to be top-level, we can iterate the directory structure and search for matching files.

It would still be nice to be able to re-publish our feeds as we currently do

I agree this would be nice-to-have. But I would also say that there are problems that bring more value to Transitous itself if we solve them. For example, we don't have RT data ~half of the time (just a guess) in many regions (e.g. France). The solution could be this. So unless another Open Source + non-profit project relies on us republishing data, I would not see a real use case for doing so.

[jbruechert]

agency-based filtering

This is already implemented with Lua, but we can also add it as a config option if needed.

Ah nice! For the limited places we use this in this might be easy enough to do in lua.

null-coordinate filtering

I'm not sure if we really need this. MOTIS stopped to connect stops around (0, 0) shortly after Transitous hit problems with it. So this does not lead to bugs or long preprocessing times anymore.

In general I don't see any issues in having stops around (0, 0) in the data. The stops still carry information (like the stop name) for the user in terms of "next stop: ..." or for counting "I need to alight in 3 stops". Of course, coordinates are nice for walk+transit connections, calculating transfers with OSM, "departures near me" function, etc. But if they are missing, the stop isn't useless because it still works as a start and destination stop (for station-station queries). So I would prefer to keep them.

This is mostly a graphical/display issue that it doesn't look nice on the map. But if that's the only issue, I would propose to add filtering to the UI code.

True. Since this will be a problem on every client app, I think we should at least have the MOTIS API return an optional value for the position, so it is clear that this may be missing.

speed-based filtering

Speed based filtering would still be needed though, right?

It would still be nice to be able to re-publish our feeds as we currently do

I agree this would be nice-to-have. But I would also say that there are problems that bring more value to Transitous itself if we solve them. For example, we don't have RT data ~half of the time (just a guess) in many regions (e.g. France). The solution could be this. So unless another Open Source + non-profit project relies on us republishing data, I would not see a real use case for doing so.

Definitely. This issue is not supposed to be a feature request in any way. More or less just that I'd consider to implement this if it feels like an interesting thing to do.

[felixguendling]

Speed based filtering would still be needed though, right?

My impression is that the majority of speed filtering overlaps with (0, 0).

What happens (or should happen) for speed filtering? Remove the whole trip? Or only segments? From what I've seen, it's usually one or two wrong coordinates that are wrong.

It would still be nice to be able to re-publish our feeds as we currently do

I think here it's a bit tricky to do it correctly (even if we say every agency time zone is UTC) because GTFS doesn't use wall-clock time.
https://gist.github.com/derhuerst/574edc94981a21ef0ce90713f1cff7f6

[felixguendling]

MOTIS API return an optional value for the position

This would be a breaking change in the API. Maybe we can add a API flag.

I'm not sure if we should try to cover up data errors in general. If the app is used by locals, this could trigger that data is fixed if they inform the right people. If we hide all data errors, this is unlikely to happen.

[Lach-anonym]

The 0,0 matter is more complicated - some feeds use regional reference systems, so their 0,0 point ends up in a different place - for example Algerian desert for French feeds

[jbruechert]

The 0,0 matter is more complicated - some feeds use regional reference systems, so their 0,0 point ends up in a different place - for example Algerian desert for French feeds

That would be a reason to add speed-based filtering.

Iirc something like this already exists for real time updates, so might be easy to do.

[Lach-anonym]

This also reminds me the issue with Spanish and Turkish feeds with broken shapes

[felixguendling]

some feeds use regional reference systems

NeTEx lets you define which reference system is used and we can use PROJ to transform them to WGS84. So I would hope that this is not a case for speed based filtering after using the correct projection.

Iirc something like this already exists for real time updates, so might be easy to do.

Speed based filtering is trivial to implement in general, if it's just to drop the whole trip. I think it's mainly a question if we want to hide/cover data errors or expose them (to have a higher likelihood that they will be fixed).

[jbruechert]

Speed based filtering is trivial to implement in general, if it's just to drop the whole trip. I think it's mainly a question if we want to hide/cover data errors or expose them (to have a higher likelihood that they will be fixed).

At this point I think the likelyhood of broken routing results is way to high to just expose all of that. Users can still report missing trips to us and we can report them if that is even possible in the affected feed.

So I'm all for the safe route there.
Flying busses all over the map also just create a very bad first impression.

[jbruechert]

some feeds use regional reference systems

NeTEx lets you define which reference system is used and we can use PROJ to transform them to WGS84. So I would hope that this is not a case for speed based filtering after using the correct projection.

This assumes that the affected data is always available in NeTEx, which I doubt.

[felixguendling]

• null-island
  • routing: null-island is special cased (= no wormholes on null-island) and trips would not need to be filtered to prevent broken routing results. I.e. a dataset with all coordinates set to null-island should be totally fine in terms of timetable routing.
  • visualization: It should be possible to filter null-island related trips from the visualization (and maybe disable this filter if ?debug is active)
• for everything else (wrong coordinates != null island, detectable via speed checks) we'll have to remove them because we can't know what's wrong and we have to generate transfers (= wormholes if coordinates are wrong).

Did I miss anything regarding broken routing results?

This assumes that the affected data is always available in NeTEx, which I doubt.

Sure. If a dataset is broken beyond repair, the only option is to skip it. I just wanted to point out that we already have PROJ integrated for NeTEx exactly to handle those cases in France. If there's no NeTEx but the coordinate system is known and this is a case that happens often enough to be relevant, we could add the same logic for GTFS and add a config variable to enable it.