Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

memory-inefficient area:id filter #359

Closed
tyrasd opened this issue Jan 2, 2017 · 2 comments · May be fixed by #364
Closed

memory-inefficient area:id filter #359

tyrasd opened this issue Jan 2, 2017 · 2 comments · May be fixed by #364
Labels
bug

Comments

@tyrasd
Copy link
Contributor

tyrasd commented Jan 2, 2017

Consider the following two almost identical queries:

way["building"](area:3600285864);

and:

area(3600285864);
way["building"](area._);

The first one fails (with a Query run out of memory using about 2048 MB of RAM runtime error) while the second one works fine. Here's the example in overpass turbo.

//cc @mocnik-science
@mmd-osm
Copy link
Contributor

mmd-osm commented Jan 2, 2017 •
edited
Loading

Thanks for reporting!

Preliminary analysis:

Case 1: way["building"](area:3600285864);

bool Area_Constraint::delivers_data(Resource_Manager& rman)
{
  if (!area->areas_from_input())
      return false;                  // <<<<--- more sophisticated logic needed here

Irrespective of the actual area size, delivers_data will be always false, leading to check_keys_late being false in query.cc

This has further impact on filter_id_list in https://github.com/drolbr/Overpass-API/blob/master/src/overpass_api/statements/query.cc#L157

Method may not stop processing after 1 Mio. entries because of check_keys_late being false. This way, the method accumulates around 210 Mio entries in new_ids!

Case 2:

More sophisticated logic to determine area size in https://github.com/drolbr/Overpass-API/blob/master/src/overpass_api/statements/area_query.cc#L294-L308

-> delivers_data is true. Early exit of filter_id_list is possible.

@drolbr drolbr added the bug label Jan 12, 2017
tyrasd added a commit to tyrasd/Overpass-API that referenced this issue Mar 2, 2017
@tyrasd
implement area size heuristic for area:id queries
4fa2e8e 
fixes drolbr#359
tyrasd added a commit to tyrasd/Overpass-API that referenced this issue Mar 2, 2017
@tyrasd
implement area size heuristic for area:id queries
200debd 
fixes drolbr#359
@tyrasd tyrasd mentioned this issue Mar 2, 2017
Open
@drolbr
Copy link
Owner

drolbr commented Apr 5, 2018

Fixed in dba448f

@drolbr drolbr closed this as completed Apr 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

implement area size heuristic for area:id queries tyrasd/Overpass-API
3 participants
@drolbr @tyrasd @mmd-osm