Skip to content

Commit

Permalink
Optimize python.QueryToScopeCase
Browse files Browse the repository at this point in the history 
Summary:
We want to ensure this is always "out to in" when constructing SNames. .. at least for very long SNames. For short ones, not sure.

something recently changed to trigger the cost of this to blow up as it enumerates SNames in the opposite direction.

Fix it by specifying an order and adding some perf tests.

Before: would be a timeout:
```
fbsource.fbcode.python>  search.python.QueryToScopeCase { ["gen_ai","genie_projects","offline_inference", "app","llama2","config"], false, _ }
{
  "id": 1419228269,
  "key": {
    "query": [ "gen_ai", "genie_projects", "offline_inference", "app", "llama2", "config" ],
    "insensitive": false,
...

1 results, 13 facts, 7464.64ms, 24105032 bytes, 12056 compiled bytes
Facts searched:
                          python.SName.4 : 142544
             python.SearchModuleByName.4 : 5
                           python.Name.4 : 1
              python.SearchClassByName.4 : 1
```

After:
```
fbsource.fbcode.python> search.python.QueryToScopeCase { ["gen_ai","genie_projects","offline_inference", "app","llama2","config"], false, _ }
{
  "id": 1419086769,
  "key": {
    "query": [ "gen_ai", "genie_projects", "offline_inference", "app", "llama2", "config" ],
    "scopeName": {
...

1 results, 13 facts, 14.80ms, 28758344 bytes, 14170 compiled bytes
Facts searched:
                          python.SName.4 : 313
                           python.Name.4 : 6
             python.SearchModuleByName.4 : 5
              python.SearchClassByName.4 : 1
```

I think this lurking issue might also explain why there was only minor movement in python scope search p99, whereas name search dropped dramatically when the name tables switched over;

 {F1973781970}

Reviewed By: phlalx

Differential Revision: D67120662

fbshipit-source-id: 3445dbf55d7d86dc8a82b25ade37b4e01edbcd25
  • Loading branch information
@donsbot @facebook-github-bot
donsbot authored and facebook-github-bot committed Dec 12, 2024
1 parent 1a4844f commit bfef488
Show file tree
Hide file tree
Showing 113 changed files with 2,152 additions and 91 deletions.
573 changes: 573 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/fileentities.out
View file

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/fileentities.perf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"@generated": null,
"facts_searched": { "python.DeclarationsByFile.4": 31 },
"full_scans": [ "python.DeclarationsByFile.4" ]
}
3 changes: 3 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/fileentities.query
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
query: codemarkup.FileEntityLocations _
perf: true
transform: [normord, []]
2 changes: 1 addition & 1 deletion glean/lang/codemarkup/tests/python/cases/search/search_by_name_class_meth.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 42, "start": 125 } },
"location": { "span": { "length": 42, "start": 260 } },
"name": "some.foo.Foo.meth"
},
"name": "meth",
Expand Down
2 changes: 1 addition & 1 deletion glean/lang/codemarkup/tests/python/cases/search/search_by_name_class_meth_lc.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 42, "start": 125 } },
"location": { "span": { "length": 42, "start": 260 } },
"name": "some.foo.Foo.meth"
},
"name": "meth",
Expand Down
2 changes: 1 addition & 1 deletion glean/lang/codemarkup/tests/python/cases/search/search_by_scope_class_meth.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 42, "start": 125 } },
"location": { "span": { "length": 42, "start": 260 } },
"name": "some.foo.Foo.meth"
},
"name": "meth",
Expand Down
4 changes: 2 additions & 2 deletions glean/lang/codemarkup/tests/python/cases/search/search_by_scope_class_meth.perf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 1,
"python.SName.4": 1,
"python.Name.4": 3,
"python.SName.4": 3,
"python.SearchClassByName.4": 1,
"python.SearchMethodByName.4": 1,
"python.SearchModuleByName.4": 2
Expand Down
2 changes: 1 addition & 1 deletion glean/lang/codemarkup/tests/python/cases/search/search_by_scope_foo.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 28, "start": 170 } },
"location": { "span": { "length": 28, "start": 305 } },
"name": "some.foo.foo"
},
"name": "foo",
Expand Down
4 changes: 2 additions & 2 deletions glean/lang/codemarkup/tests/python/cases/search/search_by_scope_foo.perf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 1,
"python.SName.4": 3,
"python.Name.4": 2,
"python.SName.4": 2,
"python.SearchFunctionByName.4": 1,
"python.SearchModuleByName.4": 2
},
Expand Down
4 changes: 2 additions & 2 deletions glean/lang/codemarkup/tests/python/cases/search/search_by_scope_foo_long.perf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 1,
"python.SName.4": 1,
"python.Name.4": 4,
"python.SName.4": 4,
"python.SearchClassByName.4": 1,
"python.SearchMethodByName.4": 1,
"python.SearchModuleByName.4": 3
Expand Down
4 changes: 2 additions & 2 deletions glean/lang/codemarkup/tests/python/cases/search/search_by_scope_foo_long_insensitive.perf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 1,
"python.SName.4": 1,
"python.Name.4": 4,
"python.SName.4": 4,
"python.SearchClassByLowerCaseName.4": 2,
"python.SearchFunctionByLowerCaseName.4": 1,
"python.SearchMethodByLowerCaseName.4": 1,
Expand Down
4 changes: 2 additions & 2 deletions glean/lang/codemarkup/tests/python/cases/search/search_by_scope_foo_long_prefix_2.perf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 1,
"python.SName.4": 1,
"python.Name.4": 2,
"python.SName.4": 2,
"python.SearchClassByName.4": 1,
"python.SearchMethodByName.4": 1,
"python.SearchModuleByName.4": 1
Expand Down
4 changes: 2 additions & 2 deletions glean/lang/codemarkup/tests/python/cases/search/search_by_scope_foo_long_prefix_3.perf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 1,
"python.SName.4": 1,
"python.Name.4": 3,
"python.SName.4": 3,
"python.SearchClassByName.4": 1,
"python.SearchMethodByName.4": 1,
"python.SearchModuleByName.4": 2
Expand Down
2 changes: 1 addition & 1 deletion glean/lang/codemarkup/tests/python/cases/search/search_by_scope_foo_medium.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 213, "start": 0 } },
"location": { "span": { "length": 1267, "start": 0 } },
"name": "some.foo"
},
"name": "foo",
Expand Down
4 changes: 2 additions & 2 deletions glean/lang/codemarkup/tests/python/cases/search/search_foo_by_name.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 28, "start": 170 } },
"location": { "span": { "length": 28, "start": 305 } },
"name": "some.foo.foo"
},
"name": "foo",
Expand Down Expand Up @@ -49,7 +49,7 @@
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 213, "start": 0 } },
"location": { "span": { "length": 1267, "start": 0 } },
"name": "some.foo"
},
"name": "foo",
Expand Down
1 change: 1 addition & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9.out
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[ "@generated" ]
1 change: 1 addition & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9.perf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{ "@generated": null, "facts_searched": { }, "full_scans": [ ] }
3 changes: 3 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9.query
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
query: codemarkup.search.SearchByScope { name = "meth_9", scope = ["some","foo","In1","In2","In3","In4","In5","In6","In7","In8","In9"], searchcase = Sensitive, language = Python }
perf: true
transform: [normord, []]
41 changes: 41 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a.out
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
[
"@generated",
{
"key": {
"entity": {
"python": {
"decl": {
"func": {
"key": {
"name": {
"key": "some.foo.In1.In2.In3.In4.In5.In6.In7.In8.In9.meth_9"
}
}
}
}
}
},
"kind": 7,
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 69, "start": 629 } },
"name": "some.foo.In1.In2.In3.In4.In5.In6.In7.In8.In9.meth_9"
},
"name": "meth_9",
"scope": [
"In1",
"In2",
"In3",
"In4",
"In5",
"In6",
"In7",
"In8",
"In9",
"foo"
],
"searchcase": 0
}
}
]
13 changes: 13 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a.perf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"@generated": null,
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 10,
"python.SName.4": 12,
"python.SearchClassByName.4": 9,
"python.SearchMethodByName.4": 1,
"python.SearchModuleByName.4": 1
},
"full_scans": [ ]
}
3 changes: 3 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a.query
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
query: codemarkup.search.SearchByScope { name = "meth_9", scope = ["foo","In1","In2","In3","In4","In5","In6","In7","In8","In9"], searchcase = Sensitive, language = Python }
perf: true
transform: [normord, []]
40 changes: 40 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a_class_kind.out
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
[
"@generated",
{
"key": {
"entity": {
"python": {
"decl": {
"cls": {
"key": {
"name": {
"key": "some.foo.In1.In2.In3.In4.In5.In6.In7.In8.In9"
}
}
}
}
}
},
"kind": 6,
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 116, "start": 582 } },
"name": "some.foo.In1.In2.In3.In4.In5.In6.In7.In8.In9"
},
"name": "In9",
"scope": [
"In1",
"In2",
"In3",
"In4",
"In5",
"In6",
"In7",
"In8",
"foo"
],
"searchcase": 0
}
}
]
12 changes: 12 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a_class_kind.perf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"@generated": null,
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 9,
"python.SName.4": 11,
"python.SearchClassByName.4": 9,
"python.SearchModuleByName.4": 1
},
"full_scans": [ ]
}
3 changes: 3 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a_class_kind.query
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
query: codemarkup.search.SearchByScope { name = "In9", scope = ["foo","In1","In2","In3","In4","In5","In6","In7","In8"], searchcase = Sensitive, language = Python, kind = { just = Class_ } }
perf: true
transform: [normord, []]
41 changes: 41 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a_kind.out
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
[
"@generated",
{
"key": {
"entity": {
"python": {
"decl": {
"func": {
"key": {
"name": {
"key": "some.foo.In1.In2.In3.In4.In5.In6.In7.In8.In9.meth_9"
}
}
}
}
}
},
"kind": 7,
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 69, "start": 629 } },
"name": "some.foo.In1.In2.In3.In4.In5.In6.In7.In8.In9.meth_9"
},
"name": "meth_9",
"scope": [
"In1",
"In2",
"In3",
"In4",
"In5",
"In6",
"In7",
"In8",
"In9",
"foo"
],
"searchcase": 0
}
}
]
13 changes: 13 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a_kind.perf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"@generated": null,
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 10,
"python.SName.4": 12,
"python.SearchClassByName.4": 9,
"python.SearchMethodByName.4": 1,
"python.SearchModuleByName.4": 1
},
"full_scans": [ ]
}
3 changes: 3 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_a_kind.query
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
query: codemarkup.search.SearchByScope { name = "meth_9", scope = ["foo","In1","In2","In3","In4","In5","In6","In7","In8","In9"], searchcase = Sensitive, language = Python, kind = { just = Method } }
perf: true
transform: [normord, []]
40 changes: 40 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_b.out
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
[
"@generated",
{
"key": {
"entity": {
"python": {
"decl": {
"func": {
"key": {
"name": {
"key": "some.foo.In1.In2.In3.In4.In5.In6.In7.In8.In9.meth_9"
}
}
}
}
}
},
"kind": 7,
"language": 4,
"location": {
"file": { "key": "some/foo.py" },
"location": { "span": { "length": 69, "start": 629 } },
"name": "some.foo.In1.In2.In3.In4.In5.In6.In7.In8.In9.meth_9"
},
"name": "meth_9",
"scope": [
"In1",
"In2",
"In3",
"In4",
"In5",
"In6",
"In7",
"In8",
"In9"
],
"searchcase": 0
}
}
]
12 changes: 12 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_b.perf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"@generated": null,
"facts_searched": {
"python.DeclarationDefinition.4": 1,
"python.DeclarationLocation.4": 1,
"python.Name.4": 9,
"python.SName.4": 9,
"python.SearchClassByName.4": 9,
"python.SearchMethodByName.4": 1
},
"full_scans": [ ]
}
3 changes: 3 additions & 0 deletions glean/lang/codemarkup/tests/python/cases/search/search_scope_long_9_b.query
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
query: codemarkup.search.SearchByScope { name = "meth_9", scope = ["In1","In2","In3","In4","In5","In6","In7","In8","In9"], searchcase = Sensitive, language = Python }
perf: true
transform: [normord, []]
Loading

0 comments on commit bfef488

Please sign in to comment.