add support for analysis of source code/scripted languages#1080

Draft

adamstorek wants to merge 51 commits intomandiant:masterfrom adamstorek:capa-scripts

+2,851-10

Commits on Jul 19, 2022

Added initial capa control flow for scripts in C#.
adamstorek
committed
Implemented some further basic TreeSitter Extractor-related concepts such as byte-range address.
adamstorek
committed
Modified mypy config file to ignore tree-sitter's missing exports.
adamstorek
committed
Implemented core tree sitter engine component with C# queries that serves as an interface to the language-specific tree-sitter queries.
adamstorek
committed
Implemented script global extraction handlers (mostly wrapping existing language-independent extractors).
adamstorek
committed
Reworked format parsing to align better with the rest of capa logic.
adamstorek
committed
Implemented a large part of the C# functionality; refactored the TreeSitterEngine class.
adamstorek
committed
Added function-level feature extraction.
adamstorek
committed
Bug fixes and code refactoring of the Tree Sitter extractor.
adamstorek
committed
Added tree_sitter to requirements in setup.py.
adamstorek
committed
Added tests for TreeSitterExtractorEngine initialization, new object and function definition parsing for a pure C# sample.
adamstorek
committed
Added more TreeSitterExtractorEngine tests for pure C#.
adamstorek
committed
Added last remaining tests for the TreeSitterExtractorEngine class and fixed bugs found in the process.
adamstorek
committed
Reverted yielding only non-empty strings in order to stay consistent and not introduce unspecified rule-exceptions.
adamstorek
committed
Removing functions that should not be used in tree-sitter extractor (default to the base extractor level).
adamstorek
committed
Modifying extraction of global statements to omit local function declarations.
adamstorek
committed
Added script language feature to freeze.
adamstorek
committed
Added test cases for TS Extractor.
adamstorek
committed
Refactored query bindings.
adamstorek
committed
Added support for template parsing.
adamstorek
committed
Added support for HTML parsing.
adamstorek
committed
Implemented the necessary modifications to support embedded templates/html: aspx.
adamstorek
committed
Added more buildings to build; minor style improvement.
adamstorek
committed
Further refactored the Tree-sitter queries and fixed minor template engine bugs.
adamstorek
committed
Refactored extractor engine tests and began adding new template tests.
adamstorek
committed
Added new tests for embedded template testing and refactored a few already implemented extractor engine tests.
adamstorek
committed
Bug fixes in extractor and HTML Tree-sitter engine.
adamstorek
committed
Fixed important namespace-parsing bugs.
adamstorek
committed
Further improvement to namespace parsing, including default namespaces for aspx; introducing TSFunctionInner dataclass to store both the node and the name of a function to make test-case creation e…
adamstorek
committed
Added more tests and a few minor bug fixes.
adamstorek
committed
Added language-specific integer parsing.
adamstorek
committed
Fixed an important bug in FileOffsetRangeAddress comparison method.
adamstorek
committed
Added more ASPX tests.
adamstorek
committed
Fixed the capa control flow to fully support capa scripts.
adamstorek
committed
Major changes: switching imports and function names to properties, streamlining queries, introducing language toolkit class to help move all additional language specific methods to the sample place.
adamstorek
committed
Fixed property-extraction bugs.
adamstorek
committed
Added few more test cases.
adamstorek
committed
Minor style improvements.
adamstorek
committed
Removed deprecated parse_integer.
adamstorek
committed
Added more tests; fixed integer parsing related bugs.
adamstorek
committed

Commits on Jul 20, 2022

Commits on Jul 26, 2022

Added support for Python.
adamstorek
committed

Commits on Jul 29, 2022

Added more python test cases; fixed a number of python bugs; further refactored language toolkit code; added extraction of global constants.
adamstorek
committed

Commits on Aug 2, 2022

Implemented namespace aliasing; further refactored the codebase.
adamstorek
committed

Commits on Aug 3, 2022

Commits on Aug 5, 2022

Added more test cases; moved script language feature to global features in order to make rules clearer; refactored the codebase to address the latest PR comments/suggestions.
adamstorek
committed

Commits on Aug 10, 2022

Introduced auto-detection to template-script parsing, builtins namespace, and modified tests.
adamstorek
committed

Commits on Aug 12, 2022

Attempted to implement the class extraction as specified last Friday (passes all test cases but by no means perfect); further clean up, especially of the signatures; synced with new Python test cases.
adamstorek
committed