Show prevalence of rules in the output

[Aayush-Goel-04]

relates to #520

Checklist

• No CHANGELOG update needed

• No new tests needed

• No documentation update needed

[Aayush-Goel-04]

we can add a prompt at bottom to reference what unknown means

[mr-tz]

Good start!
Do you have an idea on if/how to display this data for the other output modes (verbose, very verbose, and also JSON)?

[Aayush-Goel-04]

We can also add create a new field prevalence to RuleMetadata or RuleMatches.
We can directly store prevalence : rare | common | unknown (if not found) while building resultDocument, in this way while rendering we will only need slight modifications to json, -v, -vv and default render modes.

capa/capa/render/result_document.py

Lines 559 to 575 in 9d21add

	class ResultDocument(FrozenModel):
	meta: Metadata
	rules: Dict[str, RuleMatches]
	@classmethod
	def from_capa(cls, meta: Metadata, rules: RuleSet, capabilities: MatchResults) -> "ResultDocument":
	rule_matches: Dict[str, RuleMatches] = {}
	for rule_name, matches in capabilities.items():
	rule = rules[rule_name]
	if rule.meta.get("capa/subscope-rule"):
	continue
	rule_matches[rule_name] = RuleMatches(
	meta=RuleMetadata.from_capa(rule),
	source=rule.definition,
	matches=tuple(

What are your thoughts @mr-tz

[mr-tz]

We can also add create a new field prevalence to RuleMetadata or RuleMatches.

That could work well if we find a place that requires few modifications and is flexible. I think we'd want to keep prevalence data and rule information separate (with a separate DB as you're proposing here).

[Aayush-Goel-04]

for verbose we can do as follows

receive data (2 matches)
namespace    communication
description    all known techniques for receiving data from a potential C2 server
prevalence    common
scope            function
matches        0x10003A13

we can do similar for vverbose

[williballenthin]

use get_default_root()

capa/capa/main.py

Line 444 in 210a13d

def get_default_root() -> Path:

[Aayush-Goel-04]

using get_default_root works well locally but it cause circular import when being during pyinstaller build.
@williballenthin I suggest moving such functions to capa.helpers.

[mr-tz]

moving it makes sense (see #1821 also)

[Aayush-Goel-04]

@mr-tz I suggest we move ahead with proposal 3 in above mentioned PR.
moving below to a new capa.loader or we can move them to capa.helper

has_file_limitation
is_supported_format
is_supported_arch
get_arch
is_supported_os
get_os
is_running_standalone
get_default_root
get_default_signatures
get_workspace
get_extractor
get_file_extractors
get_signatures
get_sample_analysis
collect_metadata
compute_dynamic_layout
compute_static_layout
compute_layout

[mr-tz]

sounds good to me!

[williballenthin]

is the rule prevalence database distributed with capa the library? i think its important that people be able to use capa the library without maintaining this database. so perhaps we want to handle the case of the database not existing here?

[Aayush-Goel-04]

In case database is not present, all rule matches will have prevalence as unknown in the results.

[mr-tz]

maybe we can provide a warning if no db is found (in case that's not already there) pointing to one and explaining shortly what it does

[mr-tz]

while we're at it, is it worth defining a pydantic data model for the DB file/format?

[williballenthin]

looks like the format is dict[rule name, prevalence] which will be hard to represent in pydantic, unless we enumerate all the rule names as potential values. i think the type hint above is a good start. still, adding some comments here showing a snippet of the file would be valuable.

[Aayush-Goel-04]

Apologies for disappearing there for a bit – college placement stuff got pretty intense.

Back to PR, Tests are failing in pyinstaller due circular import when trying to fetch path for rules_prevalence database using get_default_root , while trying to load_rules_prevalence in result_document. What we can do is -

• move get_default_root to capa.helpers.
• We can move prevalence database to python files just like we did with COM database.

What are ur thoughts @mr-tz @williballenthin

[mr-tz]

I think moving to Python files analogous to the COM DB files sounds good.

There's been a bunch of changes recently on the API, so please ensure the PR is up to date with master.

[Aayush-Goel-04]

I have converted the database to python file. Now we just need the actual prevalence values for rules, and this will be good to go.

[VascoSch92]

Suggested change

	Returns:
	Return: