After uploading firmware with REST API error: KeyError: 'unpacker'

[paulrichardo]

The FACT version you are using

6cb9354

Your question

Hello,
After uploading ASUS Router fw for a test (zip file from their site) I get an error:
[2023-10-19 09:57:57][unpacking_scheduler][INFO]: Unpacking completed: 51d4c4b657e792ddd918b2d9e927936ede73362b6322d8a6ef6ec76e46669bda_18481152 (extracted files: 1)
[2023-10-19 09:57:57][unpacking_scheduler][ERROR]: Exception occurred during unpacking of 51d4c4b657e792ddd918b2d9e927936ede73362b6322d8a6ef6ec76e46669bda_18481152
Traceback (most recent call last):
File "/home/ubuntu/Desktop/FACT/FACT_core/src/scheduler/unpacking_scheduler.py", line 186, in _work_thread_wrapper
self.work_thread(task, container)
File "/home/ubuntu/Desktop/FACT/FACT_core/src/scheduler/unpacking_scheduler.py", line 209, in work_thread
self.post_unpack(task)
File "/home/ubuntu/Desktop/FACT/FACT_core/src/scheduler/analysis/scheduler.py", line 189, in start_analysis_of_object
self.task_scheduler.schedule_analysis_tasks(fo, fo.scheduled_analysis, mandatory=True)
File "/home/ubuntu/Desktop/FACT/FACT_core/src/scheduler/task_scheduler.py", line 22, in schedule_analysis_tasks
scheduled_analysis = self._add_dependencies_recursively(copy(scheduled_analysis) or [])
File "/home/ubuntu/Desktop/FACT/FACT_core/src/scheduler/task_scheduler.py", line 60, in _add_dependencies_recursively
new_dependencies = self.get_cumulative_remaining_dependencies(scheduled_analyses_set)
File "/home/ubuntu/Desktop/FACT/FACT_core/src/scheduler/task_scheduler.py", line 67, in get_cumulative_remaining_dependencies
return {
File "/home/ubuntu/Desktop/FACT/FACT_core/src/scheduler/task_scheduler.py", line 68, in
dependency for plugin in scheduled_analyses for dependency in self.plugins[plugin].DEPENDENCIES
KeyError: 'unpacker'

And the interface doesn't continue the analysis:

When I upload the same file through the web interface it works fine.

Just tested another firmware, which is not a zip file, with exactly same result: unpacker exception through REST API, but works fine through the web interface.

[jstucke]

did you add "unpacker" to the "requested_analysis_systems"? You don't need to do that. The "unpacker" is not a "real" analysis plugin. Instead, unpacking always happens before analysis

[jstucke]

But thank you for sharing this problem. The "requested_analysis_systems" should be checked and filtered before the firmware reaches the scheduler, so I would say this is a bug.

[jstucke]

#1148 should fix this

[paulrichardo]

Thank you.
Yes I did, since I didn't find a way to add all the plugins from the available list in the REST API docs. Am I missing something?

I added all the plugins from the analyzed firmware list of plugins:
json_data['requested_analysis_systems'] = [
'binwalk', 'cpu_architecture', 'crypto_hints', 'crypto_material', 'cve_lookup', 'cwe_checker',
'device_tree', 'elf_analysis', 'exploit_mitigations', 'file_hashes', 'file_system_metadata',
'file_type', 'hardware_analysis', 'hashlookup', 'information_leaks', 'init_systems', 'input_vectors',
'interesting_uris', 'ip_and_uri_finder', 'ipc_analyzer', 'kernel_config', 'known_vulnerabilities',
'printable_strings', 'qemu_exec', 'software_components', 'source_code_analysis', 'string_evaluator',
'tlsh', 'unpacker', 'users_and_passwords'
]

[jstucke]

What you can't know is that 'unpacker' is not actually a plugin. It is the output of the unpacking process displayed as if it was a plugin. Otherwise, all of the entries should work. But be warned: Some of the plugins can be very slow (e.g. "cwe_checker") and you may not want to run them on every file.

[paulrichardo]

Thank you.