Database and AZ machinery issue #2323
Comments
|
could it be related to another AZ issues that is in WIP? as we don't run it we can't help here |
|
Hi @doomedraven, I can't say for sure. I am trying to resolve the issue and have tried a lot of things. Just wanted to open it here to see if anyone is either facing the same issue or was able to make Azure work since the revamp of the database which happened a while ago. Also maybe a quick glance from @tbeadle might help since he's probably way better than I am on DB issues. |
|
yes make sense |
|
So I was able to make it work. It seem that for some reasons the function add_machine() from the database require a flush() and commit(). I don't know all the involvement on that but it is strange to me that AZ would be the only machinery affected by this. It could be a versionning issue somehow. Python3.10.14, SQLAlchemy==1.4.50, SQLAlchemy-Utils==0.41.1,sqlparse==0.5.0 which is exactly the version from the requirements.txt and poetry. |
|
so far versions looks ok %) but yes is strange, i did deployment of new servers on GCP 2 days ago and no issues with kvm |
|
@cccs-mog Can you try these on your end in the following order. Before/after each you should start/stop the cape.service:
I noticed that I was getting errors when a VMSS from a previous cape.service run. I'm pretty sure it was only when the previous run's machines still existed at startup. For my case, running the cleaners util fixed that. |
|
Sorry in advanced for the long comment. I have tracked down what causes my issue described above. You may not be experiencing the same thing, but something very similar.
Finally, That new session context manager at the end is the issue, but also necessary. While the main thread responsible for initializing the az machinery has still not closed, we create a subthread that creates its own session context manager. Because the In Snippet of logs demonstrating this behavior: The problem you're running into seems a little different. You're machines are somehow being added to From your logs, I think this is the part that indicates our problem is at least related. Time stamps line up. |
|
All that to say, I don't have a good idea for a fix here. Maybe it would be possible to move the For the time being, my first comment has been a sure-fire workaround for this. |
|
Hi @ChrisThibodeaux, I also faced that issue in the past and applied a similar approach for a temporary fix. It is somehow of a pain but only on each restart. It doesn't seem like anything changed so I think it's because I started with a fresh db or something like that. In any case like you point out, the issue is similar and is caused by the session span without commit interlocking needs and conditions which are contradicting. I would assume it is possible to keep the logic intact for every machinery but separate the session manager in smaller units which would fix it for AZ. I will do some tests, then PR so the fix can be tested and hopefully work well for all machinery. |
|
@cccs-mog Looking forward to the PR, would be great to have a fix for this. I am wondering now if this is related to another issue I have seen, maybe you can provide some insight here. Suppose I have:
It has only ever happened to me a couple times, and unfortunately I did not have the DB logs enabled at the time. I have patched the Edit: Had it crop up right after I made this comment... Seems like a couple of issues to unpack, all stemming from a problem on my end with Azure taking so long to delete/reimage machines. No idea what that's about.. For now, I'm going to copy what you have in your PR and bump the timeout up to 300 seconds. Those timeouts make the threads call I have a fix by checking the machine in the |
|
Hey @cccs-mog, very big thanks to you and @cccs-kevin for all the work y'all have done on this module and in the replies across different issues. Both have been an immense help to me. I have a PR with a fix for what you faced here. I addressed a few other bugs dealing with timeouts, machine failures, etc. I tried to keep my changes in step with the spirit of the machinery as it was written, so please let me know if anything looks out of place. Hope this helps! |
|
so i guess we can close this now? |
|
Exactly. Thanks a lot to you two ! |
|
Much gratitude 🙏 |
Prerequisites
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
A call to self.db.add_machine() should result on a machine being created and thus the check in initialize_machinery() listing machine should not induce an error for the AZ machinery.
Current Behavior
No machine are created at all, the running of this function doesn't seem to produce any database commands being ran.
Failure Information (for bugs)
The session goes all the way throught initialize_machinery(). The AZ machinery overwrite the initialize and initialize_check function. In all cases at some point the machinery will attempt to add machines to the database the same way as the other machinery would do it, but for some reasons it doesn't work. The connection to the database is ok because tasks can be added to it and the schema was created at the beginning. Tested on both SQLite3 and PosgreSQL. This could be a timing issue which is specific to scaling machinery but I would expect that explicitely calling the addition of a machine in the database to actually do it.
Steps to Reproduce
Context
Fresh install on Ubuntu 20.04. Base installation made from cape2.sh. Every aspect seem to work fine except this.
Failure Logs
2024-09-18 18:26:45,412 [root] DEBUG: Imported "machinery" modules:
2024-09-18 18:26:45,412 [root] DEBUG: `-- Azure
2024-09-18 18:26:45,413 [root] DEBUG: Checking for locked tasks...
2024-09-18 18:26:45,417 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-09-18 18:26:45,417 [sqlalchemy.engine.Engine] INFO: BEGIN (implicit)
2024-09-18 18:26:45,448 INFO sqlalchemy.engine.Engine SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id
WHERE tasks.status = %(status_1)s ORDER BY tasks.added_on DESC
2024-09-18 18:26:45,448 [sqlalchemy.engine.Engine] INFO: SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id
WHERE tasks.status = %(status_1)s ORDER BY tasks.added_on DESC
2024-09-18 18:26:45,448 INFO sqlalchemy.engine.Engine [generated in 0.00079s] {'status_1': 'running'}
2024-09-18 18:26:45,448 [sqlalchemy.engine.Engine] INFO: [generated in 0.00079s] {'status_1': 'running'}
2024-09-18 18:26:45,456 INFO sqlalchemy.engine.Engine COMMIT
2024-09-18 18:26:45,456 [sqlalchemy.engine.Engine] INFO: COMMIT
2024-09-18 18:26:45,479 [lib.cuckoo.core.machinery_manager] INFO: Using MachineryManager[az] with max_machines_count=0
2024-09-18 18:26:45,479 [lib.cuckoo.core.scheduler] INFO: Creating scheduler with max_analysis_count=unlimited
2024-09-18 18:27:38,006 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-09-18 18:27:38,006 [sqlalchemy.engine.Engine] INFO: BEGIN (implicit)
2024-09-18 18:27:38,009 INFO sqlalchemy.engine.Engine DELETE FROM machines_tags
2024-09-18 18:27:38,009 [sqlalchemy.engine.Engine] INFO: DELETE FROM machines_tags
2024-09-18 18:27:38,009 INFO sqlalchemy.engine.Engine [generated in 0.00066s] {}
2024-09-18 18:27:38,009 [sqlalchemy.engine.Engine] INFO: [generated in 0.00066s] {}
2024-09-18 18:27:38,013 INFO sqlalchemy.engine.Engine DELETE FROM machines
2024-09-18 18:27:38,013 [sqlalchemy.engine.Engine] INFO: DELETE FROM machines
2024-09-18 18:27:38,013 INFO sqlalchemy.engine.Engine [generated in 0.00050s] {}
2024-09-18 18:27:38,013 [sqlalchemy.engine.Engine] INFO: [generated in 0.00050s] {}
2024-09-18 18:27:38,014 [modules.machinery.az] DEBUG: Connecting to Azure for the region 'XXX'.
2024-09-18 18:27:38,638 [modules.machinery.az] DEBUG: Trying <bound method GalleryImagesOperations.get of <azure.mgmt.compute.v2023_07_03.operations._operations.GalleryImagesOperations object at 0x7fab7172d810>>(('XXX', 'cape_compute_gallery', 'XXX'),{})
2024-09-18 18:27:38,761 [modules.machinery.az] DEBUG: Trying <bound method VirtualMachineScaleSetsOperations.list of <azure.mgmt.compute.v2024_07_01.operations._operations.VirtualMachineScaleSetsOperations object at 0x7fab7172df60>>(('XXX',),{})
2024-09-18 18:27:38,913 [modules.machinery.az] DEBUG: Trying <bound method SubnetsOperations.get of <azure.mgmt.network.operations._operations.SubnetsOperations object at 0x7fab7172f730>>(('XXX', 'VNET', 'XXX'),{})
2024-09-18 18:27:43,160 [modules.machinery.az] DEBUG: Trying <bound method VirtualMachineScaleSetsOperations.begin_reimage_all of <azure.mgmt.compute.v2024_07_01.operations._operations.VirtualMachineScaleSetsOperations object at 0x7fab7172f940>>(('XXX', 'vmss-cape-win10x64-60'),{'polling_interval': 1})
2024-09-18 18:32:08,967 [modules.machinery.az] DEBUG: Adding machines to database for vmss-cape-win10x64-60.
2024-09-18 18:32:08,981 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-09-18 18:32:08,981 [sqlalchemy.engine.Engine] INFO: BEGIN (implicit)
2024-09-18 18:32:08,990 INFO sqlalchemy.engine.Engine SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
WHERE machines.reserved = false
2024-09-18 18:32:08,990 [sqlalchemy.engine.Engine] INFO: SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
WHERE machines.reserved = false
2024-09-18 18:32:08,991 INFO sqlalchemy.engine.Engine [generated in 0.00067s] {}
2024-09-18 18:32:08,991 [sqlalchemy.engine.Engine] INFO: [generated in 0.00067s] {}
2024-09-18 18:32:08,995 [modules.machinery.az] DEBUG: Trying <bound method VirtualMachineScaleSetVMsOperations.list of <azure.mgmt.compute.v2024_07_01.operations._operations.VirtualMachineScaleSetVMsOperations object at 0x7fab717add20>>(('XXX', 'vmss-cape-win10x64-60'),{})
2024-09-18 18:32:08,998 [modules.machinery.az] DEBUG: Trying <bound method NetworkInterfacesOperations.list_virtual_machine_scale_set_network_interfaces of <azure.mgmt.network.operations._operations.NetworkInterfacesOperations object at 0x7fab6fe0c040>>(('XXX', 'vmss-cape-win10x64-60'),{'api_version': '2024-07-01'})
2024-09-18 18:32:18,186 INFO sqlalchemy.engine.Engine SELECT tags.id AS tags_id, tags.name AS tags_name
FROM tags
WHERE tags.name = %(name_1)s
LIMIT %(param_1)s
2024-09-18 18:32:18,186 [sqlalchemy.engine.Engine] INFO: SELECT tags.id AS tags_id, tags.name AS tags_name
FROM tags
WHERE tags.name = %(name_1)s
LIMIT %(param_1)s
2024-09-18 18:32:18,187 INFO sqlalchemy.engine.Engine [generated in 0.00113s] {'name_1': 'win10x64', 'param_1': 1}
2024-09-18 18:32:18,187 [sqlalchemy.engine.Engine] INFO: [generated in 0.00113s] {'name_1': 'win10x64', 'param_1': 1}
2024-09-18 18:32:19,321 [modules.machinery.az] DEBUG: Machine vmss-cape-win10x64-60_0 was created and available in 0s
2024-09-18 18:32:22,244 INFO sqlalchemy.engine.Engine SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
2024-09-18 18:32:22,244 [sqlalchemy.engine.Engine] INFO: SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
2024-09-18 18:32:22,244 INFO sqlalchemy.engine.Engine [generated in 0.00080s] {}
2024-09-18 18:32:22,244 [sqlalchemy.engine.Engine] INFO: [generated in 0.00080s] {}
2024-09-18 18:32:22,248 INFO sqlalchemy.engine.Engine ROLLBACK
2024-09-18 18:32:22,248 [sqlalchemy.engine.Engine] INFO: ROLLBACK
2024-09-18 18:32:22,251 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-09-18 18:32:22,251 [sqlalchemy.engine.Engine] INFO: BEGIN (implicit)
2024-09-18 18:32:22,256 INFO sqlalchemy.engine.Engine SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
WHERE machines.locked = true AND machines.reserved = false
2024-09-18 18:32:22,256 [sqlalchemy.engine.Engine] INFO: SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
WHERE machines.locked = true AND machines.reserved = false
2024-09-18 18:32:22,256 INFO sqlalchemy.engine.Engine [generated in 0.00056s] {}
2024-09-18 18:32:22,256 [sqlalchemy.engine.Engine] INFO: [generated in 0.00056s] {}
2024-09-18 18:32:22,259 INFO sqlalchemy.engine.Engine COMMIT
2024-09-18 18:32:22,259 [sqlalchemy.engine.Engine] INFO: COMMIT
2024-09-18 18:32:22,260 [root] CRITICAL: CuckooCriticalError: No machines available
2024-09-18 18:32:38,638 [modules.machinery.az] DEBUG: Monitoring the machine pools...
2024-09-18 18:32:38,678 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-09-18 18:32:38,678 [sqlalchemy.engine.Engine] INFO: BEGIN (implicit)
2024-09-18 18:32:38,680 INFO sqlalchemy.engine.Engine SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
WHERE machines.reserved = false
2024-09-18 18:32:38,680 [sqlalchemy.engine.Engine] INFO: SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
WHERE machines.reserved = false
2024-09-18 18:32:38,683 INFO sqlalchemy.engine.Engine [cached since 29.69s ago] {}
2024-09-18 18:32:38,683 [sqlalchemy.engine.Engine] INFO: [cached since 29.69s ago] {}
2024-09-18 18:32:38,695 INFO sqlalchemy.engine.Engine SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id
WHERE tasks.status = %(status_1)s ORDER BY tasks.added_on DESC
2024-09-18 18:32:38,695 [sqlalchemy.engine.Engine] INFO: SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id
WHERE tasks.status = %(status_1)s ORDER BY tasks.added_on DESC
2024-09-18 18:32:38,703 INFO sqlalchemy.engine.Engine [cached since 353.3s ago] {'status_1': 'pending'}
2024-09-18 18:32:38,703 [sqlalchemy.engine.Engine] INFO: [cached since 353.3s ago] {'status_1': 'pending'}
2024-09-18 18:32:38,711 INFO sqlalchemy.engine.Engine SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
WHERE machines.reserved = false
2024-09-18 18:32:38,711 [sqlalchemy.engine.Engine] INFO: SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name
FROM machines LEFT OUTER JOIN (machines_tags AS machines_tags_1 JOIN tags AS tags_1 ON tags_1.id = machines_tags_1.tag_id) ON machines.id = machines_tags_1.machine_id
WHERE machines.reserved = false
2024-09-18 18:32:38,725 INFO sqlalchemy.engine.Engine [cached since 29.73s ago] {}
2024-09-18 18:32:38,725 [sqlalchemy.engine.Engine] INFO: [cached since 29.73s ago] {}
2024-09-18 18:32:38,731 [modules.machinery.az] DEBUG: Trying <bound method UsageOperations.list of <azure.mgmt.compute.v2024_07_01.operations._operations.UsageOperations object at 0x7fab720be980>>(('XXX',),{})
2024-09-18 18:32:39,030 [modules.machinery.az] DEBUG: Scaling vmsscape-win10x64-60 size from 0 -> 1
2024-09-18 18:32:39,045 [modules.machinery.az] DEBUG: Trying <bound method VirtualMachineScaleSetsOperations.get of <azure.mgmt.compute.v2024_07_01.operations._operations.VirtualMachineScaleSetsOperations object at 0x7fab7172f8e0>>(('XXX', 'vmsscape-win10x64-60'),{})
2024-09-18 18:32:39,133 [modules.machinery.az] DEBUG: Trying <bound method VirtualMachineScaleSetsOperations.begin_update of <azure.mgmt.compute.v2024_07_01.operations._operations.VirtualMachineScaleSetsOperations object at 0x7fab6fe8ec50>>(('XXX', 'vmss-cape-win10x64-60', <azure.mgmt.compute.v2024_07_01.models._models_py3.VirtualMachineScaleSet object at 0x7fab6fe8f490>),{'polling_interval': 1})
The text was updated successfully, but these errors were encountered: