Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No alert when MISP is timing out for correlation / enrichment #26

Open
romainw opened this issue Mar 7, 2024 · 1 comment
Open

No alert when MISP is timing out for correlation / enrichment #26

romainw opened this issue Mar 7, 2024 · 1 comment
Labels
bug Something isn't working enhancement New feature or request help wanted Extra attention is needed
Milestone
v0.0.4

Comments

@romainw
Copy link
Member

romainw commented Mar 7, 2024

Launching a search query for a domain or IP on a MISP instance is resource intensive and often cause the query to timeout.

Currently, pDNSSOC-cli stops processing the alert when this happens.
Instead, it should proceed and create an alert in alerts/matches.json, just with the information available and an indication that MISP timed out (again).

CRITICAL:pymisp:Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.4.182 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'MISP-session-af16f=<SNIP>', 'Content-Length': '317', 'content-type': 'application/json'}
Request body:
{"returnFormat": "json", "type": "domain", "withAttachments": 0, "metadata": 0, "enforceWarninglist": 1, "to_ids": 1, "includeEventUuid": 0, "includeEventTags": 0, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0, "excludeDecayed": 0}
Response (if any):
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

Exception in thread Thread-3 (daemonized_retro):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pdnssoccli/subcommands/daemonize.py", line 37, in daemonized_retro
    ctx.invoke(correlate, **{'retro_lookup': True, 'files':[correlation_config['archive_dir']]})
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pdnssoccli/subcommands/correlate.py", line 163, in correlate
    attributes = misp.search(controller='attributes', type_attribute='domain', to_ids=1, pythonify=True, **args)
  File "/usr/local/lib/python3.10/dist-packages/pymisp/api.py", line 2722, in search
    normalized_response = self._check_json_response(response)
  File "/usr/local/lib/python3.10/dist-packages/pymisp/api.py", line 3658, in _check_json_response
    r = self._check_response(response, expect_json=True)
  File "/usr/local/lib/python3.10/dist-packages/pymisp/api.py", line 3676, in _check_response
    raise MISPServerError(f'Error code 500:\n{response.text}')
pymisp.exceptions.MISPServerError: Error code 500:
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
@arvchristos
Copy link
Collaborator

Nice catch, this is indeed something we should implement! Thank you for the issue @romainw

@arvchristos arvchristos added bug Something isn't working enhancement New feature or request help wanted Extra attention is needed labels Mar 8, 2024
@arvchristos arvchristos added this to the v0.0.4 milestone Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
v0.0.4
Development

No branches or pull requests
2 participants
@romainw @arvchristos