Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle unknown notification formats #23

Open
bernhard-herzog opened this issue Jul 6, 2016 · 5 comments
Open

Handle unknown notification formats #23

bernhard-herzog opened this issue Jul 6, 2016 · 5 comments

Comments

@bernhard-herzog
Copy link
Member

We need to decide on how to handle notifications for which mailgen cannot determine the format to use for the message. E.g. if a notification specifies a format which mailgen simply does not know about, what should mailgen do?
bernhard-herzog added a commit that referenced this issue Jul 6, 2016
@bernhard-herzog
Simplify the format support radically to only "feed_specific"
a055bd5 
For now we only need the feed_specific format and in this step only for
the two feeds for which formats had already been defined in
intelmq-mailgen. Making this very simple hopefully will make it easier
to implement the more complex logic we actually need later on.

This solves several defects:

 - The logic that selects the feed specific CSV format now actually
   works (fixes #20)

 - The CSV format for Sinkhole-HTTP-Drone is now the same as that for
   Botnet-Drone-Hadoop. The former was apparently never tested because
   it had a typo in one of the IntelMQ identifiers (destination.id
   instead of destination.ip)

 - The template is chosen by intelmq-mailgen now. With the way the
   contact DB currently works, this is the only way to get the right
   template. The template depends on the feed-name which the contact DB
   does not know anything about at the moment.

   This will have to change in some way, but for now we get something
   that should actually work.

   How the template name is constructed is described in README.md

Other consequences:

 - There are no fallback formats anymore. It's not clear how to handle
   cases where no message can be generated because the format is
   unknown. We need a better way than what we get with this commit
   (mailgen produces an exception and the notification remains in the
   database as unsent) but simply sending a very generic mail as
   previously is probably not the right solution either. That solution
   was an ad-hoc fix/workaround for a defect in earlier versions.

   See #23
@bernhardreiter
Copy link
Member

My ideas:

  • whatever happens, there should be no silent fail that would result into no email to be send, but marked and notified
  • We have two possibilies
  1. fail (in the logs or so) and leave the notification(s) marked as unsent
  2. fallback to the most generic dataformat, that could be plain/text (fallback template) and csv with fallback columns (that may exclude some fields).

@dmth
Copy link
Contributor

dmth commented Jul 7, 2016

whatever happens, there should be no silent fail that would result into no email to be send, but marked and notified

I agree.

I tend to number 1) maybe this is an approach:

  • Create a Ticketnumber and write it to the notifications table. Do NOT set the sent_at value.
  • Log this Ticketnumber to an appropriate logfile, with the reason why the mail could not be sent,
  • Inform the user that some mails could not be sent (maybe as stderr), and mention the log file as source for details.

This has some advantages:

  • No messages are sent which might be unusable
  • Messages which have not been sent can be identified in the notifications-table
  • Messages can be resend, once a configuration was created.
  • The user is was informed about the problem

@bernhardreiter
Copy link
Member

@dmth I see a number of issues with your technical ideas for 1)
We aim for mailgen to run non-interactively so there is no user to inform, the log file will have it.
There I believe operations will have to review logfiles regularily (anyway) so using a high enough
loglevel should be fine. A ticket number may not be necessary because we could already see which noficiations have not gone out, together will the failed log-message that will be enough to identify the nofications in question.

@dmth
Copy link
Contributor

dmth commented Jul 7, 2016 •
edited
Loading

We aim for mailgen to run non-interactively so there is no user to inform, the log file will have it.

There is always a user to inform. At least root.

There I believe operations will have to review logfiles regularily (anyway) so using a high enough
loglevel should be fine.

I've no Idea how you want to use loglevels in this case.

A ticket number may not be necessary because we could already see which noficiations have not gone out, together will the failed log-message that will be enough to identify the nofications in question.

It's about referencing to the notification more easily. There will be a very very high amount of notifications some day. I don't want the user to search for failed notifications in the notifications table which occured around "yesterday at 11 o'clock". A ticketnumber is much more specific in this case and allows operations like ´intelmq-mailgen --resend 4711´ once the missing information was repaired.

@bernhardreiter
Copy link
Member

bernhardreiter commented Jul 7, 2016 •
edited
Loading

@dmth to explain

  • if you use log.critical some logfiles will only have those messages, so it can be easily spotted
  • our system should be build so that no critical situation happens, if it does, we have a software defect (or an admin who made a mistake). Anyway it is a systematic problem, not one that affects single notations.
  • if the format "xyz" fails which can be seen in the log message, an SQL selection with this restriction can easily be done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dmth @bernhardreiter @bernhard-herzog