Add Audit Logfile Capability

[patcamen]

It would be great for auditing purposes if the ldap2pg tool could write verbose output to a logfile with timestamps - while it may retain the current output format without timestamps for the console for readability purposes.
Alternatively a flag to just enable timestamps in the current verbose console output would work also - in which case the output can then be redirected to a file if needed. An ISO 8601 formatted timestamp with millisecond granularity (e.g. 2021-11-10 15:52:10.200) should be good enough for a start.
This will also help in a corporate context where logfiles are often consumed by tools such as Splunk. They often struggle to parse messages correctly without timestamps - i.e. multiple log lines may end up being treated as one single message.

[bersace]

Hi @patcamen . Interesting. I'm willing to make ldap2pg a good citizen with splunk and other auditing tools. Can you forge some sample of log lines you expect ?

For auditing, do you need only CHANGES, WARNING, INFO and ERROR messages ? Please share me a sample audit file you would like ldap2pg to generate. Or do you need the full verbose log for auditing ? Do you need to configure auditing level and format ?

Do you want a cli swithc like ldap2pg --config my-ldap2pg.yml --audit-file my-ldap2pg-audit.log ? That would be independant from color and verbosity.

Another option would be to configure auditing in yml file:

auditlog:
  destination: /var/log/audit/ldap2pg.log
  level: DEBUG

sync_map:
 ...

That would be still configurable through env vars AUDITLOG_DESTINATION, etc.

Thanks for the feedback.

[patcamen]

Hi @bersace ,
attached you find a mocked sample for what would be my ideal audit log format - ldap2pg_audit_log_sample.txt
Note: the real audit log would not feature the same timestamp on all lines of course.

The sample contains a subset of the current DEBUG messages only that I would consider as audit relevant plus all INFO level and higher messages.
The idea being to prove the exact changes that were applied to the PostgreSQL server and why (based on what information found in LDAP and YAML file).
If it was too complex to log only certain DEBUG messages to the audit file it would be also fine to log all current DEBUG messages instead.
As shown in the attached sample it would be great if the INFO LDAP coordinates would be logged in full length to the audit file - as in the console they currently get cropped. Alternatively the current DEBUG ldapsearch messages would be needed to evidence what exactly was queried from LDAP.

If I were to chose, I would probably want the audit log to be configured inside the yml file like you have shown above - as indeed other users might want to chose a different log level to go to the audit log. Also like this the audit log would be decoupled from the console verbosity set.

Thanks.

[bersace]

hi @patcamen . re-reading your sample audit log, it looks like you suggest to increase the level of some messages and decrease the level of other. That's a good feedback.

ldap2pg 6.0 has a new log format in logfmt. Almost all messages has been reviewed, including their level. I saw that some debug message of ldap2pg v5 are now at INFO level like you suggested.

It would be very helpful if you could point me DEBUG log level you'd prefer at INFO and vice versa, for ldap2pg 6. Would you ?

I prefer to make ldap2pg INFO message enougth for an audit log rather than a new auditlog feature.

[bersace]

Maybe a switch option would raise some debug message like ldapsearch commands and query to INFO level.