0.4.0

This release contains MAJOR BREAKING CHANGES

Breaking Changes

1. Removed support for Elasticsearch Half Output (issue #110)
   • This was determined to not be worth continuing development
2. Config setting for esStandardized no longer supports, half or full. New supported value is ecs, which enables ECS standardized output
3. ECS output has completely changed from the Elasticsearch Full Output version, new schema below (issue #109)

Fields:

event.action - keyword
event.category - keyword
event.created - date
event.dataset - keyword
event.id - keyword
event.ingested - date
event.kind - keyword
event.module - keyword
event.outcome - keyword
event.provider - keyword
event.type - keyword
@\timestamp - date
file.created - date
file.directory - keyword
file.extension - keyword
file.mime_type - keyword
file.mtime - date
file.name - keyword
file.owner - keyword
file.path - keyword/text
file.size - long
file.type - keyword
file.hash.md5 - keyword
file.hash.sha256 - keyword
host.id - keyword
host.name - keyword
host.hostname - keyword
host.user.email - keyword
host.user.id - keyword
host.user.name - keyword
host.user.domain - keyword
host.ip - keyword
host.geo.status - keyword
host.geo.message - keyword
host.geo.continent_name - keyword
host.geo.continent_iso_code - keyword
host.geo.country_name - keyword
host.geo.country_iso_code - keyword
host.geo.region_name - keyword
host.geo.region_iso_code - keyword
host.geo.city_name - keyword
host.geo.district - keyword
host.geo.postal_code - keyword
host.geo.lat - float
host.geo.lon - float
host.geo.timezone - keyword
host.geo.currency - keyword
host.geo.isp - keyword
host.geo.org - keyword
host.geo.as - keyword
host.geo.as_name - keyword
host.geo.reverse - keyword
host.geo.mobile - bool
host.geo.proxy - bool
host.geo.hosting - bool
host.geo.query - string
host.geo.location - geo_point
code_42.event.id - keyword
code_42.event.type - keyword
code_42.event.timestamp - date
code_42.insertion_timestamp - date
code_42.file.path - keyword/text
code_42.file.name - keyword
code_42.file.type - keyword
code_42.file.category - keyword
code_42.file.identified_extension_category - keyword
code_42.file.current_extension_category - keyword
code_42.file.size - long
code_42.file.owner - keyword
code_42.file.hash.md5 - keyword
code_42.file.hash.sha256 - keyword
code_42.file.created_timestamp - date
code_42.file.modify_timestamp - date
code_42.file.id - keyword
code_42.file.identified_extension_mime_type - keyword
code_42.file.current_extension_mime_type - keyword
code_42.file.suspicious_file_type_mismatch - bool
code_42.device.username - keyword
code_42.device.uid - keyword
code_42.os_host_name - keyword
code_42.domain_name - keyword
code_42.public_ip_address - ip
code_42.private_ip_addresses - ip
code_42.actor - keyword
code_42.directory_id - keyword
code_42.source - keyword
code_42.url.full - keyword/text
code_42.url.domain - keyword
code_42.url.extension - keyword
code_42.url.fragment - keyword
code_42.url.path - keyword
code_42.url.port - long
code_42.url.query - keyword
code_42.url.scheme - keyword
code_42.url.username - keyword
code_42.url.password - keyword
code_42.url.registered_domain - keyword
code_42.url.top_level_domain - keyword
code_42.shared - bool
code_42.shared_with - keyword
code_42.sharing_type_added - keyword
code_42.cloud_drive_id - keyword
code_42.detection_source_alias - keyword
code_42.exposure - keyword
code_42.process.owner - keyword
code_42.process.name - keyword/text
code_42.tab.window_title - keyword/text
code_42.tab.url.full - keyword/text
code_42.tab.url.domain - keyword
code_42.tab.url.extension - keyword
code_42.tab.url.fragment - keyword
code_42.tab.url.path - keyword
code_42.tab.url.port - long
code_42.tab.url.query - keyword
code_42.tab.url.scheme - keyword
code_42.tab.url.username - keyword
code_42.tab.url.password - keyword
code_42.tab.url.registered_domain - keyword
code_42.tab.url.top_level_domain - keyword
code_42.removable_media.vendor - keyword
code_42.removable_media.name - keyword
code_42.removable_media.serial_number - keyword
code_42.removable_media.capacity - long
code_42.removable_media.bus_type - keyword
code_42.removable_media.media_name - keyword
code_42.removable_media.volume_name - keyword
code_42.removable_media.partition_id - keyword
code_42.sync_destination - keyword
code_42.sync_destination_username - keyword
code_42.email_dlp.policy_names - keyword
code_42.email_dlp.subject - keyword
code_42.email_dlp.sender - keyword
code_42.email_dlp.from - keyword
code_42.email_dlp.recipients - keyword
code_42.outside_active_hours - bool
code_42.print.job_name - keyword
code_42.print.printer_name - keyword
code_42.print.printed_files_backup_path - keyword
code_42.remote_activity - keyword
code_42.trusted - keyword
code_42.logged_in_operating_system_user - keyword
code_42.destination.category - keyword
code_42.destination.name - keyword/text

0.3.1

Updates:

• Updated ip-api-go-pkg to v0.0.9
• Updated go to 15.2
• Update crashplan-ffs-go-pkg to v0.0.9 (issue #106)

0.3.0

Updates:

• Updated crashplan-ffs-go-pkg to v0.2.1 to fix issue with SyncDestination being overwritten by SyncDestinationUser

0.2.9

This release contains changes to the Elasticsearch Half and Full Output format

Updates:

• Updated crashplan-ffs-go-pkg to v0.2.0:
  • Added support for new FFS field syncDestinationUsername:
    • FFS Output:
      • syncDestinationUsername - string
    • Elasticsearch Half Output:
      • sync_destination_username - string
    • Elasticsearch Full Output:
      • file.sync_destination_user - object (user)
      • file.sync_destination_user.id - string

0.2.8

This Release contains changes to the Elasticsearch Half and Full Output format

Updates:

• Updated go to v1.14.4
• Updated olivere/elastic to v7.0.17
• Updated crashplan-ffs-go-pkg to v0.1.9
  • Added support for new FFS field
    • New Field (FFS Output):
      • LoggedInOperatingSystemUser - string
    • New Field (Elasticsearch Half Output):
      • logged_in_operating_system_user - string
    • New Fields (Elasticsearch Full Output):
      • host.user - object
      • host.user.id - string

0.2.6

Enhancements:

• Removed tracking of vendor folder (issue #97)

Updates:

• Updated crashplan-ffs-go-pkg to v0.1.8 adds support for new FFS fields (issue #96)
  • New Fields (FFS Output):
    • RemoteActivity - string
    • Trusted - boolean
  • New Fields (Elasticsearch Half Output)
    • remote_activity - string
    • trusted - boolean
  • New Fields (Elasticsearch Full Output):
    • file.remote_activity - string
    • file.trusted - boolean

0.2.5

This Release contains changes to the Elasticsearch Full Output format

Enhancements:

• When using Elasticsearch Full Output format URLs are now broken down into ECS format (issue #93)
  • New Fields:
    • file.url.domain
    • file.url.extension
    • file.url.fragment
    • file.url.password
    • file.url.path
    • file.url.port
    • file.url.query
    • file.url.registered_domain
    • file.url.scheme
    • file.url.top_level_domain
    • file.url.username
    • tab.url.domain
    • tab.url.extension
    • tab.url.fragment
    • tab.url.password
    • tab.url.path
    • tab.url.port
    • tab.url.query
    • tab.url.registered_domain
    • tab.url.scheme
    • tab.url.top_level_domain
    • tab.url.username

Updates:

• Updated crashplan-ffs-go-pkg to v0.1.7, this adds support for new FFS fields (issue #94)
  • New Fields (FFS Output):
    • PrintJobName
    • PrinterName
    • PrintedFilesBackupPath
  • New Fields (Elasticsearch Half Output):
    • print_job_name
    • printer_name
    • printed_files_backup_path
  • New Fields (Elasticsearch Full Output):
    • printing.job
    • printing.printer.name
    • printing.printed_file_backup_path

Fixes:

• Fixed issue where max concurrent queries was 1 less than it should be (issue #92)
• Fixed issue where crashplan_ffs_puller_events_total was not being incremented (issue #91)

0.2.4

This update contains breaking changes to the configuration and potentially schema

Schema Changes:

• The event value "Shared", is now a boolean instead of a string
  • (Only affects default output and half output)

Changes:

• Reworked how IP-API settings work
  • Settings are no longer per FFS Query, instead it is on global setting
• Refactored a lot of the code to improve performance as well as maintainability.
• Panics are now thrown on config validation.
  • While this isn't pretty, until I get around to better validation, this will at least tell you where errors in config exist.

Fixes:

• Fixed an issue where inProgressQueries was not being properly updated, and could result in the incorrect inProgressQueries being saved.

Enhancements:

• Added the ability to have a local cache for IP-API
  • This reduces the total time it takes to enrich events with IP-API data, it is recommended that this be enabled in the config.
• Added the ability to rate limit the max concurrent queries for each FFS Query
  • This can be achieved by adding the config option "max_concurrent_queries" under each FFS Query config
    • Default: 5
    • Setting to 0 disables the FFS Query from running
    • Setting to -1 disables the rate limiting for the FFS Query
• Added the ability to load balance multiple Logstash (TCP) and Elasticsearch hosts
  • Note: This is very simple Random Loadbalancer with no real logic behind it, this will be enhanced in a future release

Vendoring:

• Updated crashplan-ffs-go-pkg to v0.1.6
  • Includes performance improvements
• Updated github.com/olivere/elastic/v7 to v7.0.14

0.2.3

BREAKING CHANGES
Important: This update contains major changes to the full elastic output format, please look at the new structure before updating

Enhancements:

• Added basic debugging toggle to allow you to enable/disable printing of IP lookups
• Some general code optimization

Updates:

• Added support for hosting field of IP-API
• Added support for new FFS Fields

Vendoring:

• Updated BenB196/crashplan-ffs-go-pkg to v0.1.3 (issue #80)
• Updated BenB196/ip-api-go-pkg to v0.0.8 (issue #81)
• Updated google/go-cmp to v0.4.0
• Updated olivere/elastic to v7.0.13
• Updated prometheus/client_golang to v1.5.1

0.2.2

Release v0.2.2

• [Enhancement] - Updated to golang v1.13.5 (Issue #78) - This should hopefully fix Issue #38 as it appears to be a bug in golang v1.12
• [Enhancement] - Updated crashplan-ffs-go-pkg to v0.1.1 (Issue #77)
  • Important - This includes a change to the EmailDLPPolicyName field, which is now called: EmailDLPPolicyNames. If using Elasticsearch, either an index update is required, or you will need to use something to convert the field name back to EmailDLPPolicyName.
• [Bug] - Handling of extraneous or missing " in quoted-field should be properly handled and the query retried. (Issue #76)
  • Note - I am not sure why this occurs, if you are to require the offending data, it will be correct.