Skip to content

Latest commit

 

History

History
461 lines (342 loc) · 29.1 KB

CHANGELOG.next.asciidoc

File metadata and controls

461 lines (342 loc) · 29.1 KB
Raw

Beats version HEAD

Breaking changes

Affecting all Beats

  • Fix FQDN being lowercased when used as host.hostname 39993

  • Beats won’t log start up information when running under the Elastic Agent {40390}40390[40390]

  • Filebeat now needs dup3, faccessat2, prctl and setrlimit syscalls to run the journald input. If this input is not being used, the syscalls are not needed. All Beats have those syscalls allowed now because the default seccomp policy is global to all Beats. 40061

  • Beats will rate limit the logs about errors when indexing events on Elasticsearch, logging a summary every 10s. The logs sent to the event log is unchanged. 40157

  • Drop support for Debian 10 and upgrade statically linked glibc from 2.28 to 2.31 41402

  • Fix metrics not being ingested, due to "Limit of total fields [10000] has been exceeded while adding new fields […​]". The total fields limit has been increased to 12500. No significant performance impact on Elasticsearch is anticipated. 41640

  • Set default kafka version to 2.1.0 in kafka output and filebeat. 41662

Auditbeat

Filebeat

  • Convert netflow input to API v2 and disable event normalisation 37901

  • Removed deprecated Squid from Beats. See [migrate-from-deprecated-module] for migration options. 38037

  • Removed deprecated Sonicwall from Beats. Use the SonicWall Firewall Elastic integration instead. 38037

  • Removed deprecated Radware from Beats. See [migrate-from-deprecated-module] for migration options. 38037

  • Removed deprecated Netscout from Beats. See [migrate-from-deprecated-module] for migration options. 38037

  • Removed deprecated Juniper Netscreen from Beats. See [migrate-from-deprecated-module] for migration options. 38037

  • Removed deprecated Impreva from Beats. See [migrate-from-deprecated-module] for migration options. 38037

  • Removed deprecated Cylance from Beats. See [migrate-from-deprecated-module] for migration options. 38037

  • Removed deprecated Bluecoat from Beats. See [migrate-from-deprecated-module] for migration options. 38037

  • Introduce input/netmetrics and refactor netflow input metrics 38055

  • Update Salesforce module to use new Salesforce input. 37509

  • Tag events that come from a filestream in "take over" mode. 39828

  • Fix high IO and handling of a corrupted registry log file. 35893

  • Enable file ingestion to report detailed status to Elastic Agent 40075

  • Filebeat, when running with Elastic-Agent, reports status for Filestream input. 40121

  • Fix filestream’s registry GC: registry entries will never be removed if clean_inactive is set to "-1". 40258

  • Added ignore_empty_values flag in decode_cef Filebeat processor. 40268

  • Added support for hyphens in extension keys in decode_cef Filebeat processor. 40427

  • Journald: removed configuration options include_matches.or, include_matches.and, backoff, max_backoff, cursor_seek_fallback. 40061

  • Journald: include_matches.match now behaves in the same way as matchers in journalctl. Users should carefully update their input configuration. 40061

  • Journald: seek and since behaviour have been simplified, if there is a cursor (state) seek and since are ignored and the cursor is used. 40061

  • Redis: Added replication role as a field to submitted slowlogs

  • Added container.image.name to journald Filebeat input’s Docker-specific translated fields. 40450

  • Change log.file.path field in awscloudwatch input to nested object. 41099

  • Remove deprecated awscloudwatch field from Filebeat. 41089

  • The performance of ingesting SQS data with the S3 input has improved by up to 60x for queues with many small events. max_number_of_messages config for SQS mode is now ignored, as the new design no longer needs a manual cap on messages. Instead, use number_of_workers to scale ingestion rate in both S3 and SQS modes. The increased efficiency may increase network bandwidth consumption, which can be throttled by lowering number_of_workers. It may also increase number of events stored in memory, which can be throttled by lowering the configured size of the internal queue. 40699

  • Fixes filestream logging the error "filestream input with ID 'ID' already exists, this will lead to data duplication[…​]" on Kubernetes when using autodiscover. 41585

  • Add kafka compression support for ZSTD.

Heartbeat

Metricbeat

  • Setting period for counter cache for Prometheus remote_write at least to 60sec 38553

  • Remove fallback to the node limit for the kubernetes.pod.cpu.usage.limit.pct and kubernetes.pod.memory.usage.limit.pct metrics calculation

  • Add support for Kibana status metricset in v8 format 40275

  • Mark system process metricsets as running if metrics are partially available 40565

  • Added back elasticsearch.node.stats.jvm.mem.pools.* to the node_stats metricset 40571

  • Add GCP organization and project details to ECS cloud fields. 40461

  • Add support for specifying a custom endpoint for GCP service clients. 40848 40918

  • Fix incorrect handling of types in SQL module. 40090 41607

Osquerybeat

  • Add action responses data stream, allowing osquerybeat to post action results directly to elasticsearch. 39143

  • Disable allow_unsafe osquery configuration. 40130

  • Upgrade to osquery 5.12.1. 40368

  • Upgrade to osquery 5.13.1. 40849

Osquerybeat

Packetbeat

  • Use base-16 for reporting serial_number value in TLS fields in line with the ECS recommendation. 41542

  • Expire source port mappings. 41581

Winlogbeat

  • Add "event.category" and "event.type" to Sysmon module for EventIDs 8, 9, 19, 20, 27, 28, 255 35193

Functionbeat

Elastic Logging Plugin

Bugfixes

Affecting all Beats

  • Support for multiline zookeeper logs 2496

  • Add checks to ensure reloading of units if the configuration actually changed. 34346

  • Fix namespacing on self-monitoring 32336

  • Fix namespacing on self-monitoring 32336

  • Fix Beats started by agent do not respect the allow_older_versions: true configuration flag 34227 34964

  • Fix performance issues when we have a lot of inputs starting and stopping by allowing to disable global processors under fleet. 35000 35031

  • 'add_cloud_metadata' processor - add cloud.region field for GCE cloud provider

  • 'add_cloud_metadata' processor - update azure metadata api version to get missing cloud.account.id field

  • Upgraded apache arrow library used in x-pack/libbeat/reader/parquet from v11 to v12.0.1 in order to fix cross-compilation issues 35640

  • Fix panic when MaxRetryInterval is specified, but RetryInterval is not 35820

  • Support build of projects outside of beats directory 36126

  • Support Elastic Agent control protocol chunking support 37343

  • Lower logging level to debug when attempting to configure beats with unknown fields from autodiscovered events/environments 37816[37816]

  • Set timeout of 1 minute for FQDN requests 37756

  • Fix issue where old data could be saved in the memory queue after acknowledgment, increasing memory use 41356

  • Ensure Elasticsearch output can always recover from network errors 40794

  • Add translate_ldap_attribute processor. 41472

  • Remove unnecessary debug logs during idle connection teardown 40824

Auditbeat

Filebeat

  • [Gcs Input] - Added missing locks for safe concurrency 34914

  • Fix the ignore_inactive option being ignored in Filebeat’s filestream input 34770

  • Fix TestMultiEventForEOFRetryHandlerInput unit test of CometD input 34903

  • Add input instance id to request trace filename for httpjson and cel inputs 35024

  • Fixes "Can only start an input when all related states are finished" error when running under Elastic-Agent 35250 33653

  • [system] sync system/auth dataset with system integration 1.29.0. 35581

  • [GCS Input] - Fixed an issue where bucket_timeout was being applied to the entire bucket poll interval and not individual bucket object read operations. Fixed a map write concurrency issue arising from data races when using a high number of workers. Fixed the flaky tests that were present in the GCS test suit. 35605

  • Fixed concurrency and flakey tests issue in azure blob storage input. 35983 36124

  • Fix panic when sqs input metrics getter is invoked 36101 36077

  • Fix handling of Juniper SRX structured data when there is no leading junos element. 36270 36308

  • Fix Filebeat Cisco module with missing escape character 36325 36326

  • Added a fix for Crowdstrike pipeline handling process arrays 36496

  • [threatintel] MISP pagination fixes 37898

  • Fix file handle leak when handling errors in filestream 37973

  • Fix a race condition that could crash Filebeat with a "negative WaitGroup counter" error 38094

  • Fix "failed processing S3 event for object key" error on aws-s3 input when key contains the "+" character 38012 38125

  • Fix filebeat gcs input panic 38407

  • Fix filestream’s registry GC: registry entries are now removed from the in-memory and disk store when they’re older than the set TTL 36761 38488

  • Fix filestream’s registry GC: registry entries are now removed from the in-memory and disk store when they’re older than the set TTL 36761 38488

  • [threatintel] MISP splitting fix for empty responses 38739 38917

  • Prevent GCP Pub/Sub input blockage by increasing default value of max_outstanding_messages 35029 38985

  • Updated Websocket input title to align with existing inputs 39006

  • Restore netflow input on Windows 39024

  • Upgrade azure-event-hubs-go and azure-storage-blob-go dependencies. 38861

  • Fix request trace filename handling in http_endpoint input. 39410

  • Upgrade github.com/hashicorp/go-retryablehttp to mitigate CVE-2024-6104 40036

  • Fix for Google Workspace duplicate events issue by adding canonical sorting over fingerprint keys array to maintain key order. 40055 39859

  • Fix handling of deeply nested numeric values in HTTP Endpoint CEL programs. 40115

  • Prevent panic in CEL and salesforce inputs when github.com/hashicorp/go-retryablehttp exceeds maximum retries. 40144

  • Fix bug in CEL input rate limit logic. 40106 40270

  • Relax requirements in Okta entity analytics provider user and device profile data shape. 40359

  • Fix bug in Okta entity analytics rate limit logic. 40106 40267

  • Fix crashes in the journald input. 40061

  • Fix order of configuration for EntraID entity analytics provider. 40487

  • Ensure Entra ID request bodies are not truncated and trace logs are rotated before 100MB. 40494

  • The Elasticsearch output now correctly logs the event fields to the event log file 40509 40512

  • Fix the "No such input type exist: 'azure-eventhub'" error on the Windows platform 40608 40609

  • awss3 input: Fix handling of SQS notifications that don’t contain a region. 40628

  • Fix credential handling when workload identity is being used in GCS input. 39977 40663

  • Fix publication of group data from the Okta entity analytics provider. 40681

  • Ensure netflow custom field configuration is applied. 40735 40730

  • Fix replace processor handling of zero string replacement validation. 40751

  • Fix long filepaths in diagnostics exceeding max path limits on Windows. 40909

  • Add backup and delete for AWS S3 polling mode feature back. 41071

  • Fix a bug in Salesforce input to only handle responses with 200 status code 41015

  • Fixed failed job handling and removed false-positive error logs in the GCS input. 41142

  • Bump github.com/elastic/go-sfdc dependency used by x-pack/filebeat/input/salesforce. 41192

  • Log bad handshake details when websocket connection fails 41300

  • Improve modification time handling for entities and entity deletion logic in the Active Directory entityanalytics input. 41179

  • Journald input now can read events from all boots 41083 41244

  • Fix double encoding of client_secret in the Entity Analytics input’s Azure Active Directory provider 41393

  • Fix aws region in aws-s3 input s3 polling mode. 41572

  • Fix errors in SQS host resolution in the aws-s3 input when using custom (non-AWS) endpoints. 41504

  • The azure-eventhub input now correctly reports its status to the Elastic Agent on fatal errors 41469

  • Add support for Access Points in the aws-s3 input. 41495

  • Fix the "No such input type exist: 'salesforce'" error on the Windows/AIX platform. 41664

  • Fix missing key in streaming input logging. 41600

Heartbeat

Metricbeat

  • Fix Azure Monitor 429 error by causing metricbeat to retry the request again. 38294

  • Fix fields not being parsed correctly in postgresql/database 25301 37720

  • rabbitmq/queue - Change the mapping type of rabbitmq.queue.consumers.utilisation.pct to scaled_float from long because the values fall within the range of [0.0, 1.0]. Previously, conversion to integer resulted in reporting either 0 or 1.

  • Fix timeout caused by the retrival of which indices are hidden 39165

  • Fix Azure Monitor support for multiple aggregation types 39192 39204

  • Fix handling of access errors when reading process metrics 39627

  • Fix behavior of cgroups path discovery when monitoring the host system from within a container 39627

  • Fix issue where beats may report incorrect metrics for its own process when running inside a container 39627

  • Normalize AWS RDS CPU Utilization values before making the metadata API call. 39664

  • Fix behavior of pagetypeinfo metrics 39985

  • Fix query logic for temp and non-temp tablespaces in Oracle module. 38051 39787

  • Set GCP metrics config period to the default (60s) when the value is below the minimum allowed period. 30434 40020

  • Fix statistic methods for metrics collected for SQS. 40207

  • Add GCP 'instance_id' resource label in ECS cloud fields. 40033 40062

  • Fix missing metrics from CloudWatch when include_linked_accounts set to false. 40071 40135

  • Update beat module with apm-server monitoring metrics fields 40127

  • Fix Azure Monitor metric timespan to restore Storage Account PT1H metrics 40376 40367

  • Remove excessive info-level logs in cgroups setup 40491

  • Add missing ECS Cloud fields in GCP metrics metricset when using exclude_labels: true 40437 40467

  • Add AWS OwningAccount support for cross account monitoring 40570 40691

  • Use namespace for GetListMetrics when exists in AWS 41022

  • Fix http server helper SSL config. 39405

  • Fix Kubernetes metadata sometimes not being present after startup 41216

  • Do not report non-existant 0 values for RSS metrics in docker/memory 41449

  • Log Cisco Meraki getDevicePerformanceScores errors without stopping metrics collection. 41622

Osquerybeat

Packetbeat

Winlogbeat

Elastic Logging Plugin

Added

Affecting all Beats

  • Added append Processor which will append concrete values or values from a field to target. 29934 33364

  • dns processor: Add support for forward lookups (A, AAAA, and TXT). 11416 36394

  • [Enhanncement for host.ip and host.mac] Disabling netinfo.enabled option of add-host-metadata processor 36506

  • allow queue configuration settings to be set under the output. 35615 36788

  • Beats will now connect to older Elasticsearch instances by default 36884

  • Raise up logging level to warning when attempting to configure beats with unknown fields from autodiscovered events/environments

  • elasticsearch output now supports idle_connection_timeout. 35615 36843

  • Enable early event encoding in the Elasticsearch output, improving cpu and memory use 38572

  • The environment variable BEATS_ADD_CLOUD_METADATA_PROVIDERS overrides configured/default add_cloud_metadata providers 38669

  • When running under Elastic-Agent Kafka output allows dynamic topic in topic field 40415

  • The script processor has a new configuration option that only uses the cached javascript sessions and prevents the creation of new javascript sessions.

  • Update to Go 1.22.7. 41018

  • Replace Ubuntu 20.04 with 24.04 for Docker base images 40743 40942

  • Reduce memory consumption of k8s autodiscovery and the add_kubernetes_metadata processor when Deployment metadata is enabled

  • Add lowercase processor. 22254 41424

  • Add uppercase processor. 22254 41535

  • Replace compress/gzip with https://github.com/klauspost/compress/gzip library for gzip compression 41584

Auditbeat

  • Added add_session_metadata processor, which enables session viewer on Auditbeat data. 37640

  • Add linux capabilities to processes in the system/process. 37453

  • Add linux capabilities to processes in the system/process. 37453

  • Add process.entity_id, process.group.name and process.group.id in add_process_metadata processor. Make fim module with kprobes backend to always add an appropriately configured add_process_metadata processor to enrich file events 38776

Auditbeat

  • Improve logging in system/socket 41571

Auditbeat

Filebeat

  • add documentation for decode_xml_wineventlog processor field mappings. 32456

  • httpjson input: Add request tracing logger. 32402 32412

  • Add cloudflare R2 to provider list in AWS S3 input. 32620

  • Add support for single string containing multiple relation-types in getRFC5988Link. 32811

  • Added separation of transform context object inside httpjson. Introduced new clause .parent_last_response.* 33499

  • Added metric sqs_messages_waiting_gauge for aws-s3 input. 34488

  • Add nginx.ingress_controller.upstream.ip to related.ip 34645 34672

  • Add unix socket log parsing for nginx ingress_controller 34732

  • Added metric sqs_worker_utilization for aws-s3 input. 34793

  • Add MySQL authentication message parsing and related.ip and related.user fields 34810

  • Add nginx ingress_controller parsing if one of upstreams fails to return response 34787

  • Add oracle authentication messages parsing 35127

  • Add clean_session configuration setting for MQTT input. 16204

  • Add support for a simplified input configuraton when running under Elastic-Agent 36390

  • Added support for Okta OAuth2 provider in the CEL input. 36336 36521

  • Added support for new features & removed partial save mechanism in the Azure Blob Storage input. 35126 36690

  • Added support for new features and removed partial save mechanism in the GCS input. 35847 36713

  • Use filestream input with file_identity.fingerprint as default for hints autodiscover. 35984 36950

  • Add setup option --force-enable-module-filesets, that will act as if all filesets have been enabled in a module during setup. 30915 99999

  • Made Azure Blob Storage input GA and updated docs accordingly. 37128

  • Made GCS input GA and updated docs accordingly. 37127

  • Add parseDateInTZ value template for the HTTPJSON input 37738

  • Improve rate limit handling by HTTPJSON 36207 38161 38237

  • Parse more fields from Elasticsearch slowlogs 38295

  • added benchmark input 37437

  • added benchmark input and discard output 37437

  • Update CEL mito extensions to v1.11.0 to improve type checking. 39460

  • Update CEL mito extensions to v1.12.2. 39755

  • Add support for base64-encoded HMAC headers to HTTP Endpoint. 39655

  • Add user group membership support to Okta entity analytics provider. 39814 39815

  • Add request trace support for Okta and EntraID entity analytics providers. 39821

  • Fix handling of infinite rate values in CEL rate limit handling logic. 39940

  • Allow elision of set and append failure logging. 34544 39929

  • Add ability to remove request trace logs from CEL input. 39969

  • Add ability to remove request trace logs from HTTPJSON input. 40003

  • Added out of the box support for Amazon EventBridge notifications over SQS to S3 input 40006

  • Update CEL mito extensions to v1.13.0 40035

  • Add Jamf entity analytics provider. 39996

  • Add ability to remove request trace logs from http_endpoint input. 40005

  • Add ability to remove request trace logs from entityanalytics input. 40004

  • Relax constraint on Base DN in entity analytics Active Directory provider. 40054

  • Implement Elastic Agent status and health reporting for Netflow Filebeat input. 40080

  • Enhance input state reporting for CEL evaluations that return a single error object in events. 40083

  • Allow absent credentials when using GCS with Application Default Credentials. 39977 40072

  • Add SSL and username support for Redis input, now the input includes support for Redis 6.0+. 40111

  • Add scaling up support for Netflow input. 37761 40122

  • Update CEL mito extensions to v1.15.0. 40294

  • Allow cross-region bucket configuration in s3 input. 22161 40309

  • Improve logging in Okta Entity Analytics provider. 40106 40347

  • Document winlog input. 40074 40462

  • Added retry logic to websocket connections in the streaming input. 40271 40601

  • Disable event normalization for netflow input 40635

  • Allow attribute selection in the Active Directory entity analytics provider. 40482 40662

  • Improve error quality when CEL program does not correctly return an events array. 40580

  • Added support for Microsoft Entra ID RBAC authentication. 40434 40879

  • Add use_kubeadm config option for filebeat (both filbeat.input and autodiscovery) in order to toggle kubeadm-config api requests 40301

  • Make HTTP library function inclusion non-conditional in CEL input. 40912

  • Add support for Crowdstrike streaming API to the streaming input. 40264 40838

  • Add support to CEL for reading host environment variables. 40762 40779

  • Add CSV decoder to awss3 input. 40896

  • Change request trace logging to include headers instead of complete request. 41072

  • Improved GCS input documentation. 41143

  • Add CSV decoding capacity to azureblobstorage input 40978

  • Add CSV decoding capacity to gcs input 40979

  • Add support to source AWS cloudwatch logs from linked accounts. 41188

  • Jounrald input now supports filtering by facilities 41061

  • Add support to include AWS cloudwatch linked accounts when using log_group_name_prefix to define log group names. 41206

  • Improved Azure Blob Storage input documentation. 41252

  • Make ETW input GA. 41389

  • Added input metrics to GCS input. 36640 41505

  • Add support for Okta entity analytics provider to collect role and factor data for users. 41460

  • Add support for Journald in the System module. 41555

  • Add ability to remove request trace logs from http_endpoint input. 40005

  • Add ability to remove request trace logs from entityanalytics input. 40004

  • Update CEL mito extensions to v1.16.0. 41727

Auditbeat

Libbeat

  • enrich events with EC2 tags in add_cloud_metadata processor 41477

Heartbeat

  • Added status to monitor run log report.

  • Upgrade node to latest LTS v18.20.3. 40038

Metricbeat

  • Add per-thread metrics to system_summary 33614

  • Add GCP CloudSQL metadata 33066

  • Add GCP Carbon Footprint metricbeat data 34820

  • Add event loop utilization metric to Kibana module 35020

  • Add metrics grouping by dimensions and time to Azure app insights 36634

  • Align on the algorithm used to transform Prometheus histograms into Elasticsearch histograms 36647

  • Add linux IO metrics to system/process 37213

  • Add new memory/cgroup metrics to Kibana module 37232

  • Add SSL support to mysql module 37997

  • Add SSL support for aerospike module 38126

  • Add use_kubeadm config option in kubernetes module in order to toggle kubeadm-config api requests 40086

  • Log the total time taken for GCP ListTimeSeries and AggregatedList requests 40661

  • Add new metrics for the vSphere Host metricset. 40429

  • Add new metrics for the vSphere Datastore metricset. 40441

  • Add new metricset cluster for the vSphere module. 40536

  • Add new metricset network for the vSphere module. 40559

  • Add new metricset resourcepool for the vSphere module. 40456

  • Add AWS Cloudwatch capability to retrieve tags from AWS/ApiGateway resources 40755

  • Add new metricset datastorecluster for vSphere module. 40634

  • Add support for new metrics in datastorecluster metricset. 40694

  • Add new metrics for the vSphere Virtualmachine metricset. 40485

  • Add support for snapshot in vSphere virtualmachine metricset 40683

  • Update fields to use mapstr in vSphere virtualmachine metricset 40707

  • Add metrics related to triggered alarms in all the vSphere metricsets. 40714 40876

  • Add support for period based intervalID in vSphere host and datastore metricsets 40678

  • Add new metrics fot datastore and minor changes to overall vSphere metrics 40766

  • Add metrics_count to Prometheus module if metrics_count: true is set. 40411

  • Added Cisco Meraki module 40836

  • Added Palo Alto Networks module 40686

  • Restore docker.network.in.* and docker.network.out.* fields in docker module 40968

  • Add id field to all the vSphere metricsets. 41097

  • Bump aerospike-client-go to version v7.7.1 and add support for basic auth in Aerospike module 41233

  • Only watch metadata for ReplicaSets in metricbeat k8s module 41289

  • Add support for region/zone for Vertex AI service in GCP module 41551

Metricbeat

Osquerybeat

Packetbeat

Winlogbeat

  • Add handling for missing `EvtVarType`s in experimental api. 19337 41418

  • Implement exclusion range support for event_id. 38623 41639

Functionbeat

Elastic Log Driver Elastic Logging Plugin

Deprecated

Auditbeat

Filebeat

Heartbeat

Metricbeat

Osquerybeat

Packetbeat

Winlogbeat

Functionbeat

Elastic Logging Plugin

Known Issues