v0.25.0

Visit the Brim Download page to find the package for your OS platform.

• zqd: Update Zeek pointer to v3.2.1-brim6 which provides the latest geolocation data (#1795)
• zqd: Update Suricata pointer to v5.0.3-brimpre2 to generate alerts for imported pcaps (#1729)
• zqd: Make some columns more prominent (moved leftward) in Suricata alert records (#1749)
• zq: Fix an issue where returned errors could cause a panic due to type mismatches (#1720, #1727, #1728, #1740, #1773)
• python: Fix an issue where the Python client did not generate an error when zqd was absent (#1711)
• zql: Allow the len() function to work on ip and net types (#1725)
• zson: Add a draft specification of the new ZSON format (#1715, #1735, #1741, #1765)
• zng: Add support for marshaling of time values (#1743)
• zar: Fix an issue where a couldn't read trailer failure was observed during a zar zq query (#1748)
• zar: Fix an issue where zar import of a 14 GB data set triggered a SEGV (#1766)
• zql: Add a new drop processor, which replaces cut -c (#1773)
• zql: Add a new pick processor, which acts like a stricter cut (#1773, #1788)
• zqd: Improve performance when listing Spaces via the API (#1779, #1786)

v0.24.0

Visit the Brim Download page to find the package for your OS platform.

• zq: Update Zeek pointer to v3.2.1-brim5 which provides the latest geolocation data (#1713)
• zql: For functions, introduce "snake case" names and deprecate package syntax (#1575, #1609)
• zql: Add a cut() function (#1585)
• zar: Allow zar import of multiple paths (#1582)
• zar: Fix an issue where a bare word zar zq search could cause a panic (#1590)
• zq: Update Go dependency to 1.15 (#1547)
• zar: Fix an issue where zar zq yielded incorrect event counts compared to plain zq (#1588, #1602)
• zq: Fix a memory bug in collect() that caused incorrect results (#1598)
• zqd: Support log imports over the network (#1336)
• zq: Update performance results to reflect recent improvements (#1605, #1669, #1671)
• zq: Move Zeek & Suricata dependencies into package.json so Brim can point to them also (#1607, #1610)
• zql: Add support for aggregation-less group by (#1615, #1623)
• zqd: Run suricata-update at startup when Suricata pcap analysis is enabled (#1586)
• zqd: Add example Prometheus metrics (#1627)
• zq: Fix an issue where doing put of a null value caused a crash (#1631)
• zq: Add -P flag to connect two or more inputs to a ZQL query that begins with a parallel flow graph (#1628, #1618)
• zql: Add an initial join processor (#1632, #1642)
• zar: Fix an issue where consecutive timestamps caused seek index misses (#1634)
• zar: Fix an issue where time grouping was not working correctly for zar archives (#1650)
• zq/zql: Add support for ZQL comments, multi-line queries, and a -z flag for reading ZQL from a file (#1654)
• zqd: Automatically compact data via a background task (#1625)
• zq: Make ordered merge deterministic (#1663)
• zq: Fix a performance regression (#1672)
• zq: Fix an issue where the JavaScript and Go versions of ASTs could differ (#1665)
• zq: Fix an issue where a lone hyphen in an NDJSON value was output incorrectly (#1673)
• zq: Add an experimental writer for a new format called ZSON (#1681)
• zar: Fix an issue during import that could buffer too much data (#1652, #1696)
• zql: Add a network_of() function for mapping IP addresses to CIDR nets (#1700)
• zql: Add a docs example showing by grouping with non-present fields (#1703)

v0.23.0

Visit the Brim Download page to find the package for your OS platform.

• zql: Add week as a unit for time grouping with every (#1374)
• zq: Fix an issue where a null value in a JSON type definition caused a failure without an error message (#1377)
• zq: Add zst format to -i and -f command-line help (#1384)
• zq: ZNG spec and zq updates to introduce the beta ZNG storage format (#1375, #1415, #1394, #1457, #1512, #1523, #1529), also adddressing the following:
  • New data type bytes for storing sequences of bytes encoded as base64 (#1315)
  • Improvements to the enum data type (#1314)
  • Special characters like . and @ may now appear in field names (#1291)
  • A set may now only support elements of a single type (#1220, #1515)
  • Remove the byte type from the spec in favor of uint8 (#1316)
  • New data type map, which is like set but the contents are key value pairs where only keys need to be unique and the canonical order is based on the key order (#1317)
  • First-class ZNG types (#1365)
  • New numeric data types float16 and float32 (not yet implemented in zq) (#1312, #1514)
  • New numeric data type decimal (not yet implemented in zq) (#1522)
• zq: Add backward compatibility for reading the alpha ZNG storage format (#1386, #1392, #1393, #1441)
• zqd: Check and convert alpha ZNG filestores to beta ZNG (#1574, #1576)
• zq: Fix an issue where spill-to-disk file names could collide (#1391)
• zq: Allow the fuse processor to spill-to-disk to avoid memory limitations (#1355, #1402)
• zq: No longer require _path as a first column in a JSON type definition (#1370)
• zql: Improve ZQL docs for aggregate functions and grouping (#1385)
• zql: Point links for developer docs at pkg.go.dev instead of godoc.org (#1401)
• zq: Add support for timestamps with signed timezone offsets (#1389)
• zq: Add a JSON type definition for alert events in Suricata EVE logs (#1400)
• zq: Update the ZNG over JSON (ZJSON) spec and implementation (#1299)
• zar: Use buffered streaming for archive import (#1397)
• zq: Add an ast command that prints parsed ZQL as its underlying JSON object (#1416)
• zar: Fix an issue where zar would SEGV when attempting to query a non-existent index (#1449)
• zql: Allow sort by expressions and make put/cut expressions more flexible (#1468)
• zar: Move where chunk metadata is stored (#1461, #1528, #1539)
• zar: Adjust the -ranges option on zar ls and zar rm (#1472)
• zq: Choose default memory limits for sort & fuse based on the amount of system memory (#1413)
• zapi: Fix an issue where create and find were erroneously registered as root-level commands (#1477)
• zqd: Support pcap ingest into archive Spaces (#1450)
• zql: Add where filtering for use with aggregate functions (#1490, #1481, #1533)
• zql: Add union() aggregate function (#1493, #1534)
• zql: Add collect() aggregate function (#1496, #1534)
• zql: Add and() and or() aggregate functions (#1497, #1534)
• zq: Fix an issue where searches did not match field names of records with unset values (#1511)
• zq: Fix an issue where searches were not reaching into records inside arrays (#1516)
• zar: Support microindexes created with a sorted flow of records in descending order (#1526)
• zapi: Allow zapi post of S3 objects (#1532)
• zar: Add the zar compact command for combining overlapping chunk files into single chunks (#1531)
• zar: Use chunk seek index for searching chunk data files (#1537)
• zq: Make timestamp output formatting consistent (#1550, #1551, #1557)
• zq: Update LZ4 dependency to improve performance (#1556)
• zq: Fix an issue where TZNG fields containing ] were treated as a syntax error (#1561)
• zar: Fix an issue where the zar import target size didn't take compression into account (#1565)
• zapi: Add a -stats option to zapi pcappost (#1538)
• zqd: Add a Python zqd API client for use with tools like JupyterLab (#1564)

v0.22.0

Visit the Brim Download page to find the package for your OS platform.

• zq: Change the implementation of the union type to conform with the ZNG spec (#1245)
• zq: Make options/flags and version reporting consistent across CLI tools (#1249, #1254, #1256, #1296, #1323, #1334, #1328)
• zqd: Fix an issue that was preventing flows in nanosecond pcaps from opening in Brim (#1243, #1241)
• zq: Fix an issue where the TZNG reader did not recognize a bad record type as a syntax error (#1260)
• zq: Add a CSV writer (-f csv) (#1267, #1300)
• zqd: Add an endpoint for returning results in CSV format (#1280)
• zqd: Add an endpoint for returning results in NDJSON format (#1283)
• zapi: Add an option to return results as a JSON array (-e json) (#1285)
• zapi: Add output format options/flags to zapi get (#1278)
• zqd: Add an endpoint for creating/querying search indexes (#1272)
• zapi: Add commands zapi index create|find for creating/querying search indexes (#1289)
• pcap: Mention ICMP protocol filtering (-p icmp) in help text (#1281)
• zq: Point to new Slack community URL https://www.brimsecurity.com/join-slack/ in docs (#1304)
• zqd: Fix an issue where starting zqd listen created excess error messages when subdirectories were present (#1303)
• zql: Add the fuse processor for unifying records under a single schema (#1310, #1319, #1324)
• zql: Fix broken links in documentation (#1321, #1339)
• zst: Introduce the ZST format for columnar data based on ZNG (#1268, #1338)
• pcap: Fix an issue where certain pcapng files could fail import with a bad option length error (#1341)
• zql: Document the ** operator for type-specific searches that look within nested records (#1337)
• zar: Change the archive data file layout to prepare for handing chunk files with overlapping ranges and improved S3 support (#1330)
• zar: Support archive data files with overlapping time spans (#1348)
• zqd: Add a page containing guidance for users that directly access the root zqd endpoint in a browser (#1350)
• pcap: Add a pcap info command to print summary/debug details about a packet capture file (#1354)
• zqd: Fix an issue with empty records (#1353)
• zq: Fix an issue where interrupted aggregations could leave behind temporary files (#1357)
• zng: Add a marshaler to generate ZNG streams from native Go values (#1327)

v0.21.0

Visit the Brim Download page to find the package for your OS platform.

• zq: Improve performance by making fewer API calls in S3 reader (#1191)
• zq: Use memory more efficiently by reducing allocations (#1190, #1201)
• zqd: Fix an issue where a pcap moved/deleted after import caused a 404 response and white screen in Brim (#1198)
• zqd: Include details on adding observability to the docs for running zqd in Kubernetes (#1173)
• zq: Improve performance by removing unnecessary type checks (#1192, #1205)
• zq: Add additional Boyer-Moore optimizations to improve search performance (#1188)
• zq: Fix an issue where data import would sometimes fail with a "too many files" error (#1210)
• zq: Fix an issue where error messages sometimes incorrectly contained the text "(MISSING)" (#1199)
• zq: Fix an issue where non-adjacent record fields in Zeek TSV logs could not be read (#1225, #1218)
• zql: Fix an issue where cut -c sometimes returned a "bad uvarint" error (#1227)
• zq: Add support for empty ZNG records and empty NDJSON objects (#1228)
• zng: Fix the tag value examples in the ZNG spec (#1230)
• zq: Update LZ4 dependency to eliminate some memory allocations (#1232)
• zar: Add a -sortmem flag to allow zar import to use more memory to improve performance (#1203)
• zqd: Fix an issue where file paths containing URI escape codes could not be opened in Brim (#1238)

v0.20.0

Visit the Brim Download page to find the package for your OS platform.

• zqd: Publish initial docs for running zqd in Kubernetes (#1101)
• zq: Provide a better error message when an invalid IP address is parsed (#1106)
• zar: Use single files for microindexes (#1110)
• zar: Fix an issue where zar index could not handle more than 5 "levels" (#1119)
• zqd: Fix an issue where zapi pcappost incorrectly reported a canceled operation as a Zeek exit (#1139)
• zar: Add support for empty microindexes, also fixing an issue where zar index left behind empty files after an error (#1136)
• zar: Add zar map to handle "for each file" operations (#1138, #1148)
• zq: Add Boyer-Moore filter optimization to ZNG scanner to improve performance (#1080)
• zar: Change "zdx" to "microindex" (#1150)
• zar: Update the zar README to reflect recent changes in commands/output (#1149)
• zqd: Fix an issue where text stack traces could leak into ZJSON response streams (#1166)
• zq: Fix an issue where an error "slice bounds out of range" would be triggered during attempted type conversion (#1158)
• pcap: Fix an issue with pcapng files that have extra bytes at end-of-file (#1178)
• zqd: Add a hidden -brimfd flag to zqd listen so that zqd can close gracefully if Brim is terminated abruptly (#1184)
• zar: Perform zar zq queries concurrently where possible (#1165, #1145, #1138, #1074)

v0.19.1

Visit the Brim Download page to find the package for your OS platform.

• zq: Move third party license texts in zq repo to a single acknowledgments.txt file (#1107)
• zq: Automatically load AWS config from shared config file ~/.aws/config by default (#1109)
• zqd: Fix an issue with excess characters in Space names after upgrade (#1112)

v0.19.0

Visit the Brim Download page to find the package for your OS platform.

• zq: ZNG output is now LZ4-compressed by default (#1050, #1064, #1063, ZNG spec)
• zar: Adjust import size threshold to account for compression (#1082)
• zqd: Support starting zqd with datapath set to an S3 path (#1072)
• zq: Fix an issue with panics during pcap import (#1090)
• zq: Fix an issue where spilled records were not cleaned up if zq was interrupted (#1093, #1099)
• zqd: Add -loglevel flag (#1088)
• zq: Update help text for zar commands to mention S3, and other improvements (#1094)
• pcap: Fix an out-of-memory issue during import of very large pcaps (#1096)

v0.18.0

Visit the Brim Download page to find the package for your OS platform.

• zql: Fix an issue where data type casting was not working in Brim (#1008)
• zql: Add a new rename processor to rename fields in a record (#998, #1038)
• zqd: Fix an issue where API responses were being blocked in Brim due to commas in Content-Disposition headers (#1014)
• zq: Improve error messaging on S3 object-not-found (#1019)
• zapi: Fix an issue where pcappost run with -f and an existing Space name caused a panic (#1042)
• zqd: Add a -prometheus option to add Prometheus metrics routes the API (#1046)
• zq: Update README and add docs for more command-line tools (#1049)

v0.17.0

Visit the Brim Download page to find the package for your OS platform.

• zq: Fix an issue where the inferred JSON reader crashed on multiple nested fields (#948)
• zq: Introduce spill-to-disk groupby for performing very large aggregations (#932, #963)
• zql: Use syntax c=count() instead of count() as c for naming the field that holds the value returned by an aggregate function (#950)
• zql: Fix an issue where attempts to tail too much caused a panic (#958)
• zng: Readability improvements in the ZNG specification (#935)
• zql: Fix an issue where use of cut, put, and cut in the same pipeline caused a panic (#980)
• zql: Fix an issue that was preventing the uniq processor from working in the Brim app (#984)
• zq: Fix an issue where spurious type IDs were being created (#964)
• zql: Support renaming a field via the cut processor (#969)