v1.0.0

Quick Start

Visit the Brim Data download page to find the package for your platform.

Zed is distributed as a TGZ/ZIP file of command line executables for each supported platform. To install, unpack with with the appropriate utility for your platform. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v1.0.0/zed-v1.0.0.linux-amd64.tar.gz
tar xzvf zed-v1.0.0.linux-amd64.tar.gz

If you've installed the Brim desktop app and want to work with its Zed lake from the command line, we recommend using the Zed executables included with the app to ensure compatibility. See this Brim wiki article for their location.

Release Notes

• Comprehensive documentation
• Substantial improvements to the Zed language
• Revamped zed command
• New Zed lake format (see #3634 for a migration script)
• New version of the ZNG format (with read-only support for the previous version)
• New version of the ZSON format

v0.33.0

Quick Start

Visit the Brim Data download page to find the package for your platform.

Zed is distributed as a ZIP file of command line executables for each supported platform. To install, unpack with your platform's unzip utility. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.33.0/zed-v0.33.0.linux-amd64.zip
unzip zed-v0.33.0.linux-amd64.zip

If you've installed the Brim desktop app and want to work with its Zed lake from the command line, we recommend using the Zed executables included with the app to ensure compatibility. See this Brim wiki article for their location.

Release Notes

• zapi: Rename the ZED_LAKE_HOST environment variable to ZED_LAKE and rename the -host flag to -lake (#3280)
• zq: Improve ZNG read performance when the command line includes multiple input files (#3282)
• zed lake serve: Add the -rootcontentfile flag (#3283)
• Python client: Improve error messages (#3279)
• Python client: Fix Zed bytes decoding (#3278)
• Detect CSV input (#3277)
• zed lake serve: Fix an issue where POST /pool/{}/branch/{} format detection errors caused a 500 response (#3272)
• Fix an issue where the ZSON parser failed to normalize maps and sets (#3273)
• Python client: Add authentication (#3270)
• Python client: Handle query errors (#3269)
• Remove support for the TZNG format (#3263)
• zapi, zed lake serve: Add authentication with Auth0 (#3266)
• Fix an issue preventing casting from ip to ip (#3259)
• zed lake serve: Respect the Accept request header for GET /events (#3246)
• Add function documentation (#3215)
• zed lake serve: Change the default response content encoding to ZSON (#3242)
• zapi load, zed lake load: Add the -meta flag to embed custom metadata in commits (#3237)

v0.32.0

Quick Start

Visit the Brim Data download page to find the package for your platform.

Zed is distributed as a ZIP file of command line executables for each supported platform. To install, unpack with your platform's unzip utility. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.32.0/zed-v0.32.0.linux-amd64.zip
unzip zed-v0.32.0.linux-amd64.zip

If you've installed the Brim desktop app and want to work with its Zed lake from the command line, we recommend using the Zed executables included with the app to ensure compatibility. See this Brim wiki article for their location.

Release Notes

• Add create_pool() and load() methods to the Python client (#3232)
• Allow a leading split operator (#3230)
• Remove the exists() function in favor of missing() (#3225)
• Remove the iso() function in favor of time() (#3220)
• Remove deprecated GET /pool and GET /pool/{pool} from the Zed lake service API (#3219)
• Add bytes literals ("0x" followed by an even-length sequence of hexadecimal digits) to the Zed language (#3209)
• When sending a JSON response for POST /query, always send an array (#3207)
• Fix a panic when compiling SELECT ... GROUP BY ... (#3193)
• Fix a bug in which data loaded through the Zed lake service was stored uncompressed (#3198)
• Add all lake index commands to Zed lake service (#3181)
• Reorganize language documentation (#3187)
• Make fuse() output deterministic (#3190)
• Use lake indexes to speed up queries (#3158)
• Fix bug where constants blocked from operator wiring logic (#3185)
• Allow the dot operator to work on a union containing a record (#3178)
• Disable escaping of "&", "<", and ">" in JSON output (#3177)
• Change collect() to handle heterogeneous types with a type union (#3176)
• Extend the join operator to support the anti join type (#3173)
• Make lake index create output the details of the newly created rule (#3168)
• Enable ANSI escapes in command output on Windows (#3164)
• Change zed lake query -stats output to ZSON (#3159)
• Fix a ZSON quoting bug for type value field names (#3154)
• Allow pool names (in addition to pool IDs) in Zed lake service API paths (#3144)

v0.31.0

Quick Start

Visit the Brim Download page to find the package for your OS platform.

Zed is distributed as a ZIP of command line binaries for each supported platform. To install, unpack with the unzip utility for your OS. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.31.0/zed-v0.31.0.linux-amd64.zip
unzip zed-v0.31.0.linux-amd64.zip

Note that if you've also installed the Brim desktop app, a set of Zed binaries is already unpacked as part of the app installation. If you intend to use Zed at the command line to work with data in the Zed lake that's behind by Brim, it is recommended to use the binaries there were unpacked with the app, since these have been tested with that app release. See this Brim wiki article for details regarding their location.

Release Notes

• Allow indexes to handle fields containing values of different types (#3141)
• Improve CSV writer performance (#3137)
• Fix an issue preventing use of a seek index containing nulls (#3138)
• Add float32 primitive type (#3110)
• Add len() support for bytes, error, and map types (#3136)
• Allow empty ZSON maps (#3135)
• Fix an issue affecting range queries on a lake containing records with a missing or null pool key (#3134)
• Allow from ( pass => ...; ) (#3133)
• Change Go marshaling struct field tag to zed from zng (#3130)
• Fix a panic when reading CSV containing an empty quoted field (#3128)
• Improve CSV output format (#3129)
• Detect JSON input containing a top-level array (#3124)
• Decode top-level JSON arrays incrementally (#3123)
• Remove PPL license (#3116)
• Change ZSON map syntax to |{ key: value, ... }| (#3111)
• Support revert for indexes (#3101)
• Rename zson_parse() to parse_zson() (#3092)
• Add zed lake index update and zed api index update commands (#3079, #3093)
• Add parse_uri() function (#3080, #3084)
• Add from pool@branch:indexes meta query (#3078)
• Fix an issue where sort len(field) produced incorrect output (#3045)
• Remove POST /ast and POST /search from the Zed lake service API (#3065)
• Fix an issue with with record aliases in drop (#3064)

v0.30.0

Quick Start

Visit the Brim Download page to find the package for your OS platform.

Zed is distributed as a ZIP of command line binaries for each supported platform. To install, unpack with the unzip utility for your OS. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.30.0/zed-v0.30.0.linux-amd64.zip
unzip zed-v0.30.0.linux-amd64.zip

Note that if you've also installed the Brim desktop app, a set of Zed binaries is already unpacked as part of the app installation. If you intend to use Zed at the command line to work with data in the Zed lake that's behind by Brim, it is recommended to use the binaries there were unpacked with the app, since these have been tested with that app release. See this Brim wiki article for details regarding their location.

Release Notes

As you can see below, there's been many changes since the last Zed GA release! Highlights include:

• The introduction of Zed lakes for data storage, which include powerful Git-like branching. See the Zed lake README for details.
• Enhancements to the Zed language to unify search and expression syntax, introduce new operators and functions for data exploration and shaping, and more! Review the Zed language docs for details.

The exhaustive set of changes is listed below. Come talk to us on Slack if you have additional questions.

---

• Revise Zed language to unify search and expression syntax (#2072, #2152, #2252, #2304, #2294)
• Add join() and split() functions for use on strings (#2098)
• Add array slice expressions (#2100)
• Fix an issue with connection resets after several minutes when posting data to S3 (#2106)
• Fix an issue with parsing IPv6 literals (#2112)
• Make the fuse operator work on nested records (#2052)
• Fix an issue where cut(.) could cause a slice bounds out of range panic (#2107)
• Add is(), fields(), and exists() functions (#2131)
• Add auto-detection of ZSON format (#2123)
• Fix an issue where cut to the root would exit if the referenced field was missing from a record (#2121)
• Fix an issue where put to the root would panic on a non-record field (#2136)
• Add support for parsing map types in ZSON (#2142)
• Add a fuse() aggregate function (#2115)
• Remove backward compatibility with alpha ZNG format (#2158)
• Simplify ZSON by dropping type decorators when a complex value is fully implied (#2160)
• Add a switch operator to allow branched processing (#2087, #2364, #2318, #2336)
• Add constants and type literals to the Zed language (#2181)
• The -I option in zq is now used for file includes (and allows multiple files), while -z now used for compact ZSON output (#2180, #2208)
• Add support for shaping arrays and sets (#2173)
• Fix an issue where outer aliases were being lost when ZSON was read into ZNG (#2189)
• Add the sample operator that returns an example value for a named field, or for each unique record type (#2200, #2211, #2623)
• Make the current record (i.e., this or .) an implicit argument to shape() (#2199)
• Begin deprecating current TZNG format in favor of ZSON (#2208, #2312, #2333, #2338, #2337, #2339, #2340, #2355, #2367, #2377, #2387, #2388, #2389, #2395, #2477, #2485, #2480, #2513, #2520)
• Fix an issue where accidentally reading non-Zed binary data caused a zq panic (#2206)
• Fix an issue where time-sorted aggregations were returning non-deterministic results (#2220)
• Add canonical Zed and the summarize operator as an explicit keyword before invoking aggregate functions (#2217, #2378, #2430, #2698)
• Add support for casting the duration type (#2194)
• Extend join to support inner (now the default), left, and right variations (#2210)
• Fix an issue where Zed would not compile on FreeBSD (#2233)
• Add the zson_parse() function (#2242)
• Fix an issue where filenames containing : could not be read (#2240)
• Handle aliases and typedefs in shaper functions, which also fixes a panic (#2257)
• Improve Zeek reader performance (#2265, #2268)
• Fix an issue where const references were not honored during query execution (#2260)
• Fix an issue where shapers did not handle aliases to different castable types (#2280)
• Add an unflatten() function that turns fields with dot-separated names into fields of nested records (#2277)
• Fix an issue where querying an index in a Zed lake did not return all matched records (#2273)
• Accept type definition names and aliases in shaper functions (#2289)
• Add a reference shaper for Zeek data (#2300, #2368, #2448, #2489, #2601)
• Fix an issue where accessing a null array element in a by grouping caused a panic (#2310)
• Add support for parsing timestamps with offset format ±[hh][mm] (#2297)
• Remove cropping from shape() (#2309)
• Apply a Zed shaper when reading Suricata EVE data, instead of legacy JSON typing (#2298, #2370, #2400)
• Add support for reading comma-separated value (CSV) files (#2317, #2858, #2942, #2963)
• Fix an issue where reading a Zeek TSV log line would cause a panic if it contained too few fields (#2325)
• Add a shape operator, which is useful for cleaning up CSV inputs (#2327)
• Fix an issue where querying a Zed lake index for a named field could cause a panic (#2319)
• Make casting to time and duration types more flexible (#2334, #2442)
• Fix an issue where null values were not output consistently in a group-by aggregation (#2363)
• Fix an issue where the confirmation messages from adding an index were sometimes incomplete (#2361)
• Finalize ZSON duration format to be an extension of durations in Prometheus (#2358, #2371, #2381, #2396, #2405)
• Add functions missing(), has(), and nameof() (#2393, #2708)
• Add prototype support for SQL expressions (#2392)
• Allow type definitions to be redefined (#2386)
• Fix an issue where casting to a named type caused the loss of the type definition name (#2384)
• Add support for Parquet output and rework the Parquet reader (#2227)
• Don't interpret the first zq argument as a query if there are no additional arguments (#2382)
• Fix an issue that was preventing the reference in an expression to a field name containing a . (#2407)
• Add support for ISO time literals and support durations and time literals in expressions (#2406)
• Add support for complex literals (#2403)
• Code/repo reorganization for phasing out "ZQL" or "Z" in favor of "Zed language", or just "Zed" if context allows (#2416, #2431, #2455, #2831)
• Support in with the map data type (#2421)
• Normalize map values created from Zed expressions (#2423)
• Switch to function-style casting (e.g., int64(123) instead of 123:int64) (#2427, #2438)
• Allow shapers to to refer to the contents of input records to determine the type to apply (#2426)
• Fix an issue where referencing a non-existent table in a SQL query caused a panic (#2432)
• Accept - (stdin) as a zapi argument for loading data (#2435)
• Fix an issue where a single bad cast could cause input processing to halt (#2446)
• Create the zed command with sub-commands like query and api, but shortcut commands (e.g., zq, zapi) still remain (#2450, #2465, #2466, #2463, #2624, #2620)
• Rename ZAR_ROOT environment variable to ZED_LAKE_ROOT (#2469)
• Revise the top-level Zed README to reflect reorganization of the repo and new/changed tools (#2461)
• Remove the -P flag from zq in favor of using from in the Zed language (#2491)
• Add casting of the net data type (#2493, #2496)
• zq now reads its inputs sequentially rather than the prior merged behavior (#2492)
• Extend the len() function to return the number of fields in a record (#2494)
• Remove the -E flag in zed commands that displayed time values as epoch (#2495)
• Add the Zed lake design README document (#2500, #2569, #2595, #2781, #2940, #3014, #3034, #3035)
• Fix an issue where escaping quotes caused a parse error (#2510)
• Fix an issue where multiple ZSON type definitions would be output when only the first was needed (#2511)
• Use less buffer when decoding ZSON (#2515)
• Allow aliases of all primitive types to be expressed in ZSON (#2519)
• Revert the "auto-fuse CSV" behavior originally added in #1908 (#2522)
• Add support for Git-style Zed lakes (#2548, #2556, #2562, #2563, #2564, #2566, #2571, #2577, #2580, #2616, #2613, #2738, #2763, #2806, #2808, #2811, #2816, #2860, #2861, #2931, #2944, #2954, #2960, #2976, #2994, #3007, #3013, #3020, #3023, #3024, #3026, #3030, #3031, #3039, #3046)
• Add support for reading JSON format input data via -i json (#2573, #2608)
• Remove the legacy approach for applying Zed types to NDJSON input, as this is now done via Zed shapers (#2587)
• Fix a Go client issue where ZNG marshal of unexported struct fields caused a panic (#2589)
• Show a warning rather than failing when an unset value tries to be cut to the root (#2591)
• Standardize -h usage in Zed CLI tools for showing help text (#2596, #2618)
• Fix an issue where type names that started with prim...

v0.29.0

Visit the Brim Download page to find the package for your OS platform.

• zqd: Update Zeek pointer to v3.2.1-brim10 which provides the latest geolocation data (#2081)
• zql: Add shaping primitive functions cast(), fill(), crop(), and order(), along with fit() and shape() (#1984, #2059, #2073, #2033)
• zson: Read ZSON incrementally rather than all at once (#2031)
• zson: Tighten whitespace in ZSON -pretty=0 output (#2030)
• zql: Change parallel graph syntax to use split and => (#2037)
• zson: Add duration to the implied type list (#2039)
• zq: Fix an issue with rename where a subsequent count() would return no results (#2046)
• zq: Fix an issue where multiple alias typedefs were generated for the same type, causing a TZNG read failure (#2047)
• zson: Fix an issue with string scanning in the ZSON parser that caused the failure parse error: parsing string literal (#2048)
• zq: Fix an issue on Windows where - was not being treated as a way to read from stdin (#2061)
• zq: Add support in put for assigning to . and to nested fields (#2018)
• zson: Fix an issue where reading ZSON caused the failure parse error: mismatched braces while parsing record type (#2058)
• zson: Fix an issue where casting null values to string types caused invalid output (#2077)

v0.28.0

Visit the Brim Download page to find the package for your OS platform.

---

NOTE - Beginning with this release, a subset of the source code in the github.com/brimsec/zq GitHub repository is covered by a source-available style license, the Polyform Perimeter License (PPL). We've moved the PPL-covered code under a ppl/ directory in the repository. The majority of our source code retains the existing BSD-3-Clause license.

The overwhelming majority of zq/zqd users and developers will not be impacted by this change, including those using zq/zqd in commercial settings. The use of the source-available Polyform Perimeter license prevents use cases like marketing a work as a "as-a-service" style offering for server components like zqd while using material covered under the PPL.

In general, we are making this change to ensure technology giants can't use the PPL-covered code to make replacement offerings of our projects. We believe users and developers should have access to the source code for our projects, and we need a sustainable business model to continue funding our work. Using the source-available Polyform Perimeter license on portions of the source code lets us realize both.

For more detail regarding licensing, see the CONTRIBUTING.md doc, and feel free to come talk to us on Slack if you have additional questions.

---

• zqd: Update Zeek pointer to v3.2.1-brim9 which provides the latest geolocation data (#2010)
• zqd: Update Suricata pointer to v5.0.3-brim1 which disables checksum checks, allowing for alert creation on more types of pcaps (#1975)
• zson: Update Zeek Interoperability doc to include current ZSON syntax (#1956)
• zq: Ensure the output from the fuse processor is deterministic (#1958)
• zq: Fix an issue where the presence of the Greek µ character caused a ZSON read parsing error (#1967)
• zqd: Fix an issue where Zeek events generated during pcap import and written to an archivestore were only visible after ingest completion (#1973)
• zqd: Change the logger configuration to output stacktraces on messages of level "warn" and higher (#1990)
• zq: Update performance results to include ZSON read/write (#1974)

v0.27.1

Visit the Brim Download page to find the package for your OS platform.

• zq: Fix an issue where nested nulls caused a panic in CSV output (#1954)

v0.27.0

Visit the Brim Download page to find the package for your OS platform.

• zqd: Update Zeek pointer to v3.2.1-brim8 which provides the latest geolocation data (#1928)
• zson: Allow characters . and / in ZSON type names, and fix an issue when accessing fields in aliased records (#1850)
• zson: Add a ZSON marshaler and clean up the ZNG marshaler (#1854)
• zq: Add the source field to the JSON typing config to prepare for Zeek v4.x weird events (#1884)
• zq: Add initial Z "shaper" for performing ETL on logs at import time (#1870)
• zq: Make all aggregators decomposable (#1893)
• zq/zqd: Invoke fuse automatically when CSV output is requested (#1908)
• zq: Fix an issue where fuse was not preserving record order (#1909)
• zar: Create indices when data is imported or chunks are compacted (#1794)
• zqd: Fix an issue where warnings returned from the /log/path endpoint were being dropped (#1903)
• zq: Fix an issue where an attempted search of an empty record caused a panic (#1911)
• zq: Fix an issue where a top-level field in a Zeek TSV log was incorrectly read into a nested record (#1930)
• zq: Fix an issue where files could not be opened from Windows UNC paths (#1929)

v0.26.0

Visit the Brim Download page to find the package for your OS platform.

• zqd: Update Zeek pointer to v3.2.1-brim7 which provides the latest geolocation data (#1855)
• zq: Improve the error message shown when row size exceeds max read buffer (#1808)
• zqd: Remove listen -pprof flag (profiling data is now always made available) (#1800)
• zson: Add initial ZSON parser and reader (#1806, #1829, #1830, #1832)
• zar: Use a newly-created index package to create archive indices (#1745)
• zq: Fix issues with incorrectly-formatted CSV output (#1828, #1818, #1827)
• zq: Add support for inferring data types of "extra" fields in imported NDJSON (#1842)
• zqd: Send a warning when unknown fields are encountered in NDJSON logs generated from pcap ingest (i.e. Suricata) (#1847)
• zq: Add NDJSON typing configuration for the Suricata "vlan" field (#1851)