Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inline shapers vs. pointers to files that contain shaper code #42

Open
philrz opened this issue Apr 19, 2021 · 0 comments
Open

Inline shapers vs. pointers to files that contain shaper code #42

philrz opened this issue Apr 19, 2021 · 0 comments

Comments

@philrz
Copy link
Contributor

philrz commented Apr 19, 2021

While acting as a new user of the brimcap analyze -config YAML, I made the mistake of thinking the shaper parameter was the name of a file containing my Zed shaper code, as opposed to it being the shaper code itself. I felt like this was an innocent mistake since there's other contexts where shapers are treated as files, such as the -I option to zed query, or the way that when we mention reference shaper configs in our docs we point at files in our repos.

If my would-be "shaper" had been somehow invalid and had caused a Zed parse error, I might have found my mistake sooner. Unfortunately, it ended up being valid Zed, effectively doing a text search for the string that's the pathname I provided. It matched nothing, effectively sending my logs to /dev/null. This gave me the mistaken impression that maybe the contents of my shaper script file were fundamentally broken and/or that there was a bug in Brimcap, so I burned a fair amount of time before figuring out my mistake.

A few thoughts for consideration:

  1. If we do nothing else, I can just make sure we emphasize this strongly in the user docs.
  2. The fact that degenerate Zed forms a quiet, no-op shaper could be seen as a general hazard of Zed itself. If there's a way we can identify what it means to be a "shaper" as opposed to any ol' Zed that might legitimately contain just a search, perhaps we could parse it and return errors in contexts like this that only expect a shaper.
  3. Perhaps Brimcap itself could be enhanced such that the shaper parameter supports both in-line Zed as well as files, and the way each approach is specified in the YAML could be self-documenting in a way that would make it very difficult to repeat my mistake.
@philrz philrz added this to the Data MVP0 milestone Apr 21, 2021
@philrz philrz modified the milestones: Data MVP0, Data MVP1 May 10, 2021
@philrz philrz removed this from the ETL Lake milestone Oct 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant