Overhaul of the Manual

[01mf02]

In the last days, I've completely overhauled the jq manual. Some of the most important changes:

• I've regrouped the chapters. I tried to make it such that the chapters build on each other; for example, to explain basic filters such as |, it is useful to know how to construct values first, and to understand paths such as .posts[0], it is useful to understand | first.
• To help with finding the information, the TOC can now be expanded to show subsections of sections.
• I've annotated many filters with compatibility notes that document divergent behaviour for other implementations. Currently, this covers mostly my own jq interpreter jaq, but I expect this to be useful too for gojq.
• I've reworked the visual appearance of the man page, such that longer blocks (such as examples or notes) are visible as such.
• I've corrected bugs in the manual as I went along; for example, the behaviour of arithmetic-update assignments.
• I've also added a large number of new examples.
• I significantly reworked the sections on paths, definitions, and assignments. These are nearly rewrites.

All in all, this PR adds a lot of new information to the jq manual and tries to improve its structure.

To see it in all its glory, you need Pandoc >3.0. Go to jq/docs, run pipenv sync, then

pipenv run python3 build_website.py --root /output && python3 -m http.server

and open http://0.0.0.0:8000/output/manual in your browser.

Please let me know if I can do something to get this merged.
This builds on and supersedes my previous PR (#3186).

[01mf02]

When using input, it is generally necessary to invoke jq with the -n command-line option, otherwise the first entity will be lost.

At first blush, this gives the impression that the warning does not apply to inputs. Also, since the first example immediately below shows the opposite case, it might be worthwhile coming up with a more helpful phrasing, perhaps:

When using input and/or inputs, it is often necessary to invoke jq with the -n command-line option to avoid losing the first value in the input stream.

Good catch. I have integrated your suggestion.

[01mf02]

It is now also possible to generate a PDF version of the manual. For this, you need Typst (I use 0.11.1).

pandoc content/manual/manual.md --lua-filter filters/filter.lua -o manual.typ
typst c manual.typ

Here is the current output. The appearance is still a bit rough at some places (especially example tables) --- keep in mind that I have not applied any Typst-specific styling. With a bit of work, this could be made to look quite nice. Right now, this is more of a demo to show that the new manual enables a wide range of output formats with relatively little effort.

[01mf02]

Sorry to spam you again, but: @wader, @nicowilliams, @itchyny, @pkoppstein, @emanuele6, may I ask you what the status is on this PR? I've invested quite a significant amount of time into this, and it would be nice to have at least some basic feedback (which @pkoppstein has already given). If you have concerns about certain parts of this PR, then I'd be happy to address these.

[wader]

Nice work! note that as i'm reading thru nearly all of the document I might have added comments about texts that are already present.

Some questions/issues/reflections:

• Read/Edit directly in markdown is much nicer than yaml i would say
• What to do with old versions of the manual? Keep as is? Try move things into one big manual with version compatibility section etc?
• How to get it merged? approval from 3 maintainers is enough? also i think personally i would rather merge early, as it includes so many improvements, and then fix up stuff in further PRs rather than keep iterating on the same PR for a long time. That would also maker it easier to work in parallell.

[wader]

Maybe be even more explicit like "they take an input, that is ignored, and produce the same literal output"?

[01mf02]

Added in 80ebbfe.

[wader]

Maybe in the future this could also include a section about debug techniques etc?

[wader]

Can skip "named"?

[wader]

Maybe we could mention backtracking and use in recursive functions here? or refer to the section about functions and recursion?

[pkoppstein]

@wader wrote:

also i think personally i would rather merge early, as it includes so many improvements

I wholeheartedly agree. In particular, I hope that any potential revisions regarding gojq won't hold things up. Indeed, there's a lot to be said for waiting until the jq+jaq version of the manual has been merged before embarking on a gojq-related project. Perhaps that's something @itchyny might be willing to undertake?

[01mf02]

Nice work! note that as i'm reading thru nearly all of the document I might have added comments about texts that are already present.

Sorry to answer a bit late --- I have been quite busy with other stuff, and did not have the time so far to integrate your comments.

Read/Edit directly in markdown is much nicer than yaml i would say

Yes, that's also why I did this change in the first place. :)

What to do with old versions of the manual? Keep as is? Try move things into one big manual with version compatibility section etc?

I would propose to keep the old versions of the manual up to (including) 1.7, and to start annotating the current version with compatibility notes for added/changed functions. That way, finding out which functions are available in older versions of jq should become much easier, because you have the information in the manual. (Nowadays, you would have to browse through all manuals from previous versions until you do not find a reference to a function anymore.)

How to get it merged? approval from 3 maintainers is enough? also i think personally i would rather merge early, as it includes so many improvements, and then fix up stuff in further PRs rather than keep iterating on the same PR for a long time. That would also maker it easier to work in parallell.

Of course, I do not have a say in this, but I wholeheartedly agree with this. It would be really great if some other maintainers would speak up here ...

[wader]

Sorry to answer a bit late --- I have been quite busy with other stuff, and did not have the time so far to integrate your comments.

No worries! hope they are useful

I would propose to keep the old versions of the manual up to (including) 1.7, and to start annotating the current version with compatibility notes for added/changed functions. That way, finding out which functions are available in older versions of jq should become much easier, because you have the information in the manual. (Nowadays, you would have to browse through all manuals from previous versions until you do not find a reference to a function anymore.)

Yes that is a good idea i think. Would that be per implementation?

Of course, I do not have a say in this, but I wholeheartedly agree with this. It would be really great if some other maintainers would speak up here ...

Yeap would be great