Add a script that reads cabal's plan.json (and a CHaP index) and generates a markdown table of links to changelogs for each package

[fraser-iohk]

Description

• Add a script that reads cabal's plan.json (and a CHaP index) and generates a markdown table of links to changelogs for each package
  • Reads a cabal-generated plan.json
  • Picks out all packages sourced from CHaP
  • Retrieves the corresponding meta.toml for each package-version pair to find its GitHub repository / revision / subdirectory
  • Check for a CHANGELOG.md-looking file in that directory on GitHub (using its API)
  • Generate a giant GitHub-Flavored Markdown table with the package's name, version, and a link to its changelog for that version

Checklist

• Commit sequence broadly makes sense and commits have useful messages
• New tests are added if needed and existing tests are updated. These may include:
  • golden tests
  • property tests
  • roundtrip tests
  • integration tests
    See Runnings tests for more details
• Any changes are noted in the CHANGELOG.md for affected package
• The version bounds in .cabal files are updated
• CI passes. See note on CI. The following CI checks are required:
  • Code is linted with hlint. See .github/workflows/check-hlint.yml to get the hlint version
  • Code is formatted with stylish-haskell. See .github/workflows/stylish-haskell.yml to get the stylish-haskell version
  • Code builds on Linux, MacOS and Windows for ghc-8.10.7 and ghc-9.2.7
• Self-reviewed the diff

Note on CI

If your PR is from a fork, the necessary CI jobs won't trigger automatically for security reasons.
You will need to get someone with write privileges. Please contact IOG node developers to do this
for you.

[andreabedini]

Love this, time permitting I'll have a look today.

Note that foliage produces a json file with the repository metadata along with the repository itself (in foliage/packages.json) so you could also use that.
This avoids the GitHub look up and makes sure the repo metadata content is exactly in perfectly sync with the the project.
(I am assuming a nix provided environment, cabal will not fetch that file)
(In any case packages are immutable so you approach is correct (aside exceptional rewrite-the-history situations))

[fraser-iohk]

Thanks for taking a look!

This avoids the GitHub look up and makes sure the repo metadata content is exactly in perfectly sync with the the project.

Do you mean that the first GitHub lookup -- grabbing all relevant meta.toml files -- can be avoided (assuming that we're in a nix environment)? Or does the generated foliage.json also include extra-source-files and so we can safely "guess" the changelog file path?

[andreabedini]

I was time pressed and I have been unclear. A picture snippet is worth a thousand words:

https://input-output-hk.github.io/cardano-haskell-packages/foliage/packages.json

[
  {
    "pkg-name": "Win32-network",
    "pkg-version": "0.1.0.0",
    "timestamp": "2022-10-17T00:00:00Z",
    "url": "github:input-output-hk/Win32-network/3825d3abf75f83f406c1f7161883c438dac7277d"
  },
...

it contains the package part of meta.toml but, so far, not the revisions (see input-output-hk/foliage#44).

Regarding nix vs not nix, the difference is incidental. Given how haskell.nix works today, it will fetch the whole git repository which hosts the website https://github.com/input-output-hk/cardano-haskell-packages/tree/repo, so it will fetch foliage/packages.json too. We already explot this in the workbench to create a precise manifest.

https://github.com/input-output-hk/cardano-node/blob/408d8ae10a2792ace3a822e312433960e47de4e9/nix/workbench/default.nix#L11-L12

cabal will only fetch the 01-index.tar.gz file (and some other json files related to hackage-security), but of course you can just fetch it yourself.

w.r.t the changelogs: no, we don't pass the package description (and therefore the extra-source-files) into package.json. Maybe we could? Your are welcome to open an issue on the foliage repo ☺️

[fraser-iohk]

Ah, I understand what you mean now. You're right, it'd be easier to pull it from a built foliage/packages.json rather than a pile of meta.tomls.

I should be able to make the script more resilient by parsing packages.json, though it'll involve a little more parsing for the url field (and we'd still be running into GitHub rate limits without a token because of the CHANGELOG.md lookup) 🤔

[fraser-iohk]

Added info about the script to the RELEASE.md script, and it now pulls package info from packages.json instead of many meta.toml files (so doesn't need access to a cardano-haskell-packages repo)