Add support for deprecated-versions

[michaelpj]

This is pretty handy, as it's a useful tool in case of the not-so-uncommon problem of "oops, I released something broken".

[andreabedini]

See also #8

[yvan-sraka]

@angerman advised me yesterday to start looking at this issue. I spent sometimes understanding what this project does, how to use it etc … Thinking a bit about this issue, I could put the deprecated attribute in the <package>/<version>/meta.toml file, and as far that I understand I should start by modifying the way these files are parsed: https://github.com/andreabedini/foliage/blob/586b692c5c0c63c7a79f87ef66f2b7a1172fc990/app/Foliage/CmdBuild.hs#L262-L273
But @andreabedini suggests rather in #8 to parse a new <package>/meta.toml package-level metadata file, that then I understand I would parse after: https://github.com/andreabedini/foliage/blob/586b692c5c0c63c7a79f87ef66f2b7a1172fc990/app/Foliage/CmdBuild.hs#L91

Would it make sense that I start by implementing the depreciated error mechanism and only after to implement the preferred-version mechanism as described in #8 (comment)?

[michaelpj]

I think they're a pair: they're the same underlying mechanism so we probably actually want to do them together.

[yvan-sraka]

I understand “depreciated” as by extend “all previous versions are depreciated”, should we rather here talk about a “yanked” version as “this particular version couldn't longer be used”?

[michaelpj]

Well, I think that's not quite the case, since I think you can override it with sufficiently stringent constraints on the user side. Also I think we want to follow Hackage's naming here to avoid confusion!

[andreabedini]

If we where to implement package version deprecation, we need to figure out how to encode this in the input metadata.

• We need to support packages being deprecated and undeprecated. Each event with its own timestamp.
• We need to reject a package version being deprecated or un-derprecated twice (i.e. (un-)deprecated while it's already (un-)deprecated); since this will become a new preferred-versions file in the index with the same content. This could be done either through the encoding (by making this situation unrepresentable) or as a validation step.
• We need to make sure we add the preferred-versions file in the index in a predictable order (perhaps solving Give more guarantees on the ordering of the tarball entries #20 along the way).

[michaelpj]

Brainstorming a bit. At the moment we essentially have a list of events that change the status of the package.

• Adding the package
• Revising the package
• Changing the deprecation status of the package

All of these have timestamps because they happen at a particular time (they're events!).

The package addition can only happen once and must happen for a package to be included, so we put its metadata in the root.

We represent a revision event with a [[revision]] block, which seems fine.

So that suggests to me we want a [[deprecation]] block. How about

[[deprecation]]
timestamp: <whatever>

[[deprecation]]
timestamp: <whatever>
deprecated: false

i.e. we have a deprecated field with a default value of true which says what the deprecation status is changed to. So you can un-deprecate by including it set to false.

We still need to do the checks that Andrea points out, though.

[yvan-sraka]

Okay, I think there are 2 questions here:

• What API do we want to bring to the user?
  I got we chose in Add support for preferred-versions #8 to go for a _sources/<package>/<version>/meta.toml, to map on how revisions/patches works. And then refine on the exact syntax/semantic we want, like the proposition suggested by @michaelpj.

• How to encode depreciated versions in a Hackage (and so Cabal) compatible way?
  Doing a few experiments on this matter with a depreciated version of a package that @andreabedini give as example https://hackage.haskell.org/package/composition-prelude-2.0.3.0, inspecting the associated composition-prelude/2.0.3.0/package.json does not make mention of any depreciation:

  {
     "signatures":[],
     "signed":{
        "_type":"Targets",
        "expires":null,
        "targets":{
           "<repo>/package/composition-prelude-2.0.3.0.tar.gz":{
              "hashes":{
                 "md5":"e2edefd1f890b2a3b608e2a60f3d9561",
                 "sha256":"58756a862ffcdc3f5fb7c7571327534f4f9cb96d246ca60f7e98780461717c75"
              },
              "length":3474
           }
        },
        "version":0
     }
  }

  … because (starting to understand all this matter of “preferred versions”), this is part of the composition-prelude/preferred-versions file:

  composition-prelude <2.0.3.0 || >2.0.3.0 && <3.0.0.1 || >=3.0.0.2
  

  … on which I understand Cabal will rely! Then foriage build will have to generate _repo/<package>/preferred-versions anyway, right?

[michaelpj]

The underlying mechanism is just preferred versions. Deprecated versions are implemented by preferring everything else, I believe. The point is that we only want to expose the concept of deprecation to our users.

[yvan-sraka]

The underlying mechanism is just preferred versions. Deprecated versions are implemented by preferring everything else, I believe. The point is that we only want to expose the concept of deprecation to our users.

So basically, the mechanism implemented will be to construct the _repo/<package>/preferred-versions file given a user-defined list of depreciated versions in _sources/<package>/*/meta.toml.

And the list could be most likely constructed from a list of timestamped depreciation / undepreciation steps, right?

[yvan-sraka]

A minimal example would be:

• _sources/foobar/0.1.0/meta.toml:

  ...
  
  [[deprecation]]
  timestamp = "whatever"
  
  [[deprecation]]
  # should it be more recent than previous timestamp?
  timestamp = "whatever"
  deprecated = false

• _sources/foobar/0.1.1/meta.toml:

  ...
  
  [[deprecation]]
  timestamp = "whatever" # IDEM?

• _sources/foobar/0.1.2/meta.toml: file exist, but no depreciation are mentioned.

Then the computed depreciated version list is [ "0.1.1" ], so the preferred version list is [ "0.1.0", "0.1.2" ], and the _repo/foobar/preferred-versions generated would be:

foobar =0.1.0 || =0.1.2

Am I correct?

[andreabedini]

@yvan-sraka I think you got it.

The file package.json is just for the TUF signatures and no other metadata is stored in there (that I know of). So we can forget about it.

To see how the sausage index is made, you are welcome to look at the hackage-server source code (E.g https://github.com/haskell/hackage-server/blob/master/src/Distribution/Server/Features/PreferredVersions.hs#L311-L336 is a bit in the middle). And you can look at cabal for how it is used (idk, https://github.com/haskell/cabal/blob/master/cabal-install/src/Distribution/Client/IndexUtils.hs#L866-L875 the preferred versions are stored as a VersionRange in that Dependency).

The logic of computing the preferred versions seems to be here. In particular the VersionRange is computed from the preferred versions ranges and the deprecated versions with this function.

consolidateRanges :: [VersionRange] -> [Version] -> Maybe VersionRange
consolidateRanges ranges depr =
    let range = simplifyVersionRange $ foldr intersectVersionRanges anyVersion (map notThisVersion depr ++ ranges)
    in if isAnyVersion range || isNoVersion range
        then Nothing
        else Just range

The version handling functions are exported by Cabal-syntax so we can use the same ones.