Schedule job to sync Scaladex with sbt-plugin-releases

[adpi2]

Motivation

Many sbt plugins are published to JFrog Artifactory https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/.

When a new artifact in published to it it is not automatically pushed to Scaladex. We used to fetch all new artifacts from sbt-plugin-releases by executing UpdateBintraySbtPlugins manually. This is not convenient anymore, since we use an SQL database.

Instead we can create a new scheduled job to download new artifacts from JFrog Artifactory.

Identified Obstacles

1. Artifactory API

Is the Artifactory API the same as the Bintray API?

2. Rate limit

The Artifact API has some rate limit mechanisms. We should make sure to use most of the rate but not hit the limit.

3. Download all artifacts from a date

Can we download all new artifacts from a specific date? If so we need to store the date of the last update into the database.

Implementation guidelines

1. Create a ArtifactoryClient that can call the Artifactory API to find and download the new artifacts.
2. Create a new scheduler that uses the SchedulerDatabase and the ArtifactoryClient to index the new artifacts.
3. Configure this new scheduler to run every day.

Expectations

1. The ArtifactoryClient should be tested (in IntegrationTest).
2. The ArtifactoryClient should be protected against rate limit errors.