Skip to content

Add proposal for proxy cache referrer's API #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stonezdj wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add proposal for proxy cache referrer's API #270

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions proposals/new/proxy_cache_referer_api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
Proposal: Support referrer API in proxy cache

Author: Stone Zhang

## Abstract

Harbor already implements the referrers API defined in [OCI Distribution Spec 1.1](https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/) to support the use case of querying artifacts referring to a specific artifact, such as querying signatures or SBOMs referring to an image. However, the current implementation only supports artifacts stored in the local registry and does not fetch referrers stored in the upstream registry. This proposal aims to enhance the current implementation to support fetching referrers from the upstream registry when the image is pulled from a proxy cache project.

## Motivation

Users want to get all referrers of an artifact, regardless the artifact is stored in Harbor or in the upstream registry of a proxy cache project.
For example, if a user generate a SBOM for library/ldaputils:latest HarborA, then calls the referrer API to get the sbom, it works as expected.



Set up a proxy cache project in HarborB to proxy artifacts from HarborA, then pull the image from HarborB
```
docker pull <HarborB>/proxycache_project/library/ldaputils:latest
```
When calling the referrer API in HarborB, it only returns referrers stored in HarborB, but not the SBOM stored in HarborA.
```
GET http://<HarborB>/v2/proxycache_project/library/ldaputils/referrers/sha256:6ed0f1192837ea7e8630b95d262d5a7aa9ce84b5db2dbf99c2ca02c0a2e20046
```
The response is:
```
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": []
}
```
The current proxy cache doesn't support fetching referrers from upstream registry, this proposal is to enhance the current implementation to support this feature.

After this feature completed, the referrer's API send GET request to the proxy cache in HarborB, it should return the same content in HarborA

```
GET http://<HarborB>/v2/proxycache_project/library/ldaputils/referrers/sha256:6ed0f1192837ea7e8630b95d262d5a7aa9ce84b5db2dbf99c2ca02c0a2e20046
```
The response should be
```
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"digest": "sha256:e1d424dc9f484622730e3308ead73ab07fa4a7616384f73abfbcc5c5aab4721a",
"size": 767,
"annotations": {
"created": "2025-11-14T05:55:08Z",
"created-by": "Harbor",
"org.opencontainers.artifact.created": "2025-11-14T05:55:08Z",
"org.opencontainers.artifact.description": "SPDX JSON SBOM"
},
"artifactType": "application/vnd.goharbor.harbor.sbom.v1"
}
]
}
```

## Issues

[Harbor proxy cache doesn't proxy the referrer API](https://github.com/goharbor/harbor/issues/20808)

## Solution

Add an option then creating proxy cache project, to enable or disable the referrers API proxying feature. When enabled, Harbor will fetch referrers from the upstream registry and cache the content in local, so that when the upstream registry is offline, the local cached content can be served. When this option is not enabled, the referrers API will behave as it does with a non proxy cache project, it doesn't fetch referrers from upstream registry. the default value is disabled.

When a referrers API request to a proxy cache project, the following steps should be performed:

1. Check if the upstream is healthy, if not, return the cached referrers from local storage.
2. Check if the upstream referrers API is supported, if not, return the referrers from local storage.
3. Call the upstream referrers API to get referrers list. if the format is invalid, return the referrers from local storage. if it is valid, return the referrers list.
4. Cache referrers list in redis for future use (in background).


## Implementation Details

Add a new middleware to handle the referrers API request for proxy cache projects, it should be registered before the existing referrers API handler.

```
root.NewRoute().
Method(http.MethodGet).
Path("/*/referrers/:reference").
Middleware(metric.InjectOpIDMiddleware(metric.ReferrersOperationID)).
Middleware(repoproxy.ProxyReferrerMiddleware()). // referrers proxy middleware
Handler(newReferrersHandler())
```

In the `repoproxy` package, implement a new middleware `ProxyReferrerMiddleware` to handle the referrers API request for proxy cache projects.

The `ProxyReferrerMiddleware` should perform the following steps:
1. Extract the project and artifact information from the request context.
2. Check if the project is a proxy cache project, if not, call the next handler in the chain.
3. Check if the upstream registry can be proxied, if not, call the next handler in the chain.
4. Check if the upstream registry supports the referrers API, if not, call the next handler in the chain.
5. Call the upstream referrers API to get the referrers list. return the referrers list if the format is valid.
7. Cache the referrers list in the redis for future use. when upstream registry is not available, the cached referrers list can be returned.

Because the proxy cache's registry interface doesn't include the referrers API, a new interface `ListReferrers` should be added to the registry client interface to support calling the referrers API in the upstream registry.

```go
type RegistryClient interface {
// existing methods...

// ListReferrers lists the referrers of the specified artifact.
ListReferrers(repository, ref string, rawQuery string) (*v1.Index, error)
}
```

The authorization of the upstream referrers API call should be handled in the existing registry client interface implementation. it uses the same authorization mechanism as other API calls to the upstream registry. but it should handle the referrers API specific URL path.
The referrers API URL path is `/v2/<repository>/referrers/<reference>`, in the method parseScope, the registry client implementation should extract the repository from the URL path and pass it to the getToken method to get the authorization token.


## Limitations

1. Because it is difficult to merge the referrers from upstream registry and local storage correctly, this proposal only return the referrers from upstream registry or local storage, but not both.
1. Only Harbor is officially supported as the upstream registry for proxy cache projects. support for other registries can be considered in future enhancements.

1. This feature only supports the referrers API defined in OCI Distribution Spec 1.1, other related APIs such as artifact signatures or SBOMs are not covered in this proposal. for example, there is an option to enable or disable the deployment security for notation or cosign signatures, because they are using the accessory API in Harbor, not the referrers API. so the proxy cache project can not proxy the accessory API.