Allow disabling pagination or configuring limit when fetching Docker tags

[skycaptain]

What would you like Renovate to be able to do?

We're using Artifactory to proxy public Docker images, including renovate's image, which has more than 10000 tags by now. Artifactory currently has a bug where an invalid pagination link is returned (See discussion in #15030 (comment)). Suggested workarounds, like switching to "Subdomain mode" on Artifactory (#5894 (comment)) is impractical for us. Since we're running a private on-prem Artifactory instance, we do not have any API restrictions, hence we would like to workaround the Artifactory issue by either being able to configure the pagination limit ourselves or to disable pagination. Like discussed in #9176 (comment) controlling pagination limits should be an admin/self-hosted configuration option only.

If you have any ideas on how this should be implemented, please tell us here.

Adding an option to configure or remove the limit here:

renovate/lib/modules/datasource/docker/index.ts

Line 677 in 3ab24f9

const limit = ecrRegex.test(registryHost) ? 1000 : 10000;

Please note that when n=0 or n=-1 is used, the Artifactory API returns all tags without pagination.

Since different host might use different limits, an option in hostRules might be suiteable.

Is this a feature you are interested in implementing yourself?

Maybe

[rarkins]

So for Artifactory the ideal solution is no limit in the URL? And if so, is there any way we can detect the host is Artifactory without requiring configuration?

[viceice]

What does curl -sv https://docker.artifactory.test/v2/ returns as headers? Any visible artifactory header we can detect?

[skycaptain]

So for Artifactory the ideal solution is no limit in the URL?

At least for private on-prem instances, I guess so.

And if so, is there any way we can detect the host is Artifactory without requiring configuration?

What does curl -sv https://docker.artifactory.test/v2/ returns as headers? Any visible artifactory header we can detect?

Depends on whether you have access to the repository. Grabbing https://docker.artifactory.test/v2/ requires authentication. Otherwise you'll get an HTTP 401 error, which does not include any resilient hints. Here is the output from Jfrogs own cloud instance:

$ curl -sv "https://releases-docker.jfrog.io/v2/"
*   Trying 3.228.248.14:443...
* Connected to releases-docker.jfrog.io (3.228.248.14) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.jfrog.io
*  start date: Jan 25 00:00:00 2022 GMT
*  expire date: Feb 25 23:59:59 2023 GMT
*  subjectAltName: host "releases-docker.jfrog.io" matched cert's "*.jfrog.io"
*  issuer: C=US; O=DigiCert Inc; CN=GeoTrust TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.
> GET /v2/ HTTP/1.1
> Host: releases-docker.jfrog.io
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< Date: Wed, 13 Apr 2022 07:58:07 GMT
< Content-Type: application/json;charset=ISO-8859-1
< Content-Length: 87
< Connection: keep-alive
< Docker-Distribution-Api-Version: registry/2.0
< WWW-Authenticate: Bearer realm="https://releases-docker.jfrog.io/artifactory/api/docker/docker/v2/token",service="releases-docker.jfrog.io"
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host releases-docker.jfrog.io left intact

If I use authentication, I get the following headers:

$ curl -u [redacted] -sv "https://releases-docker.jfrog.io/v2/"
...
< HTTP/1.1 200 OK
< Date: Wed, 13 Apr 2022 08:12:45 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-JFrog-Version: Artifactory/7.37.11 73711900
< X-Artifactory-Id: 1faa83cd21f661ba33a12d3d1407edd7e9a253f5
< X-Artifactory-Node-Id: releases-artifactory-primary-2
< Strict-Transport-Security: max-age=15724800; includeSubDomains
...

However, let's say I want to use docker images that are accessible without authentication (e.g. anonymus access). In this case, I would most likely not set any credentials in the hostRules. So, I can still access the docker image, but would get HTTP 401 on accessing https://docker.artifactory.test/v2/ directly.

Interesting enough, I get the same headers with HTTP 404 errors (when the URL includes the image path), even without authentication (jfrog/jfrog-cli-v2-jf is the image here):

$ curl -sv "https://releases-docker.jfrog.io/v2/jfrog/jfrog-cli-v2-jf"
...
< HTTP/1.1 404
< Date: Wed, 13 Apr 2022 08:16:23 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-JFrog-Version: Artifactory/7.37.11 73711900
< X-Artifactory-Id: 6283f32de04013f62e0593c00da461e9393cd108
< X-Artifactory-Node-Id: releases-artifactory-primary-0
< Strict-Transport-Security: max-age=15724800; includeSubDomains
...

From what I have seen, there is no way to determin whether the instance is on-prem or a cloud instance from these headers alone.

[schuch]

Suggested workarounds, like switching to "Subdomain mode" on Artifactory [...]

The pagination error caught us, although our Artifactory Docker Registry is running in subdomain mode.

We have an open support request at JFrog and they raised the following public ticket out of it: RTFACT-26815.

Unfortunately, JFrog does not treat this topic with high priority. Incomprehensibly, they recently decided to close the bug ticket and turned it into a feature request RTFACT-27035.

[rarkins]

If you're running locally, you can try setting RENOVATE_PAGINATE_ALL=true in env to remove the 10 page restriction.

[schuch]

If you're running locally, you can try setting RENOVATE_PAGINATE_ALL=true in env to remove the 10 page restriction.

RENOVATE_PAGINATE_ALL=true did not solve the issue for us in our self-hosted environment (renovate 32.2.2)

How is this flag related to list docker tags pagination? The documentation suggests it is only related to the pagination of github requests.

[rarkins]

Good point, it may not apply to all just yet. I think for Artifactory your request was to simply allow all tags and not impose any limit?

[skycaptain]

Yes. I think RENOVATE_PAGINATE_ALL to disable the 10 page restriction won't help. Pagination in general is the issue, as Artifactory reports wrong pagination links. A variable to change the limit per page (currently 10000) or a mechanism to disable pagination for Artifactory instances could solve the issue.

[tharakadesilva]

We are also facing this issue, so I was looking at submitting a PR.

The recommended options by @skycaptain were to either:

1. Allow a custom page limit
2. Disable pagination

(2) can be achieved by:

• Not setting n
• Setting n to either 0 or -1

I feel that providing an explicit option for (2) is not necessary. Instead, we can ask the consumer to provide 0 or -1 in their hostRule. In our code, if a value <= 0 is provided, we can opt not to use n or default to 0; what do you think?

---

For a similar limitation with page limits, Renovate has set the page limit of ECR repositories to 1000: https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/docker/index.ts#L821.

What do you think about not doing the custom limit option mentioned above and adding a similar check to disable pagination for artifactory?

Advantages:

• Better developer experience for folks using Artifactory as a Docker mirror as they don't have to manually do this or find out about it when they face an issue.

Disadvantages:

• We will have to keep adding exceptions for bugs in different registries.

[rarkins]

Do you think it can be done by:

• Trying default
• If errored, check if the server is Artifactory (can this be determined from response headers alone?)
• If artifactory, save that information about the host in memory and then retry without pagination?
• On subsequent requests, skip pagination from the start because we already know the host is artifactory

That way once we determine that a host is artifactory the first time, we can cache that information and not to try/fail/retry every time.

[tharakadesilva]

Thank you, @rarkins, for the feedback!

To highlight, with this approach, we are going to make three calls the first time:

• /artifactory/v2/renovate/renovate/tags/list?n=10000 (200)
• /artifactory/v2/renovate/renovate/tags/list?n=10000&last=32.32-slim (404)
• /artifactory/v2/renovate/renovate/tags/list (200)

After implementing caching, subsequent requests should only be making the last call.

I started working on an implementation here (I didn't create a PR yet as I haven't added the caching, unit tests, manually tested, etc.).

1. Is this what you were thinking as well?
2. Could you help me with the caching part (or point me where it is used / docs), please?

[rarkins]

Can we determine the host is artifactory using the first response's headers and would that help?

[tharakadesilva]

Hi @rarkins, yes, we can do that. It's going to add a conditional on each iteration though. But, as I am typing this out, I feel it's probably not that much of an overhead...

[Morl99]

Hello,

I wanted to chime in to drop a note, that there is a chance, that this issue will actually be fixed upstream by JFrog, see https://www.jfrog.com/confluence/display/JFROG/Artifactory+Release+Notes#ArtifactoryReleaseNotes-Artifactory7.46 and https://www.jfrog.com/jira/browse/RTFACT-27278.

We are in contact with the JFrog support about this and finally, they fixed something. Since this is only available in the cloud release, we are still waiting for an official release in order to test this. But imho it looks promising.

[setchy]

We're scheduled to upgrade on Nov 13th, so will be able to report back shortly after this as to whether RTFACT-27278 did resolve the root issue(s)

[Morl99]

That is not what I observe. For me, in 7.46.17 (which we have installed), this now works as expected. although I wonder, if your example is also right, and you expect something else? The last parameter is basically the cursor, and it keeps moving. This can be easily tested, if you decrese n, to 10 for example. See this:

http /artifactory/api/docker/docker-hub-remote/v2/renovate/renovate/tags/list n==10
HTTP/1.1 200 OK
Link: </v2/renovate/renovate/tags/list?last=10&n=10>; rel="next"

{
    "name": "renovate/renovate",
    "tags": [
        "0",
        "0-slim",
        "0.0",
        "0.0-slim",
        "0.0.1",
        "0.0.1-slim",
        "0.0.2",
        "0.0.3",
        "0.0.4",
        "0.0.4-slim"
    ]
}

and then the next one

http /artifactory/api/docker/docker-hub-remote/v2/renovate/renovate/tags/list n==10  last==10 
HTTP/1.1 200 OK
Link: </v2/renovate/renovate/tags/list?last=20&n=10>; rel="next"

{
    "name": "renovate/renovate",
    "tags": [
        "11",
        "11.35",
        "11.35.10",
        "11.35.11",
        "11.35.12",
        "11.35.13",
        "11.35.14",
        "11.35.15",
        "11.35.16",
        "11.35.3"
    ]
}

[skycaptain]

AFAIK renovate does not use the API endpoint, so in my example the next page requested would be https://artifactory.[redacted].org/v2/renovate/renovate/tags/list?last=10000&n=10000, which returns 404, as the name of the virtual repository (docker in this case) is missing in the next link. Artifactory should return link: </v2/docker/renovate/renovate/tags/list?last=10000&n=10000>; rel="next" instead.

[skycaptain]

However, your comment reminds me of another workaround. Since we are using Nginx as a reverse proxy one could add a rewrite rule to redirect the broken link to the API endpoint of a fixed docker repository. Jfrog does sth similar in their Helm chart template: https://github.com/jfrog/charts/blob/master/stable/artifactory-ha/values.yaml#L1760. Here one would need to set $repo to a hard-coded repository, which is the downside.

[dc2tom]

Hi, we use Sonatype Nexus as a proxy for Docker hub due to renovate not being available in ECR public at the present time. We now find that renovate is unable to patch itself, due to the API call being made to the Docker list tags operation of:

https://our-docker-hub-proxy:5000/v2/renovate/renovate/tags/list?n=10000

When looking at the renovate project in docker hub, there are a huge number of image tags - is it possible this could be cleaned up? At present with the 10000 limit we only get as far as tag: "32.209.0-slim" which is obviously far from the latest..

It's a bit frustrating not having renovate be able to renovate itself anymore :)

I don't think this limitation is dependant on the docker hub proxy being used, it seems that renovate either needs to be able to be configured to ask for a higher number of tags to be listed when calling the API, or the renovate repo have the number of tags reduced... it would be great if the Docker list-tags api supported listing tags in descending order but this doesn't seem to be possible.

Thanks in advance for any advice.

[mmaeller]

Hi @dc2tom,

We also got a Sonatype Nexus and the once per day update works well.
We are using a custom helm chart where the values.yaml will be updated.

image:
  registry: our-docker-hub-proxy:5000
  repository: renovate/renovate
  tag: 34.159.1

The corresponding renovate.json is quite basic:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:base",
    ":automergeDigest",
    ":automergeBranch",
    ":automergeMinor",
    ":disableDependencyDashboard",
    ":skipStatusChecks"
  ],
  "labels": [
    "renovate"
  ],
  "reviewers": [ "user1", "user2", "..."  ],
  "ignoreDeps": [],
  "packageRules": [],
  "enabledManagers": ["helm-values"],
  "branchPrefix": "renovate-",
  "prHourlyLimit": 0,
  "prConcurrentLimit": 0,
  "rebaseWhen": "auto"
}

What manager are you using?

[dc2tom]

Hi @mmaeller it's the Docker manager in this case that pulls the renovate image from docker hub; there are more than 10,000 tags there and renovate only seems to ever retrieve the first 10000 so it never finds the newest image tags

It works perfectly for dockerfile and .gitlab-ci.yml files that reference docker images that have fewer than that many tags in their repos

[mmaeller]

Hi @mmaeller it's the Docker manager in this case that pulls the renovate image from docker hub; there are more than 10,000 tags there and renovate only seems to ever retrieve the first 10000 so it never finds the newest image tags

It works perfectly for dockerfile and .gitlab-ci.yml files that reference docker images that have fewer than that many tags in their repos

But they should have the same docker datasource.
https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/docker/index.ts#L896
Loops 20 times, so 200k tags should bei possible.

[setchy]

What is the version of Renovate that you're running @dc2tom?

[NPhMKgbDNy1M]

I also have the problem that Renovate can not update itself. I use version "35.54.0". Our nexus is supporting the "list?last=32.209.0-slim" but misses the Link-Header on "list?n=10000".

[chris-str-cst]

We are facing the same issue regarding Renovate updating itself. We are running on 36.54.3 any fixes are appreciated

[rarkins]

Docker Hub still contains 25612 releases of renovate/renovate at this point. Meanwhile ghcr.io/renovatebot/renovate has less than 3000, so anyone experiencing this problem for updating Renovate could switch as a workaround.

As a full solution, perhaps hostRules with a paginationSize and paginationLimit parameter would work? The Docker datasource already queries hostRules.

One reason why this is remaining on status:requirements is that none of the maintainers have this problem nor do we have Nexus or similar to test against.

[viceice]

this max help for self-hosted users:

https://docs.renovatebot.com/self-hosted-experimental/#renovate_x_docker_max_pages

[XFNeo]

Is the problem resolved for artifactory?
On the latest version I have the same problem

DEBUG: Dependency envoyproxy/envoy has unsupported/unversioned value latest (versioning=docker) (repository=PROD.Platform.Cloud_Core/ingres-gateway)
DEBUG: GET https://artifactorycn.company.com:17064/v2/envoyproxy/envoy/tags/list?n=10000&last=distroless-dev-b9f66fec8d678936a3273f19d6001cd227d50b4c = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=11) (repository=PROD.Platform.Cloud_Core/ingres-gateway)
DEBUG: JFrog Artifactory: Retrying Tags for https://artifactorycn.company.com:17064/envoyproxy/envoy using library/ path between JFrog virtual repository and image (repository=PROD.Platform.Cloud_Core/ingres-gateway)
DEBUG: GET https://artifactorycn.company.com:17064/v2/envoyproxy/library/envoy/tags/list?n=10000 = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=263) (repository=PROD.Platform.Cloud_Core/ingres-gateway)
DEBUG: Datasource 404 (repository=PROD.Platform.Cloud_Core/ingres-gateway)
       "datasource": "docker",
       "packageName": "artifactorycn.company.com:17064/envoyproxy/envoy",
       "url": "https://artifactorycn.company.com:17064/v2/envoyproxy/library/envoy/tags/list?n=10000"
DEBUG: Failed to look up docker package artifactorycn.company.com:17064/envoyproxy/envoy (repository=PROD.Platform.Cloud_Core/ingres-gateway, packageFile=Dockerfile, dependency=artifactorycn.company.com:17064/envoyproxy/envoy)

https://artifactorycn.company.com:17064/v2/envoyproxy/envoy/tags/list returns 14к + tags

As written in artifactory docs parameter last must be numeric
https://jfrog.com/help/r/jfrog-rest-apis/list-docker-repositories
but you provide the tag name
https://artifactorycn.company.com:17064/v2/envoyproxy/envoy/tags/list?n=10000&last=distroless-dev-b9f66fec8d678936a3273f19d6001cd227d50b4c

[XFNeo]

I guess problem can be on our artifactory. We are investigating.

[skycaptain]

We switched back to DockerHub from using ghcr.io because Artifactory is currently unable to paginate the tags from upstream ghcr.io. This causes Renovate to not update some images, as it can't pull all the tags from upstream.

$ crane ls ghcr.io/renovatebot/renovate | wc -l
    8498
$ crane ls artifactory.company.com/ghcr/renovatebot/renovate | wc -l
     400

$ crane ls renovate/renovate | wc -l
   32611
$ crane ls artifactory.company.com/docker/renovate/renovate | wc -l
   32603

[Goose29]

@skycaptain I encountered the same issue, and contacted JFrog.

I was informed the following system property artifactory.remote.fetching.list.maximum.iteration=500 can be used to fix this issue.

I was not able to find this property documented anywhere in their official documentation or on the internet.

[skycaptain]

Right, we received the same answer from JFrog. However, my concern with that response is that it merely artificially increases the limit before Artifactory still silently aborts the pagination in the remote repository. This will eventually lead to the same issue if not set to an incredibly high number. Additionally, if I recall correctly, JFrog couldn't confirm whether this has any side effects, such as client connections timing out before Artifactory finishes fetching tags from the remote. In the meantime, I've discovered that the root of the issue might lie in the Distribution API itself, which is currently under consideration for redesign to address such use cases.