Never use private repo for public dependencies

[vertti]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Gitlab. Renovate 39.60.0

Please tell us more about your question or problem

I have in my renovate.json

  "packageRules": [
    {
      "description": "Use S3 registry for MyCore dependencies",
      "matchDatasources": [
        "maven"
      ],
      "matchDepTypes": [
        "dependencies",
        "plugins"
      ],
      "registryUrls": [
        "s3://mycore-redacted/java/"
      ],
      "matchPackageNames": [
        "/^@mycore/",
        "/^mycore/",
        "/^com.mycompany.mycore/"
      ]
    },
    {
      "description": "Default Maven repositories for other dependencies",
      "matchDatasources": [
        "maven"
      ],
      "matchDepTypes": [
        "dependencies"
      ],
      "registryUrls": [
        "https://repo.maven.apache.org/maven2/"
      ],
      "matchPackageNames": [
        "!/^@mycore/",
        "!/^mycore/",
        "!/^com.mycompany.mycore/"
      ]
    },
    {
      "description": "Default Maven repositories for other plugins",
      "matchDatasources": [
        "maven"
      ],
      "matchDepTypes": [
        "plugins"
      ],
      "registryUrls": [
        "https://plugins.gradle.org/m2"
      ],
      "matchPackageNames": [
        "!/^@mycore/",
        "!/^mycore/",
        "!/^com.mycompany.mycore/"
      ]
    },

So basically, "hey, for all things mycore, please use our private s3 bucket as repository. And for all other plugins and dependencies, use the standard public repos".

But in the Renovate logs, i'm still getting lines like:

{"name":"renovate","hostname":"runner-szgaykhx-project-40382358-concurrent-0","pid":16,"level":20,"logContext":"RUqzGz56MIBWLU0JS6lec","repository":"mycompany/mycore/switch","failedUrl":"s3://mycore-redacted/java/org/springframework/boot/org.springframework.boot.gradle.plugin/maven-metadata.xml","err":{"name":"AccessDenied","$fault":"client","$metadata":{"httpStatusCode":403,"requestId":"SYJK9G5A2RHFJ6VB","extendedRequestId":"eagsHFnKWXEMKVkamae7bmBXA8PPSkwAp06h5Re+6Lm3bk2UtAj4JmsgRxwiAG8jxYz/+XEgNHM=","attempts":1,"totalRetryDelay":0},"Code":"AccessDenied","RequestId":"SYJK9G5A2RHFJ6VB","HostId":"eagsHFnKWXEMKVkamae7bmBXA8PPSkwAp06h5Re+6Lm3bk2UtAj4JmsgRxwiAG8jxYz/+XEgNHM=","message":"Access Denied","stack":"AccessDenied: Access Denied\n    at throwDefaultError (/usr/local/renovate/node_modules/.pnpm/@smithy+smithy-client@3.4.5/node_modules/@smithy/smithy-client/dist-cjs/index.js:836:20)\n    at /usr/local/renovate/node_modules/.pnpm/@smithy+smithy-client@3.4.5/node_modules/@smithy/smithy-client/dist-cjs/index.js:845:5\n    at de_CommandError (/usr/local/renovate/node_modules/.pnpm/@aws-sdk+client-s3@3.701.0/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4831:14)\n    at processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at /usr/local/renovate/node_modules/.pnpm/@smithy+middleware-serde@3.0.10/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n    at /usr/local/renovate/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.696.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:485:18\n    at /usr/local/renovate/node_modules/.pnpm/@smithy+middleware-retry@3.0.28/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38\n    at /usr/local/renovate/node_modules/.pnpm/@aws-sdk+middleware-flexible-checksums@3.701.0/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:263:18\n    at /usr/local/renovate/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.696.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22\n    at /usr/local/renovate/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.696.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:138:14\n    at /usr/local/renovate/node_modules/.pnpm/@aws-sdk+middleware-logger@3.696.0/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22\n    at downloadS3Protocol (/usr/local/renovate/lib/modules/datasource/maven/util.ts:176:10)\n    at downloadMavenXml (/usr/local/renovate/lib/modules/datasource/maven/util.ts:397:23)\n    at MavenDatasource.fetchVersionsFromMetadata (/usr/local/renovate/lib/modules/datasource/maven/index.ts:96:49)\n    at MavenDatasource.getReleases (/usr/local/renovate/lib/modules/datasource/maven/index.ts:130:30)\n    at getRegistryReleases (/usr/local/renovate/lib/modules/datasource/index.ts:85:15)\n    at mergeRegistries (/usr/local/renovate/lib/modules/datasource/index.ts:169:19)\n    at fetchReleases (/usr/local/renovate/lib/modules/datasource/index.ts:322:15)\n    at lookupUpdates (/usr/local/renovate/lib/workers/repository/process/lookup/index.ts:142:56)\n    at /usr/local/renovate/lib/workers/repository/process/fetch.ts:86:12"},"msg":"Maven S3 lookup error: unknown error","time":"2024-12-10T07:09:55.185Z","v":0}

So it's still trying to get for example Spring Framework from our private s3 bucket.

Is this is a bug or is my configuration wrong somehow?

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[github-actions[bot]]

Hi there,

A maintainer wants you to find and share a specific log message (packageFiles with updates) from your repository or environment.

If you self-host Renovate: make sure you run Renovate with LOG_LEVEL=debug to get the debug log messages! Next, open the debug-level logs and search for packageFiles with updates. This text marks the start of a structured log message that shows every package file, dependency, and update that Renovate found.

Find the relevant dependency/dependencies in the log message, and copy/paste those parts into this discussion. If you do not know which bits we need, you can copy/paste the full log message.

Read the Renovate docs, Troubleshooting to learn more about getting the docs, and getting the correct type of logs.

Thanks, the Renovate team

[rarkins]

Using the example you gave:

               {
                 "commitMessageTopic": "plugin org.springframework.boot",
                 "currentValue": "3.4.0",
                 "currentVersion": "3.4.0",
                 "datasource": "maven",
                 "depName": "org.springframework.boot",
                 "depType": "plugin",
                 "fileReplacePosition": 175,
                 "fixedVersion": "3.4.0",
                 "managerData": {
                   "fileReplacePosition": 175,
                   "packageFile": "app/build.gradle"
                 },
                 "packageName": "org.springframework.boot:org.springframework.boot.gradle.plugin",
                 "packageScope": "org.springframework.boot",
                 "registryUrl": "https://plugins.gradle.org/m2",
                 "registryUrls": [
                   "https://plugins.gradle.org/m2/",
                   "s3://mycore-artifacts/java"
                 ],
                 "versioning": "gradle",
                 "warnings": [],
                 "updates": []
               }
             ],
             "packageFile": "app/build.gradle"
           },

Note the registryUrls. The behavior does not seem to match what you say the config is. The only next step unless you can debug it yourself is for a reproduction for us to debug. See instructions below.

BTW is this s3:// string contained within the repo too? Maybe it's being detected as part of the pom.xml extract function, and is not caused by any packageRules, but could be fixable by packageRules.

[vertti]

Yes, the "s3:// string" is of course contained in the repo too, meaning it's defined in the build.gradle, how would gradle otherwise know to fetch dependencies from there. How would you fix it with packageRules?

[rarkins]

You look at the packageFiles with updates message for every dependency with the s3:// url in registryUrls, and for all the ones which you don't want that, you have a package rule which matches those packages and sets registryUrls to be the public registry only. Alternatively if the list of packages which use s3 is smaller, then set one rule which overrides registryUrls to public-only, and a second one which overrides it again to public+private or private only, for the private packages.

[vertti]

I don't quite get how that would differ from the config i posted in the original message. That matches my private packages and sets the repositories to just s3 and another rule that matches everything except the private packages and sets just the public repo. I don't know why that doesn't result in s3 being gone from the public deps.

[rarkins]

There's a good chance you'll discover a config mistake when you're putting together a minimal reproduction. Eg maybe one of your matchDepTypes is too narrow

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

1. Read our guide on creating a minimal reproduction.
2. Go to our minimal reproduction template repository.
3. Select the Use this template button to create a new repository based on the template.
4. Work on the minimal reproduction in your own repository.
5. Fill out the information in your repository's README.md.
6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

If you need help with running renovate on your minimal reproduction repository, please refer to our Running Renovate guide.
Good luck,

The Renovate team