Renovate removes digest from docker image after mutex timeout

[mvitz]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab, Renovate version 38.44.3

Please tell us more about your question or problem

Today there were a few runs where Renovate opened a MR for removing the digest of Docker images.
Because in my setting, digest updates are set to auto-merge, these were even merged.

Looking at the logs (see below for INFO, and attachment for a full DEBUG log) it contains ERROR entries for several Docker image updates with a mutex timeout.

I would expect that in case of such a failure, renovate would skip any actions and not remove the digest.

What looks a bit suspicious is that the regex manager detected 3076 dependencies, which is far too much. The attached DEBUG log shows that the it's a result of finding the same packages over and over. This packages are found by "regexManagers:dockerfileVersions" which is included via a preset.

Logs (if relevant)

Logs

...
 INFO: Repository started (repository=xxx/yyy)
       "renovateVersion": "38.45.0"
 INFO: Dependency extraction complete (repository=xxx/yyy, baseBranch=production)
       "stats": {
         "managers": {
           "dockerfile": {"fileCount": 2, "depCount": 3},
           "gitlabci": {"fileCount": 1, "depCount": 6},
           "maven": {"fileCount": 1, "depCount": 3},
           "maven-wrapper": {"fileCount": 1, "depCount": 1},
           "npm": {"fileCount": 1, "depCount": 5},
           "nvm": {"fileCount": 1, "depCount": 1},
           "regex": {"fileCount": 5, "depCount": 3076}
         },
         "total": {"fileCount": 12, "depCount": 3095}
       }
ERROR: lookupUpdates error (repository=xxx/yyy, packageFile=Dockerfile)
       "currentDigest": "sha256:4a1f95535b852da36a339d51fe8f1144e31a2716da7bbfa152a5b4e58a0cdf84",
       "currentValue": "jre-21.0.3-cds-slim-stream-musl",
       "datasource": "docker",
       "packageName": "bellsoft/liberica-runtime-container",
       "digestOneAndOnly": undefined,
       "followTag": null,
       "lockedVersion": undefined,
       "pinDigests": true,
       "rollbackPrs": false,
       "isVulnerabilityAlert": undefined,
       "updatePinnedDependencies": true,
       "unconstrainedValue": false,
       "err": {
         "message": "timeout while waiting for mutex to become available",
         "stack": "Error: timeout while waiting for mutex to become available\n    at Object.<anonymous> (/usr/local/renovate/node_modules/.pnpm/async-mutex@0.5.0/node_modules/async-mutex/lib/errors.js:4:21)\n    at Module._compile (node:internal/modules/cjs/loader:1358:14)\n    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1416:10)\n    at Module.load (node:internal/modules/cjs/loader:1208:32)\n    at Function.Module._load (node:internal/modules/cjs/loader:1024:12)\n    at Module.require (node:internal/modules/cjs/loader:1233:19)\n    at require (node:internal/modules/helpers:179:18)\n    at Object.<anonymous> (/usr/local/renovate/node_modules/.pnpm/async-mutex@0.5.0/node_modules/async-mutex/lib/Semaphore.js:4:16)\n    at Module._compile (node:internal/modules/cjs/loader:1358:14)\n    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1416:10)\n    at Module.load (node:internal/modules/cjs/loader:1208:32)\n    at Function.Module._load (node:internal/modules/cjs/loader:1024:12)\n    at Module.require (node:internal/modules/cjs/loader:1233:19)\n    at require (node:internal/modules/helpers:179:18)\n    at Object.<anonymous> (/usr/local/renovate/node_modules/.pnpm/async-mutex@0.5.0/node_modules/async-mutex/lib/Mutex.js:4:19)\n    at Module._compile (node:internal/modules/cjs/loader:1358:14)\n    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1416:10)\n    at Module.load (node:internal/modules/cjs/loader:1208:32)\n    at Function.Module._load (node:internal/modules/cjs/loader:1024:12)\n    at Module.require (node:internal/modules/cjs/loader:1233:19)"
       }
ERROR: lookupUpdates error (repository=xxx/yyy, packageFile=.gitlab-ci.yml)
       "currentDigest": "sha256:2e5515536bf789843b48030fdca3e3719463ba85c43b1da7d5687f5997b79d26",
       "currentValue": "latest",
       "datasource": "docker",
       "packageName": "docker",
       "digestOneAndOnly": undefined,
       "followTag": null,
       "lockedVersion": undefined,
       "pinDigests": true,
       "rollbackPrs": false,
       "isVulnerabilityAlert": undefined,
       "updatePinnedDependencies": true,
       "unconstrainedValue": false,
       "err": {
         "message": "timeout while waiting for mutex to become available",
         "stack": "Error: timeout while waiting for mutex to become available\n    at Object.<anonymous> (/usr/local/renovate/node_modules/.pnpm/async-mutex@0.5.0/node_modules/async-mutex/lib/errors.js:4:21)\n    at Module._compile (node:internal/modules/cjs/loader:1358:14)\n    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1416:10)\n    at Module.load (node:internal/modules/cjs/loader:1208:32)\n    at Function.Module._load (node:internal/modules/cjs/loader:1024:12)\n    at Module.require (node:internal/modules/cjs/loader:1233:19)\n    at require (node:internal/modules/helpers:179:18)\n    at Object.<anonymous> (/usr/local/renovate/node_modules/.pnpm/async-mutex@0.5.0/node_modules/async-mutex/lib/Semaphore.js:4:16)\n    at Module._compile (node:internal/modules/cjs/loader:1358:14)\n    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1416:10)\n    at Module.load (node:internal/modules/cjs/loader:1208:32)\n    at Function.Module._load (node:internal/modules/cjs/loader:1024:12)\n    at Module.require (node:internal/modules/cjs/loader:1233:19)\n    at require (node:internal/modules/helpers:179:18)\n    at Object.<anonymous> (/usr/local/renovate/node_modules/.pnpm/async-mutex@0.5.0/node_modules/async-mutex/lib/Mutex.js:4:19)\n    at Module._compile (node:internal/modules/cjs/loader:1358:14)\n    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1416:10)\n    at Module.load (node:internal/modules/cjs/loader:1208:32)\n    at Function.Module._load (node:internal/modules/cjs/loader:1024:12)\n    at Module.require (node:internal/modules/cjs/loader:1233:19)"
       }
 INFO: Branch created (repository=xxx/yyy, branch=renovate/bellsoft-liberica-runtime-container-jre-21.0.3-cds-slim-stream-musl)
       "commitSha": "59cd5b2045aad41ca211c39f01954cd1ec4f0925"
 INFO: PR created (repository=xxx/yyy, branch=renovate/bellsoft-liberica-runtime-container-jre-21.0.3-cds-slim-stream-musl)
       "pr": 427,
       "prTitle": "Update bellsoft/liberica-runtime-container:jre-21.0.3-cds-slim-stream-musl Docker digest to"
 INFO: Branch created (repository=xxx/yyy, branch=renovate/docker-latest)
       "commitSha": "8da67d5876b13a2c44e4403d8ced08686ea0eb66"
 INFO: PR created (repository=xxx/yyy, branch=renovate/docker-latest)
       "pr": 428,
       "prTitle": "Update docker:latest Docker digest to"
 INFO: Repository finished (repository=xxx/yyy)
       "cloned": true,
       "durationMs": 179279
...

[github-actions[bot]]

Hi, please format any copy/pasted code or logs so they're readable.

You can find a Markdown code formatting guide here as well as some GitHub-specific information formatting code blocks here.

Thanks, the Renovate team

[rarkins]

Please split the log by \n and also ideally include more logs, including a good run and the bad run. Also clarify what exactly did renovate do. Eg create a new PR to remove digest or update an existing one, etc

[rarkins]

Today there were a few runs where Renovate opened a MR for removing the digest of Docker images

Was this PR only created at the time of the error?
Did this PR keep the version the same and only remove the SHA? Can you provide a diff or screenshot?

[mvitz]

Yes these were only created at the time of the error and yes it only removed the digest (see Screenshot below).
The next successful run after that opened another PR to pin the digest and added it again.

Right now, I suspect that the mutex timeout occurs because Renovate finds >3k Dependencies within a Dockerfile which I suspect is a result of me still using regexManagers:dockerfileVersions. I changed that to "customManagers:dockerfileVersions" and are waiting for the next run to see if that helps.

[vitoyucepi]

Hi @mvitz,
Could you check if the number of dependencies has increased in the last few runs with renovate>=38.41. It looks to me like you're experiencing the same dependency growth bug I described in #30906.

[mvitz]

Yes, I can confirm that the number of the dependencies which are found by multiple custom managers increase over several runs.

[mvitz]

So the mutex timeout may be the result of too many dependencies. Nevertheless, I think changing the digest in case of that mutex error is another problem that maybe should be fixed.