Metadata retrieval errors when using `COREPACK_NPM_REGISTRY` in combination with Sonatype Nexus #479

PayBas · 2024-05-13T21:26:56Z

@aduh95 @arcanis
#436 has broken COREPACK_NPM_REGISTRY in combination with Sonatype Nexus repository manager.

ARG YARN_VERSION
ARG NPM_REGISTRY_URL="https://nexus.megacorp.com/repository/npmjs-proxy/"
ENV COREPACK_NPM_REGISTRY $NPM_REGISTRY_URL

RUN  npm config set registry $NPM_REGISTRY_URL \
  && npm install --global corepack@latest \
  && corepack enable \
  && corepack install --global yarn@${YARN_VERSION} \
  && yarn config set --home npmRegistryServer $NPM_REGISTRY_URL

Results in:

Installing yarn@4.2.1...
Internal Error: Server answered with HTTP 400 when performing the request to https://nexus.megacorp.com/repository/npmjs-proxy//@yarnpkg/cli-dist/4.2.1; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting
    at fetch (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:22769:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async fetchAsJson (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:22776:20)
    at async fetchTarballURLAndSignature (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:22724:27)
    at async installVersion (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:22987:52)
    at async Engine.ensurePackageManager (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:23449:32)
    at async InstallGlobalCommand.installFromDescriptor (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:23846:5)
    at async InstallGlobalCommand.execute (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:23828:9)
    at async InstallGlobalCommand.validateAndExecute (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:20954:22)
    at async _Cli.run (/home/jenkins/.npm-global/lib/node_modules/corepack/dist/lib/corepack.cjs:21929:18)

Nexus doesn't provide metadata at the ${npmRegistryUrl}/${packageName}/${version} url.
I believe it only serves metadata at the ${npmRegistryUrl}/${packageName} url.

So this change breaks corepack for Nexus and perhaps Artifactory as well.

Had to revert to corepack 0.26.0

Update

I've found a public Nexus instance to show what I mean:
Web view: https://nexus3.onap.org/#browse/browse:npm:%40yarnpkg%2Fcli-dist
Artifact: https://nexus3.onap.org/repository/npm/%40yarnpkg/cli-dist/-/cli-dist-4.2.1.tgz
Metadata: https://nexus3.onap.org/repository/npm/%40yarnpkg/cli-dist

There is no metadata available at https://nexus3.onap.org/repository/npm/%40yarnpkg/cli-dist/4.2.1 !

The text was updated successfully, but these errors were encountered:

BasixKOR · 2024-07-16T08:37:07Z

Is this reported to Sonatype as well? It seems like the incompatiblity lies on Nexus itself rather than the Corepack implementation.

aduh95 · 2024-07-16T14:11:10Z

Possibly a duplicate of #498. Can you test with Corepack 0.29.x?

jasonschroeder-sfdc · 2024-07-16T16:32:54Z

Sonatype changed behavior in NEXUS-42854 , mentioned in the release notes , but it doesn't seem to be a sufficient fix.

PayBas · 2024-07-30T11:35:18Z

Sonatype changed behavior in NEXUS-42854 , mentioned in the release notes , but it doesn't seem to be a sufficient fix.

Indeed NXRM 3.70.0 has changed this behavior, but it is still not compatible with corepack.

https://registry.npmjs.com/@yarnpkg/cli-dist/4.3.1

{
  "name": "@yarnpkg/cli-dist",
  "version": "4.3.1",
  "license": "BSD-2-Clause",
  "_id": "@yarnpkg/cli-dist@4.3.1",
  "bin": {
    "yarn": "bin/yarn.js",
    "yarnpkg": "bin/yarn.js"
  },
  "dist": {
    "shasum": "409cdab09b1f792d4e6bad5aa687320943b0d4cc",
    "tarball": "https://registry.npmjs.org/@yarnpkg/cli-dist/-/cli-dist-4.3.1.tgz",
    "fileCount": 5,
    "integrity": "sha512-Vpi/Nbu2SLXGRdKvuxhT0WNe3jOL/LM0Wl58yxUN9WcaQnCYyuIILNS3R35lujao1ZXoAN35d9vAsevzStDreQ==",
    "signatures": [
      {
        "sig": "MEYCIQDXpotyvZmuMzXobmJiotkmf/yvk+2IcPLdleVWTjZHlAIhAJA1Lh0fuNvB6nRSi5GzocTWyNej/F346E7HhuUGefSD",
        "keyid": "SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"
      }
    ],
    "unpackedSize": 2747220
  },
  "engines": {
    "node": ">=18.12.0"
  },
  "_npmUser": {
    "name": "yarnbot",
    "email": "nison.mael+yarnbot.npm@gmail.com"
  },
  "repository": {
    "url": "ssh://git@github.com/yarnpkg/berry.git",
    "type": "git",
    "directory": "packages/yarnpkg-cli"
  },
  "directories": {},
  "_hasShrinkwrap": false,
  "_npmOperationalInternal": {
    "tmp": "tmp/cli-dist_4.3.1_1718952731591_0.6413408756169847",
    "host": "s3://npm-registry-packages"
  }
}

https://nexus.megacorp.com/repository/npmjs-proxy/%40yarnpkg/cli-dist/4.3.1

{
  "_id": "@yarnpkg/cli-dist@4.3.1",
  "maintainers": [
    {
      "name": "daniel15",
      "email": "npm@d.sb"
    },
    {
      "name": "bestander",
      "email": "bestander@gmail.com"
    },
    {
      "name": "cpojer",
      "email": "christoph.pojer@gmail.com"
    },
    {
      "name": "arcanis",
      "email": "nison.mael@gmail.com"
    },
    {
      "name": "yarnbot",
      "email": "nison.mael+yarnbot.npm@gmail.com"
    }
  ],
  "license": "BSD-2-Clause",
  "dist-tags": {
    "v3": "3.8.3",
    "latest": "4.3.1"
  },
  "versions": {
    huge list of versions
  },
  "_rev": "66-3a3158dea3a016d10f8c72876b5d7be4",
  "name": "@yarnpkg/cli-dist",
  "time": {
    "created": "2021-04-09T11:18:13.039Z",
    "modified": "2024-07-25T12:13:04.535Z",
    "2.4.1": "2021-04-09T11:18:13.374Z",
    "3.0.0-rc.1": "2021-04-12T08:37:17.751Z",
    "3.0.0-rc.2": "2021-04-12T14:54:14.320Z",
    "3.0.0-rc.3": "2021-06-03T14:55:53.984Z",
    "3.0.0-rc.4": "2021-06-03T15:35:43.365Z",
    "2.4.2": "2021-06-03T16:01:55.314Z",
    "3.0.0": "2021-07-26T16:10:51.916Z",
    "3.0.1": "2021-08-22T21:01:32.655Z",
    "3.0.2": "2021-09-03T12:25:05.172Z",
    "3.1.0": "2021-10-25T14:57:38.351Z",
    "3.1.1": "2021-11-26T13:36:24.297Z",
    "3.2.0": "2022-02-21T13:04:45.372Z",
    "3.2.1": "2022-05-13T10:35:13.285Z",
    "3.2.2": "2022-07-21T12:52:26.715Z",
    "3.2.3": "2022-08-24T18:35:28.355Z",
    "3.2.4": "2022-10-05T16:44:57.592Z",
    "3.3.0": "2022-11-16T09:06:30.157Z",
    "3.3.1": "2022-12-20T16:05:09.449Z",
    "4.0.0-rc.35": "2023-01-09T01:13:52.390Z",
    "4.0.0-rc.36": "2023-01-18T16:59:29.806Z",
    "4.0.0-rc.37": "2023-01-29T12:51:45.270Z",
    "3.4.0": "2023-02-01T09:28:36.780Z",
    "3.4.1": "2023-02-01T16:15:20.181Z",
    "4.0.0-rc.38": "2023-02-04T13:11:54.127Z",
    "4.0.0-rc.39": "2023-02-08T07:53:10.481Z",
    "4.0.0-rc.40": "2023-03-05T16:51:01.498Z",
    "3.5.0": "2023-03-16T21:30:03.314Z",
    "4.0.0-rc.41": "2023-03-27T11:28:58.453Z",
    "4.0.0-rc.42": "2023-03-30T07:49:51.073Z",
    "3.5.1": "2023-05-01T18:58:44.561Z",
    "4.0.0-rc.43": "2023-05-01T20:13:10.935Z",
    "4.0.0-rc.44": "2023-05-17T14:51:46.551Z",
    "3.6.0": "2023-06-01T21:15:42.274Z",
    "4.0.0-rc.45": "2023-06-01T21:56:27.007Z",
    "3.6.0-git.20230603.hash-45f6ecc9": "2023-06-03T17:11:27.541Z",
    "3.6.0-git.20230603.hash-9645df4d": "2023-06-03T17:32:48.119Z",
    "3.6.0-git.20230603.hash-3c8237cb": "2023-06-03T17:38:39.424Z",
    "4.0.0-rc.46": "2023-06-22T08:20:11.007Z",
    "4.0.0-rc.47": "2023-06-29T09:12:39.333Z",
    "3.6.1": "2023-06-30T22:12:43.702Z",
    "4.0.0-rc.48": "2023-07-02T15:01:11.596Z",
    "4.0.0-rc.49": "2023-08-17T09:34:15.045Z",
    "3.6.2": "2023-08-17T19:10:10.089Z",
    "3.6.3": "2023-08-23T22:14:03.188Z",
    "4.0.0-rc.50": "2023-08-23T22:46:04.799Z",
    "4.0.0-rc.51": "2023-09-17T14:22:43.249Z",
    "4.0.0-rc.52": "2023-09-29T22:02:14.739Z",
    "3.6.4": "2023-10-03T22:19:02.653Z",
    "4.0.0-rc.53": "2023-10-03T23:34:15.182Z",
    "4.0.0": "2023-10-22T16:56:59.265Z",
    "4.0.1": "2023-10-28T15:26:56.339Z",
    "4.0.2": "2023-11-14T09:22:36.270Z",
    "3.7.0": "2023-11-14T18:04:35.535Z",
    "4.1.0": "2024-01-30T15:49:15.231Z",
    "3.8.0": "2024-02-01T20:19:11.188Z",
    "3.8.1": "2024-03-04T22:24:18.570Z",
    "4.1.1": "2024-03-04T23:11:57.106Z",
    "4.2.0": "2024-05-02T16:22:33.560Z",
    "3.8.2": "2024-05-02T17:04:36.111Z",
    "4.2.1": "2024-05-02T17:51:55.024Z",
    "4.2.2": "2024-05-08T17:50:42.768Z",
    "4.3.0": "2024-06-10T18:52:21.867Z",
    "4.3.1": "2024-06-21T06:52:11.814Z",
    "3.8.3": "2024-06-21T15:32:33.189Z"
  },
  "readme": "",
  "readmeFilename": "",
  "repository": {
    "url": "ssh://git@github.com/yarnpkg/berry.git",
    "type": "git",
    "directory": "packages/yarnpkg-cli"
  }
}

I've opened a support ticket at Sonatype in the hopes that they change the version-specific metadata to include a singlar version instead of a versions object containing all versions.

yasinkocak · 2024-07-30T12:24:37Z

We got the same issue with our organization, we can not update corepack

PayBas · 2024-07-30T13:51:42Z

Yes, this is known issue, it is fixed in the upcoming 3.71.0 release, which is currently targeted to come out on August 6th.

Direct quote from Sonatype.

aduh95 · 2024-08-09T10:17:06Z

Is this still an issue?

smsalisbury · 2024-08-13T17:15:15Z

Yes, this is known issue, it is fixed in the upcoming 3.71.0 release, which is currently targeted to come out on August 6th.

Direct quote from Sonatype.

3.71.0 was released last week. Can anyone who has already upgraded confirm that the release fixed this issue for them?

PayBas · 2024-08-19T18:23:09Z

A quick test shows that unfortunately, the issue persists. I cannot see any difference between Nexus 3.70.1 and 3.71.0. There is also no mention of the issue in the 3.71.0 release notes

I'll reopen the Sonatype support ticket.

I guess we're stuck on corepack@0.26.0 for at least another couple weeks.

Update: reply from Sonatype:

I do apologize, but there appears to have been some slippage in the release schedule for this fix.
It is actually marked as being released with the 3.72.0 version.

av-mc · 2024-08-21T16:07:07Z

Thanks @PayBas for the update.
I'm having the mismatch hash issue (which is solved in issue 296) with corepack@0.26.0, so I have to update to corepack@0.28.0, and now I'm stuck with this issue.
Any suggestion to work around?

PayBas · 2024-08-22T11:46:09Z

Thanks @PayBas for the update. I'm having the mismatch hash issue (which is solved in issue 296) with corepack@0.26.0, so I have to update to corepack@0.28.0, and now I'm stuck with this issue. Any suggestion to work around?

As long as your CI server and all your developers use the exact same COREPACK_NPM_REGISTRY value, then the "packageManager": "yarn@..." hash should be stable.

Just replace the hash in your package.json with the one in your error message. That's how we fixed it.

av-mc · 2024-08-22T13:19:57Z

Just replace the hash in your package.json with the one in your error message. That's how we fixed it.

Awesome. This works for me with corepack@0.26.0. Thank you so much!

Robbson · 2024-08-28T19:15:34Z

This error regrading Sonatype Nexus reminds me of a similar issue when trying to download a package manager using Corepack, starting with Yarn:

Internal Error: Server answered with HTTP 404 when performing the request to 
https://****/repository/proxy_npm_official/@yarnpkg/cli-dist/4.3.1; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting
at fetch (/home/containeruser/lib/node_modules/corepack/dist/lib/corepack.cjs:21616:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async fetchAsJson (/home/containeruser/lib/node_modules/corepack/dist/lib/corepack.cjs:21623:20)
at async fetchTarballURLAndSignature (/home/containeruser/lib/node_modules/corepack/dist/lib/corepack.cjs:21571:27)
at async installVersion (/home/containeruser/lib/node_modules/corepack/dist/lib/corepack.cjs:21833:52)
at async Engine.ensurePackageManager (/home/containeruser/lib/node_modules/corepack/dist/lib/corepack.cjs:22310:32)
at async InstallGlobalCommand.installFromDescriptor (/home/containeruser/lib/node_modules/corepack/dist/lib/corepack.cjs:22707:5)
at async Promise.all (index 0)
at async InstallGlobalCommand.execute (/home/containeruser/lib/node_modules/corepack/dist/lib/corepack.cjs:22685:5)
at async InstallGlobalCommand.validateAndExecute (/home/containeruser/lib/node_modules/corepack/dist/lib/corepack.cjs:19835:22)

The issue appears as soon as we switch to Corepack 0.24.0 or later. I guess it's all related to this decision.

That's pretty strange because we don't have any install/download issues at all for packages coming from Nexus V3.66 using npm, pnpm or yarn. So Corepack does something special which leads to a 404 error instead.

Of course, you could remove the COREPACK_NPM_REGISTRY env variable so it fetches the tool from the original yarn source like before 0.24.0. But that way other package managers like pnpm can't be installed because without COREPACK_NPM_REGISTRY the original npm registry is requested, which is not available for us.

So COREPACK_NPM_REGISTRY has to be enabled or disabled depending on which package manager you are going to install? That's kind of ridiculous, isn't it? I guess that's why Corepack is still described as experimental in the NodeJS docs.

So switching back to 0.23.0 is the best and easiest solution for us so far.

PayBas · 2024-09-09T09:07:47Z

3.72.0 release notes mention:

NEXUS-43608 : Requests for version-specific scoped npm metadata return the expected metadata.

This should be the fix. Haven't had the opportunity to test it yet though.

jackmtpt · 2024-09-09T13:54:01Z

3.72.0 includes a partial fix it seems - the version-specific metadata is there...but the .dist.tarball property still points at the upstream feed URL instead of pointing back into the Nexus Repository server 🤦

PayBas · 2024-09-09T22:00:57Z

3.72.0 includes a partial fix it seems - the version-specific metadata is there...but the .dist.tarball property still points at the upstream feed URL instead of pointing back into the Nexus Repository server 🤦

Sigh. I'll open another ticket...

Update: Sonatype has acknowledged the issue and are tracking it under internal ticket NEXUS-44175. Whether this will result in a 3.72.1 or if we have to wait for 3.73.0 remains to be seen. It probably depends on whether the issue breaks current deployments.

PayBas · 2024-09-26T08:42:59Z

Work on NEXUS-44175 has been completed. It didn't make the cut for 3.73.0, so it will be in the 3.74.0 release. That release is currently targeted to come out in the first week of November.

Guess we'll have to wait quite a while longer. 😞

PayBas · 2024-11-19T00:16:09Z

https://help.sonatype.com/en/sonatype-nexus-repository-3-74-0-release-notes.html contains:

NEXUS-44175 - Requests for version-specific npm package metadata returns the correct download URL.

Haven't had time to test it yet, but with any luck this might finally solve this issues.

Update: tested 3.74.0, but there's still an issue with the tarball metadata value, so that will probably still prevent it from working (although I haven't actually tested it with corepack yet).

Created yet another support ticket.

the URL is still not correct (the @4.1.0 does not belong in the URL). I have entered defect NEXUS-45088 to have this addressed.

jackmtpt · 2024-12-05T14:29:07Z

Update: tested 3.74.0, but there's still an issue with the tarball metadata value, so that will probably still prevent it from working (although I haven't actually tested it with corepack yet).

I have tested it with corepack; it doesn't work. My support ticket has also been linked to NEXUS-45088.

jackmtpt · 2024-12-23T09:47:43Z

Apparently this issue isn't expected to be fixed until the February release at the earliest.

PayBas mentioned this issue Jul 23, 2024

downloads package managers only from npm by default #495

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metadata retrieval errors when using `COREPACK_NPM_REGISTRY` in combination with Sonatype Nexus #479

Metadata retrieval errors when using `COREPACK_NPM_REGISTRY` in combination with Sonatype Nexus #479

PayBas commented May 13, 2024 •

edited by aduh95

Loading

BasixKOR commented Jul 16, 2024

aduh95 commented Jul 16, 2024

jasonschroeder-sfdc commented Jul 16, 2024

PayBas commented Jul 30, 2024 •

edited

Loading

yasinkocak commented Jul 30, 2024

PayBas commented Jul 30, 2024

aduh95 commented Aug 9, 2024

smsalisbury commented Aug 13, 2024

PayBas commented Aug 19, 2024 •

edited

Loading

av-mc commented Aug 21, 2024 •

edited

Loading

PayBas commented Aug 22, 2024 •

edited

Loading

av-mc commented Aug 22, 2024

Robbson commented Aug 28, 2024

PayBas commented Sep 9, 2024

jackmtpt commented Sep 9, 2024

PayBas commented Sep 9, 2024 •

edited

Loading

PayBas commented Sep 26, 2024

PayBas commented Nov 19, 2024 •

edited

Loading

jackmtpt commented Dec 5, 2024

jackmtpt commented Dec 23, 2024

Metadata retrieval errors when using COREPACK_NPM_REGISTRY in combination with Sonatype Nexus #479

Metadata retrieval errors when using COREPACK_NPM_REGISTRY in combination with Sonatype Nexus #479

Comments

PayBas commented May 13, 2024 • edited by aduh95 Loading

Update

BasixKOR commented Jul 16, 2024

aduh95 commented Jul 16, 2024

jasonschroeder-sfdc commented Jul 16, 2024

PayBas commented Jul 30, 2024 • edited Loading

yasinkocak commented Jul 30, 2024

PayBas commented Jul 30, 2024

aduh95 commented Aug 9, 2024

smsalisbury commented Aug 13, 2024

PayBas commented Aug 19, 2024 • edited Loading

av-mc commented Aug 21, 2024 • edited Loading

PayBas commented Aug 22, 2024 • edited Loading

av-mc commented Aug 22, 2024

Robbson commented Aug 28, 2024

PayBas commented Sep 9, 2024

jackmtpt commented Sep 9, 2024

PayBas commented Sep 9, 2024 • edited Loading

PayBas commented Sep 26, 2024

PayBas commented Nov 19, 2024 • edited Loading

jackmtpt commented Dec 5, 2024

jackmtpt commented Dec 23, 2024

Metadata retrieval errors when using `COREPACK_NPM_REGISTRY` in combination with Sonatype Nexus #479

Metadata retrieval errors when using `COREPACK_NPM_REGISTRY` in combination with Sonatype Nexus #479

PayBas commented May 13, 2024 •

edited by aduh95

Loading

PayBas commented Jul 30, 2024 •

edited

Loading

PayBas commented Aug 19, 2024 •

edited

Loading

av-mc commented Aug 21, 2024 •

edited

Loading

PayBas commented Aug 22, 2024 •

edited

Loading

PayBas commented Sep 9, 2024 •

edited

Loading

PayBas commented Nov 19, 2024 •

edited

Loading