feat(datasource/azure-pipelines-tasks): Azure DevOps API based datasource #32966

bdovaz · 2024-12-07T15:39:18Z

Changes

Initially as I do not have much experience of the Renovate API, I tried (looking at other code examples in this repository) that only changes the behavior for the case that you are running it from platform: azure because this way there is no need to create any new variable and it can use the values of RENOVATE_PLATFORM, RENOVATE_ENDPOINT and RENOVATE_TOKEN.

This PR does not solve the case that platform is a value different to azure (in that case it makes fallback to what it was already doing) because for that case I suppose that new variables will be needed and I consider that this can go in another future PR because it requires more analysis.

Context

The need comes from the fact that we currently take as a source of data the static files that are generated from this repository: https://github.com/renovatebot/azure-devops-marketplace

The problem is that task updates are not propagated at the same speed to all Azure Devops organizations so there are times when renovate mistakenly suggests updating to versions of tasks that do not exist in the organization and if you do not know why it happens it is very confusing.

There is an open discussion: #24820

Documentation (please check one with an [x])

I have updated the documentation, or
No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

Code inspection only, or
Newly added/modified unit tests, or
No unit tests but ran on a real repository, or
Both unit tests + ran on a real repository

…ally

CLAassistant · 2024-12-07T15:39:24Z

All committers have signed the CLA.

rarkins

Are you sure that this can replace the existing functionality in a backwards-compatible way?

How efficient is this approach? e.g. how many API calls and how many MB should it require each run?

Any caching possible?

rarkins · 2024-12-14T09:44:26Z

lib/modules/datasource/azure-pipelines-tasks/index.ts

-      return { releases };
+    if (platform === 'azure' && endpoint) {
+      const auth = Buffer.from(
+        `renovate:${process.env.RENOVATE_TOKEN}`,


It's not certain that RENOVATE_TOKEN is defined. Instead, use hostRules to get the token

I have used the GlobalConfig and hostRules based on other data sources but I don't know if I have done it right.

I don't see any API documentation, I've only seen the pages you recommend here: https://github.com/renovatebot/renovate/blob/main/.github/contributing.md#code

@rarkins can you answer please?

bdovaz · 2024-12-14T12:38:57Z

Are you sure that this can replace the existing functionality in a backwards-compatible way?

How efficient is this approach? e.g. how many API calls and how many MB should it require each run?

Any caching possible?

Doesn't the caching already do it with the code that was there?

https://github.com/bdovaz/renovate/blob/620fa90803e995e404e29c478b1cfcd32ad028f8/lib/modules/datasource/azure-pipelines-tasks/index.ts#L81

And the backwards-compatible thing is for when these conditions are not met that it at least does what it did before:

https://github.com/bdovaz/renovate/blob/620fa90803e995e404e29c478b1cfcd32ad028f8/lib/modules/datasource/azure-pipelines-tasks/index.ts#L36

I've also added more tests to compare function so we have 100% code coverage.

rarkins · 2024-12-17T07:19:17Z

So it caches once per 24 hours, if the user has persistent datasource caching? (note: most people running Renovate in pipelines do not have such persistence)

I don't understand your answer to my backwards-compatibility question. Is it replacing existing logic, or adding to it? If it's replacing it, is it fully backwards compatible?

How many API calls and how many MB should it require?

bdovaz · 2024-12-17T20:13:50Z

So it caches once per 24 hours, if the user has persistent datasource caching? (note: most people running Renovate in pipelines do not have such persistence)

@rarkins is there any documentation on how to implement persistence? Without documentation or examples I will hardly be able to do it since it is the first time I do a PR. As I said, the persistence there was already implemented, I haven't touched anything in this PR.

I don't understand your answer to my backwards-compatibility question. Is it replacing existing logic, or adding to it? If it's replacing it, is it fully backwards compatible?

If these conditions are met: https://github.com/renovatebot/renovate/pull/32966/files#diff-2ef2acf656366f4dd2036d71f106b4d8d0b41cd546e1349942a4e40efffde808R36

It will use the new logic, otherwise it will do what it did before. The difference is that if you work against Azure DevOps and provide a PAT (personal access token), it will resolve the task versions according to your organization and not to a static file as now (https://github.com/renovatebot/azure-devops-marketplace/blob/main/azure-pipelines-builtin-tasks.json). Not all organizations propagate task updates at the same time and with the current implementation the problem is that it suggests updates that have not yet been propagated in your organization (= do not exist yet) and it is very confusing.

In the discussion of where this PR comes from that I mention in my first post of this PR #24820 puts you in context of what I am trying to solve. One of the keys is here: renovatebot/azure-devops-marketplace#42 (comment)

How many API calls and how many MB should it require?

The API (https://dev.azure.com/{org}/_apis/distributedtask/tasks/) resolves which are the latest versions of each task so that this call can be cached and used equally for all subsequent requests. In the tests that I add you can see an example of response.

The MB depends on each organization, because in my organization I can have 5 third party extensions and another organization 50. This API lists the latest versions of the official/built-in tasks (same for all organizations) + those of third parties/marketplace (these are the ones that can not be measured, it depends on each organization).

Example API result in my organization (built-in + third party installed): 1.93 MB

rarkins · 2024-12-18T05:18:17Z

lib/modules/datasource/azure-pipelines-tasks/index.ts

+    const platform = GlobalConfig.get('platform');
+    const endpoint = GlobalConfig.get('endpoint');
+    const { token } = hostRules.find({
+      hostType: AzurePipelinesTasksDatasource.id,


I'm not sure this is set, you might want to drop this line

Do you mean the line to call "find" with “hostType”? If I remove it I think it won't work, if you see the implementation of that "find" method both the “hostType” and “url” fields have to be defined, unless I'm wrong, as I say, I'm new to TS:

renovate/lib/util/host-rules.ts

Line 133 in 98693e2

if ([search.hostType, search.url].every(is.falsy)) {

@rarkins can you answer please?

rarkins · 2024-12-18T05:18:38Z

lib/modules/datasource/azure-pipelines-tasks/index.ts

-      const releases = versions.map((version) => ({ version }));
-      return { releases };
+    if (platform === 'azure' && endpoint && token) {
+      const auth = Buffer.from(`renovate:${token}`).toString('base64');


username doesn't matter for this endpoint?

In this link it mentions it but if you look at the code example, it doesn't even use it: https://learn.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-7.2#assemble-the-request

I personally use that API and I know that the user is not necessary and you can pass any value.

@rarkins can you answer please?

rarkins · 2024-12-18T05:20:25Z

lib/modules/datasource/azure-pipelines-tasks/types.ts

Best to use zod parsing to ensure the structure matches what we expect

Can you give an example of that I have to change? I'm new to TS, I don't even know what is "zod parsing". Thanks.

@zharinov could you guide? Or even PR?

@rarkins done! I've figured out what the whole zod thing is about.

zharinov · 2024-12-22T21:02:39Z

lib/modules/datasource/azure-pipelines-tasks/index.ts

+    ttlMinutes: 24 * 60,
+  })
+  async getFallbackTasks<ResT>(url: string): Promise<ResT> {
+    const { body } = await this.http.getJson<ResT>(url);


Our http functions accept schema object, so internally it will check against it and return the proper type, so you don't have to specify generic

UPD. I see you use it above, please do here also. As a rule, your code should end up using zero generics, with all the types inferred from schema.

I was going to do it but I don't know how to convert Record<string, string[]> to a schema.

That code you mention is not from this PR, we are talking about the fallback that already existed: https://github.com/renovatebot/azure-devops-marketplace/blob/main/azure-pipelines-builtin-tasks.json

I have simply split the methods in 2 so that “the new” uses schemas.

Use z.record(z.string().array())

Done, thanks!

bdovaz · 2024-12-23T11:35:10Z

@rarkins @zharinov I think it's all done, I just need you to answer some doubts you had to see if any more changes are needed or not.

Besides, I don't understand this workflow, it fails by adding files to the tests? https://github.com/renovatebot/renovate/actions/runs/12457890306/job/34782564100

rarkins · 2024-12-23T14:53:48Z

We prefer inline test fixtures instead of dedicated file fixtures, so the test checks for that. It's possible to override and still merge

bdovaz · 2024-12-23T15:08:31Z

We prefer inline test fixtures instead of dedicated file fixtures, so the test checks for that. It's possible to override and still merge

I suppose that in this case, as it stands, it is correct and the inline is only for when there is little content.

bdovaz added 3 commits December 5, 2024 18:19

Add support for Azure DevOps API-based datasource

fde2cec

Add support for Azure DevOps API-based datasource only in Azure initi…

65d3584

…ally

Add support for Azure DevOps API-based datasource only in Azure initi…

c9b3133

…ally

Merge branch 'main' into Azure-DevOps-API-based-datasource

17c08aa

bdovaz changed the title ~~Azure DevOps API based datasource~~ feat(datasource/azure-pipelines-tasks): Azure DevOps API based datasource Dec 7, 2024

bdovaz added 2 commits December 9, 2024 09:28

Merge branch 'main' into Azure-DevOps-API-based-datasource

26a2206

Update index.ts

67ff5eb

rarkins requested changes Dec 14, 2024

View reviewed changes

bdovaz and others added 4 commits December 14, 2024 13:06

Add compare semver tests

2cead7d

Remove unneeded type

68da3b0

Merge branch 'main' into Azure-DevOps-API-based-datasource

c3360f8

Use host rules

620fa90

rarkins requested changes Dec 18, 2024

View reviewed changes

bdovaz added 2 commits December 22, 2024 21:48

Use zod schema

6063969

Rename file

ebe7e19

zharinov reviewed Dec 22, 2024

View reviewed changes

Use schema for fallback tasks

3c435cb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(datasource/azure-pipelines-tasks): Azure DevOps API based datasource #32966

feat(datasource/azure-pipelines-tasks): Azure DevOps API based datasource #32966

bdovaz commented Dec 7, 2024

CLAassistant commented Dec 7, 2024 •

edited

Loading

rarkins left a comment

rarkins Dec 14, 2024

bdovaz Dec 14, 2024

bdovaz Dec 22, 2024

bdovaz commented Dec 14, 2024 •

edited

Loading

rarkins commented Dec 17, 2024

bdovaz commented Dec 17, 2024

rarkins Dec 18, 2024

bdovaz Dec 18, 2024 •

edited

Loading

bdovaz Dec 22, 2024

rarkins Dec 18, 2024

bdovaz Dec 18, 2024

bdovaz Dec 22, 2024

rarkins Dec 18, 2024

bdovaz Dec 18, 2024

rarkins Dec 18, 2024

bdovaz Dec 22, 2024

zharinov Dec 22, 2024 •

edited

Loading

bdovaz Dec 22, 2024

zharinov Dec 22, 2024

bdovaz Dec 22, 2024

bdovaz commented Dec 23, 2024

rarkins commented Dec 23, 2024

bdovaz commented Dec 23, 2024

feat(datasource/azure-pipelines-tasks): Azure DevOps API based datasource #32966

Are you sure you want to change the base?

feat(datasource/azure-pipelines-tasks): Azure DevOps API based datasource #32966

Conversation

bdovaz commented Dec 7, 2024

Changes

Context

Documentation (please check one with an [x])

How I've tested my work (please select one)

CLAassistant commented Dec 7, 2024 • edited Loading

rarkins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdovaz commented Dec 14, 2024 • edited Loading

rarkins commented Dec 17, 2024

bdovaz commented Dec 17, 2024

Choose a reason for hiding this comment

bdovaz Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zharinov Dec 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdovaz commented Dec 23, 2024

rarkins commented Dec 23, 2024

bdovaz commented Dec 23, 2024

CLAassistant commented Dec 7, 2024 •

edited

Loading

bdovaz commented Dec 14, 2024 •

edited

Loading

bdovaz Dec 18, 2024 •

edited

Loading

zharinov Dec 22, 2024 •

edited

Loading