Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix slow cache restore on Windows #515

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

antontroshin
Copy link

@antontroshin antontroshin commented Nov 19, 2024

Description:
The issue with running setup-go and restoring relatively big cache files, causes the cache restore process to be very slow on Github Action Windows runners.
Similar to the change that was made in version v5.0.0 (PR #411), I've made a change to override the default GOCACHE and GOMODCACHE variables on Windows runners to be using drive D.
On average, restoring ~1.7Gb cache on Windows pipelines took about 13-15 minutes to extract.
With this change, the extraction time was reduced to ~1m.

Restore cache logs before the change:

Wed, 20 Nov 2024 02:24:29 GMT Cache Size: ~1784 MB (1870225672 B)
Wed, 20 Nov 2024 02:24:30 GMT Received 1870225672 of 1870225672 (100.0%), 295.9 MBs/sec
Wed, 20 Nov 2024 02:37:21 GMT Cache restored successfully
Restore elapsed: 12 min 51 sec
Wed, 20 Nov 2024 01:38:54 GMT Cache Size: ~1784 MB (1870225672 B)
Wed, 20 Nov 2024 01:38:55 GMT Received 1870225672 of 1870225672 (100.0%), 118.7 MBs/sec
Wed, 20 Nov 2024 01:54:19 GMT Cache restored successfully
Restore elapsed: 15 min 24 sec

Restore cache logs after the change:

Wed, 20 Nov 2024 03:38:03 GMT Cache Size: ~1780 MB (1866088667 B)
Wed, 20 Nov 2024 03:38:04 GMT Received 1866088667 of 1866088667 (100.0%), 197.3 MBs/sec 
Wed, 20 Nov 2024 03:38:59 GMT Cache restored successfully
Restore elapsed: 55 sec
Wed, 20 Nov 2024 03:38:10 GMT Cache Size: ~1780 MB (1866088667 B)
Wed, 20 Nov 2024 03:38:11 GMT Received 1866088667 of 1866088667 (100.0%), 197.6 MBs/sec
Wed, 20 Nov 2024 03:39:06 GMT Cache restored successfully
Restore elapsed: 55 sec
Wed, 20 Nov 2024 04:04:07 GMT Cache Size: ~1780 MB (1866088667 B)
Wed, 20 Nov 2024 04:04:07 GMT Received 1866088667 of 1866088667 (100.0%), 88.8 MBs/sec
Wed, 20 Nov 2024 04:05:05 GMT Cache restored successfully
Restore elapsed: 1 min 2 sec

Solution:

At first, I tried to implement a solution similar to #411 by using symlink, however, during testing, it was clear that the GOCACHE and GOMODCACHE directories were created early by the tooling, which did not present an easy way to create symlinks without deleting the already in-use directories and introducing bigger code changes.

The solution to override the path for env vars as early as possible only on Windows runners is easier.

This PR adds the code to make this change by default for Windows runners, this "workaround" is possible to use via the Github Action code itself, by manually overriding the env vars.
Preferably, this change would be accepted and merged (after review of course), so users can enjoy this improvement by default without manually modifying each and every Github action.

Related issue:
Numerous related issues were raised regarding this issue, some improvements were made to address it.

Check list:

  • Mark if documentation changes are required.
  • Mark if tests were added or updated to cover the changes.

Use D drive for faster cache restore

Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
…rs for Windows

Signed-off-by: Anton Troshin <anton@diagrid.io>
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin antontroshin changed the title Add GOCACHE AND GOMODCACHE symlink on Windows Fix slow cache restore on Windows Nov 20, 2024
@antontroshin antontroshin marked this pull request as ready for review November 20, 2024 04:33
@antontroshin antontroshin requested a review from a team as a code owner November 20, 2024 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant