[GOG]: Refactor almost all GOG components #3020

imLinguin · 2023-08-30T19:47:13Z

This is massive PR which will take some more time. Most critical parts are practically done.

At this moment it's required to use gogdl binary overwrite, obtainable from https://github.com/Heroic-Games-Launcher/heroic-gogdl/tree/download_refactor (if using python wrapper from bin directory any AbortController signals are not handled, pyinstaller binary works) PR includes latest gogdl binaries

Features checklist at this moment (will be updated):

NOTE: content-system features (version rollback, beta channels, language change) are not available for Linux native games due to backup installers limitations

Added dependencies:

sanitize-html - changelog renderer santizer
react-beautiful-dnd - REDmod integration load order management
@cospired/i18n-iso-languages - mapping of language codes to human readable string for scriptinterpreter.exe

Related issues this may solve:
#2791 #3013 #2995 #2958 #3012 #1874

Use the following Checklist if you have changed something on the Backend or Frontend:

Tested the feature and it's working on a current and clean install.
Tested the main App features and they are still working on a current and clean install. (Login, Install, Play, Uninstall, Move games, etc.)
Created / Updated Tests (If necessary)
Created / Updated documentation (If necessary)

…info based on branch

… modify install is working now

arielj · 2023-09-02T00:04:23Z

I'm experiencing issues with this branch.

I get this in the logs first:

[1/9/2023, 20:56:10] GOGDL_CONFIG_PATH=/home/ariel/.config/heroic/gogdlConfig /home/ariel/dev/oss/HeroicGamesLauncher/public/bin/linux/gogdl --auth-config-path /home/ariel/.config/heroic/gog_store/auth.json info gog-redist --os windows
[API] INFO: Getting info from products endpoint for id: gog-redist
[API] ERROR: Request failed <Response [404]>
Traceback (most recent call last):
  File "gogdl/cli.py", line 51, in <module>
  File "gogdl/cli.py", line 47, in main
  File "gogdl/dl/manager.py", line 36, in calculate_download_size
  File "gogdl/dl/manager.py", line 89, in get_download_metadata
TypeError: 'NoneType' object does not support item assignment
[521477] Failed to execute script 'cli' due to unhandled exception!
[1/9/2023, 20:56:12] GOGDL_CONFIG_PATH=/home/ariel/.config/heroic/gogdlConfig /home/ariel/dev/oss/HeroicGamesLauncher/public/bin/linux/gogdl --auth-config-path /home/ariel/.config/heroic/gog_store/auth.json redist --ids ISI --path /home/ariel/.config/heroic/tools/redist/gog
usage: gogdl [-h] [--version] --auth-config-path AUTH_CONFIG_PATH
             {auth,download,repair,update,import,info,launch,save-sync,save-clear}
             ...
gogdl: error: argument command: invalid choice: 'redist' (choose from 'auth', 'download', 'repair', 'update', 'import', 'info', 'launch', 'save-sync', 'save-clear')

I had an update for Skyrim and after clicking the update button, the game status changed to queue but the updated never started, this is the log for that, there's no error in GOG's side:

[1/9/2023, 20:56:29] GOGDL_CONFIG_PATH=/home/ariel/.config/heroic/gogdlConfig /home/ariel/dev/oss/HeroicGamesLauncher/public/bin/linux/gogdl --auth-config-path /home/ariel/.config/heroic/gog_store/auth.json info 1711230643 --os windows
[API] INFO: Getting info from products endpoint for id: 1711230643
[DOWNLOAD_MANAGER] INFO: Checking compatibility of The Elder Scrolls V: Skyrim Special Edition with windows
[DOWNLOAD_MANAGER] INFO: Game is compatible
[DOWNLOAD_MANAGER] INFO: Depot version: 2
[DOWNLOAD_MANAGER] INFO: Getting Meta data
[API] INFO: Getting Dependencies repository
[API] INFO: Getting repository manifest
[DOWNLOAD_MANAGER] INFO: Calculating download size
{"download_size": 100641555, "disk_size": 102930955, "dlcs": [{"title": "The Elder Scrolls V: Skyrim Anniversary Upgrade", "app_name": "1162721350"}], "buildId": "55846306920288567", "languages": ["de-DE", "fr-FR", "pl-PL", "ja-JP", "en-US", "zh-Hans", "ru-RU", "es-ES", "it-IT"], "folder_name": "Skyrim Anniversary Edition", "versionEtag": "\"3232439858\"", "versionName": "1.6.659.0.8"}

But there's an error in heroic's general logs:

(20:58:03) INFO:    [Gog]:              Getting game metadata: GOGDL_CONFIG_PATH=/home/ariel/.config/heroic/gogdlConfig /home/ariel/dev/oss/HeroicGamesLauncher/public/bin/linux/gogdl --auth-config-path /home/ariel/.config/heroic/gog_store/auth.json info 1666688320 --os windows

(20:58:06) ERROR:   [Gog]:              TypeError: Cannot read properties of undefined (reading '*')
    at Module.getInstallInfo$3 [as getInstallInfo] (/home/ariel/dev/oss/HeroicGamesLauncher/build/electron/main.7afa43da.js:4534:38)
    at async /home/ariel/dev/oss/HeroicGamesLauncher/build/electron/main.7afa43da.js:17355:20
    at async node:electron/js2c/browser_init:2:98394

Then I tried to download a game, the dialog never ends calculating the download/install sizes, this is the log for that:

[1/9/2023, 20:58:31] GOGDL_CONFIG_PATH=/home/ariel/.config/heroic/gogdlConfig /home/ariel/dev/oss/HeroicGamesLauncher/public/bin/linux/gogdl --auth-config-path /home/ariel/.config/heroic/gog_store/auth.json info 1666688320 --os windows
[API] INFO: Getting info from products endpoint for id: 1666688320
[DOWNLOAD_MANAGER] INFO: Checking compatibility of Hero of the Kingdom II with windows
[DOWNLOAD_MANAGER] INFO: Game is compatible
[DOWNLOAD_MANAGER] INFO: Depot version: 2
[DOWNLOAD_MANAGER] INFO: Getting Meta data
[API] INFO: Getting Dependencies repository
[API] INFO: Getting repository manifest
[DOWNLOAD_MANAGER] INFO: Calculating download size
{"download_size": 0, "disk_size": 0, "dlcs": [], "buildId": "56003675960523808", "languages": ["en-US"], "folder_name": "Hero of the Kingdom II", "versionEtag": "\"697727299\"", "versionName": "1.25"}

Same, no error there in GOG's logs, but same error in the general logs:

(21:02:03) INFO:    [Gog]:              Running command: GOGDL_CONFIG_PATH=/home/ariel/.config/heroic/gogdlConfig /home/ariel/dev/oss/HeroicGamesLauncher/public/bin/linux/gogdl --auth-config-path /home/ariel/.config/heroic/gog_store/auth.json auth
(21:02:03) DEBUG:   [Backend]:          {"free":132835164160,"diskSize":982820896768,"message":"123.71 GiB / 915.32 GiB","validPath":true}
(21:02:03) INFO:    [Gog]:              Getting game metadata: GOGDL_CONFIG_PATH=/home/ariel/.config/heroic/gogdlConfig /home/ariel/dev/oss/HeroicGamesLauncher/public/bin/linux/gogdl --auth-config-path /home/ariel/.config/heroic/gog_store/auth.json info 1666688320 --os windows

(21:02:07) ERROR:   [Gog]:              TypeError: Cannot read properties of undefined (reading '*')
    at Module.getInstallInfo$3 [as getInstallInfo] (/home/ariel/dev/oss/HeroicGamesLauncher/build/electron/main.7afa43da.js:4534:38)
    at async /home/ariel/dev/oss/HeroicGamesLauncher/build/electron/main.7afa43da.js:17355:20
    at async node:electron/js2c/browser_init:2:98394

imLinguin · 2023-09-02T09:09:58Z

At this moment it's required to use gogdl binary overwrite, obtainable from https://github.com/Heroic-Games-Launcher/heroic-gogdl/tree/download_refactor (if using python wrapper from bin directory any AbortController signals are not handled, pyinstaller binary works)

Looks like this is your issue

arielj · 2023-09-02T19:05:42Z

At this moment it's required to use gogdl binary overwrite, obtainable from https://github.com/Heroic-Games-Launcher/heroic-gogdl/tree/download_refactor (if using python wrapper from bin directory any AbortController signals are not handled, pyinstaller binary works)

Looks like this is your issue

oh, I missed that, any reason why you are not including the updated binaries in the PR? that would make it easier to test

imLinguin · 2023-09-02T19:21:24Z

I'm still working on the gogdl part on separate branch which isn't in the GitHub actions scope. I'll update binaries soon

imLinguin · 2023-09-04T10:01:57Z

Pre-built binaries are now available on this pull request Heroic-Games-Launcher/heroic-gogdl#43

arielj · 2023-09-10T14:58:09Z

I tested installing, uninstalling and running games. Also changing the language.

The automatic redist installation works great. One suggestion here is to show some feedback to the user that we are installing dependencies. Currently the button shows Playing but it's actually installing some things in the background. I imagine something like we have to the Ubisoft installation information. Just so users don't think it's just taking to long to start without any feedback.

Maybe the Modify Installation dialog's button can be Confirm or Apply instead of OK?

imLinguin · 2023-09-10T20:09:58Z

About redist I still need to add logic to check if we should trigger redist update/installation before the game installation. Currently it runs for all games that have any redist

Updating state on frontend is a good idea, I'll get to that soon.

Apply button also sounds reasonable here.

Bitwolfies · 2023-09-11T01:17:43Z

Question, will this allow the DLC's that only appear when using Galaxy with Cyberpunk to work? Just curious.

imLinguin · 2023-09-11T05:37:58Z

If you mean Galaxy exclusive content like in game items, the answer is no Communication Service implementation is separate thing https://github.com/imLinguin/comet

Bitwolfies · 2023-09-11T06:01:15Z

Gotcha, refactor still looks super exciting, thanks for all the work you put in!

…and native) for use by steam runtime

imLinguin · 2023-12-14T06:15:15Z

it's not clear why that gets updated

If you uninstall games we don't remove unused redists immediately, because they might be needed by some game you'll install in the same session, kind of network usage optimization.

arielj · 2023-12-14T16:14:29Z

__redist folder is replaced with a new management system in the form of Galaxy Common Redistributables - in tools directory. The support directory was also moved to gogdl config directory.

what does this mean? you lost me there

is it expected that the files are removed from the game folder? monkey island is 400mb less when installed with this PR and the game doesn't work so I thought it was missing some files

arielj · 2023-12-14T16:16:23Z

About missing size it's tricky, we can't predict it the same way we do with games, I'll try to find some solution.

Not a big deal with this though, if it's not easy we can maybe just hide that info

I'll look into the download queue, I changed a few things there...

one comment though is that the download queue itself shows the correct status of the installed games, the issue happens in the library and in the game details pages

Also, are you running those games on clean prefix? I'm pretty sure redists should be installed

yes, all clean, no previous prefixes (removing the prefix when uninstalling the game before the next test)

arielj · 2023-12-14T16:18:14Z

it's not clear why that gets updated

If you uninstall games we don't remove unused redists immediately, because they might be needed by some game you'll install in the same session, kind of network usage optimization.

what's confusing to me is that I uninstalled all games and heroic tells me it's updating Galaxy Common Redistributables on the next start, why is it updating something if I don't even have GOG games installed? is there some check performed to know if it actually need to update that tool? cause I can trigger an update at any time, even if it was just installed or updated a minute ago

imLinguin · 2023-12-14T17:40:38Z

what does this mean? you lost me there

I'm bad at explaining things lmao.

Redistributables are no longer in game directory CONFIG_DIR/tools/redist/gog is a new place.
All support files are moved to one location too CONFIG_DIR/gogdlConfig/heroic-gogdl/APP_NAME

arielj · 2023-12-14T20:14:10Z

what does this mean? you lost me there

I'm bad at explaining things lmao.

Redistributables are no longer in game directory CONFIG_DIR/tools/redist/gog is a new place. All support files are moved to one location too CONFIG_DIR/gogdlConfig/heroic-gogdl/APP_NAME

just curious, why is that needed? this could be an issue if users want to save space in some drive installing games in another and we move things to the other drive without telling them (for Monkey island 2 it was 400mb, and it's a fairly small game)

I can also imagine this as an issue if, for some reason you delete heroic's config folder, you would have to repair all the games after, right?

I think that if I run verify and repair then those folders appear into the game folder, is that not expected then? what if I import a game? will it move those folder to the new location? if not, will heroic understand the files are in the game folder in some cases?

imLinguin · 2023-12-14T21:01:35Z

I think that if I run verify and repair then those folders appear into the game folder, is that not expected then

That's not supposed to happen, good call

imLinguin · 2023-12-14T21:03:08Z

why is that needed?

for redistributables we save space as we can re-use them across multiple games.

Support directories are not that big, I think a few kilobytes is fine

arielj · 2023-12-15T02:39:55Z

For some reason, both STALKER call of prypiat and monkey island 2 worked fine once today, but they failed other times (I keep installing uninstalling to try things).

One new thing I noticed is that when I import a game, after the first time I try to launch it, heroic starts thinking there's an update for the game (and there's not really an update available)

arielj · 2023-12-15T02:43:41Z

Support directories are not that big, I think a few kilobytes is fine

not sure I understand why this is moved into heroic's config folder though, why can't those files be in the game's install directory? is it to make it easier to do something else?

MattSturgeon · 2023-12-15T21:04:56Z

Redistributables are no longer in game directory CONFIG_DIR/tools/redist/gog is a new place.
All support files are moved to one location too CONFIG_DIR/gogdlConfig/heroic-gogdl/APP_NAME

Shouldn't that really be in XDG_DATA_HOME (or maybe XDG_CACHE_HOME)?

XDG_CONFIG_HOME is meant to be exclusively for configuration; the kinda stuff a user might track in a dotfiles repo for example.

Or am I conflating "CONFIG_DIR" and XDG_CONFIG_HOME and you actually mean something else?

flavioislima · 2024-01-04T16:32:12Z

Testing here and so far so good for all main features:

Login
Logout
List games
Install
Uninstall
Modify installation

One thing that might be redundant is this option though:

Because we already have a setting to Ignore Game updates. That I believe does the same.

EDIT: ok, my bad. They don't do the same since you can downgrade to a previous version with the new setting. NIce.

Replaced new prerequisites with redist introduced in this PR

arielj · 2024-01-06T19:18:06Z

I merged #3393 and I imagine it will create a conflict for the status this PR adds, keep the change of your PR when fixing the conflict (I was just removing the old message from main because it was not correct)

Etaash-mathamsetty · 2024-01-23T03:04:44Z

@Bitwolfies
We didn't want to make this information public at this time, so Linguin added a shush. Good for you that you found out anyway. The plan was to reveal it with 2.13's release.

…uncher into gogdl_refactor

arielj

I'll merge this so we can test it better during development, it's been here for tooooo long.

imLinguin · 2024-01-27T18:52:15Z

Galaxy for Linux era begins now

imLinguin added 9 commits August 25, 2023 23:59

initial changes to downloading

2fd3101

feat: changelog with html sanitizer, gogdl config sandboxing

c844302

ui for gog modfiy install, and minor additions for obtaining install …

e8c6b65

…info based on branch

split reusable frontend components, add Cyberpunk REDmod support, gog…

7e3f61b

… modify install is working now

re-enable playtime sync

c1dd3e2

feat: private branches, small fixes to redMod

f1e1130

impov: redmod windows support

d2aff7e

feat: redistributable management

d9e1f75

feat: support instruction parsing compliant with backup installers

9a79fa2

imLinguin added the pr:wip WIP, don't merge. label Aug 30, 2023

imLinguin added 3 commits August 30, 2023 21:52

fix: lint

d766c20

improv: improvements to backup installers interop

fe1788d

feat: setup should work on windows

dc0d230

add very important comment

21dda3a

imLinguin added 6 commits September 13, 2023 16:36

improv: avoid refreshing library too much

4489c1f

Pull upstream main changes

feed9b5

i18n: generate locales

7b7b758

chore: remove merge leftovers

c138cba

linux: default to sniper steam runtime

4770278

improv: use steam_compat_install_path env var on Linux (wine, proton …

d1769f8

…and native) for use by steam runtime

imLinguin mentioned this pull request Dec 14, 2023

Add support for GoG Rollback feature #3318

Closed

fix: redist not being installed sometimes

7ae8b57

fix: library refresh + game push race condition

2680b41

Pull upstream main

f1bdf6c

Replaced new prerequisites with redist introduced in this PR

imLinguin added 4 commits January 6, 2024 22:14

Resolve conflicts afer merging Heroic-Games-Launcher#3393

d741c03

fix: let the uninstall command know if we are removing the prefix

abf4ddf

🤫

551abdb

Pull upstream main v2.12.0

9cd8b35

This comment has been minimized.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Merge branch 'main' of github.com:Heroic-Games-Launcher/HeroicGamesLa…

4ee3548

…uncher into gogdl_refactor

arielj approved these changes Jan 27, 2024

View reviewed changes

arielj merged commit 2f88dc9 into Heroic-Games-Launcher:main Jan 27, 2024
9 checks passed

Heroic-Games-Launcher locked and limited conversation to collaborators Jan 27, 2024

imLinguin deleted the gogdl_refactor branch January 27, 2024 18:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GOG]: Refactor almost all GOG components #3020

[GOG]: Refactor almost all GOG components #3020

imLinguin commented Aug 30, 2023 •

edited

Loading

arielj commented Sep 2, 2023

imLinguin commented Sep 2, 2023

arielj commented Sep 2, 2023

imLinguin commented Sep 2, 2023

imLinguin commented Sep 4, 2023

arielj commented Sep 10, 2023

imLinguin commented Sep 10, 2023 •

edited

Loading

Bitwolfies commented Sep 11, 2023

imLinguin commented Sep 11, 2023

Bitwolfies commented Sep 11, 2023

imLinguin commented Dec 14, 2023

arielj commented Dec 14, 2023

arielj commented Dec 14, 2023

arielj commented Dec 14, 2023

imLinguin commented Dec 14, 2023

arielj commented Dec 14, 2023 •

edited

Loading

imLinguin commented Dec 14, 2023

imLinguin commented Dec 14, 2023

arielj commented Dec 15, 2023 •

edited

Loading

arielj commented Dec 15, 2023

MattSturgeon commented Dec 15, 2023

flavioislima commented Jan 4, 2024 •

edited

Loading

arielj commented Jan 6, 2024

This comment has been minimized.

This comment was marked as off-topic.

Etaash-mathamsetty commented Jan 23, 2024 •

edited

Loading

arielj left a comment

imLinguin commented Jan 27, 2024

[GOG]: Refactor almost all GOG components #3020

[GOG]: Refactor almost all GOG components #3020

Conversation

imLinguin commented Aug 30, 2023 • edited Loading

arielj commented Sep 2, 2023

imLinguin commented Sep 2, 2023

arielj commented Sep 2, 2023

imLinguin commented Sep 2, 2023

imLinguin commented Sep 4, 2023

arielj commented Sep 10, 2023

imLinguin commented Sep 10, 2023 • edited Loading

Bitwolfies commented Sep 11, 2023

imLinguin commented Sep 11, 2023

Bitwolfies commented Sep 11, 2023

imLinguin commented Dec 14, 2023

arielj commented Dec 14, 2023

arielj commented Dec 14, 2023

arielj commented Dec 14, 2023

imLinguin commented Dec 14, 2023

arielj commented Dec 14, 2023 • edited Loading

imLinguin commented Dec 14, 2023

imLinguin commented Dec 14, 2023

arielj commented Dec 15, 2023 • edited Loading

arielj commented Dec 15, 2023

MattSturgeon commented Dec 15, 2023

flavioislima commented Jan 4, 2024 • edited Loading

arielj commented Jan 6, 2024

This comment has been minimized.

This comment was marked as off-topic.

Etaash-mathamsetty commented Jan 23, 2024 • edited Loading

arielj left a comment

Choose a reason for hiding this comment

imLinguin commented Jan 27, 2024

imLinguin commented Aug 30, 2023 •

edited

Loading

imLinguin commented Sep 10, 2023 •

edited

Loading

arielj commented Dec 14, 2023 •

edited

Loading

arielj commented Dec 15, 2023 •

edited

Loading

flavioislima commented Jan 4, 2024 •

edited

Loading

Etaash-mathamsetty commented Jan 23, 2024 •

edited

Loading