Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when querying get_home() #668

Open
Batwam opened this issue Nov 1, 2024 · 3 comments
Open

Error when querying get_home() #668

Batwam opened this issue Nov 1, 2024 · 3 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@Batwam
Copy link

Batwam commented Nov 1, 2024

get_home() query no longer works
Hello, I noticed recently that the script I used to get music suggestions no longer works. The error message isn't very clear so I'm not sure what the issue might be but the script worked a few days ago and also returns results when using something like yt.search("Oasis Wonderwall")...

To Reproduce

Test Script using get_home ( doesn't work)

from ytmusicapi import YTMusic
import pprint

yt = YTMusic(oauth.json")
search_results = yt.get_home(30)
pprint.pprint(search_results)

Result

Traceback (most recent call last):
  File "/home/user/.local/share/pipx/venvs/ytmusicapi/lib/python3.12/site-packages/ytmusicapi/navigation.py", line 107, in nav
    root = root[k]
           ~~~~^^^
KeyError: 'watchEndpoint'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/bin/ytmusicapi_home.py", line 14, in <module>
    search_results = yt.get_home(30)
                     ^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/ytmusicapi/lib/python3.12/site-packages/ytmusicapi/mixins/browsing.py", line 118, in get_home
    home.extend(parse_mixed_content(results))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/ytmusicapi/lib/python3.12/site-packages/ytmusicapi/parsers/browsing.py", line 27, in parse_mixed_content
    content = parse_song(data)
              ^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/ytmusicapi/lib/python3.12/site-packages/ytmusicapi/parsers/browsing.py", line 86, in parse_song
    "videoId": nav(result, NAVIGATION_VIDEO_ID),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/ytmusicapi/lib/python3.12/site-packages/ytmusicapi/navigation.py", line 111, in nav
    raise type(e)(f"Unable to find '{k}' using path {items!r} on {root!r}, exception: {e}")
KeyError: "Unable to find 'watchEndpoint' using path ['navigationEndpoint', 'watchEndpoint', 'videoId'] on {'clickTrackingParams': 'CLECEKCzAhgAIhMIiZiNtpq6iQMVTgq3AB2ZTTys', 'browseEndpoint': {'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}, exception: 'watchEndpoint'"

Note that I just tried to generate oauth using another account and this appears to work. Also, I just modified navigation.py to print the content of the root variable and it does contain data, it even contains keys called watchEndpoint so I'm not 100% sure what the case of the issue is... could it be the some issue with the type of playlists (music Vs videos?)

Edit2: I tried using get_home(1) and it worked, then tried again and it no longer work. So perhaps an again issue with the content of the results?

Edit3: using timestamps, I was able to establish that the script does run for a few entries. However, when it lands on some it cannot find the 'watchEndpoint' key and crashing instead of skipping to the next. Is it possible to continue rather than crashing?

@Batwam
Copy link
Author

Batwam commented Nov 1, 2024

ok, I temporarily "solved it" by updating navigation.py from

    try:
        for k in items:
            root = root[k]
    except (KeyError, IndexError, TypeError) as e:
        if none_if_absent:
            return None
        raise type(e)(f"Unable to find '{k}' using path {items!r} on {root!r}, exception: {e}")
    return root

to

    try:
        for k in items:
            root = root[k]
    except (KeyError, IndexError, TypeError) as e:
        if none_if_absent:
            return None
            raise type(e)(f"Unable to find '{k}' using path {items!r} on {root!r}, exception: {e}")
    return root

but frankly, I still don't think that this shouldn't result in a crash, at best it should be a warning and skip?

@sigma67
Copy link
Owner

sigma67 commented Dec 17, 2024

Hi @Batwam

get_home is hard to debug because it is very user dependent. Could you provide reproduction instructions?

What did your homepage look like?

@sigma67 sigma67 added the help wanted Extra attention is needed label Dec 17, 2024
@sigma67 sigma67 closed this as not planned Won't fix, can't repro, duplicate, stale Dec 20, 2024
@Batwam
Copy link
Author

Batwam commented Dec 27, 2024

Hello, apologies for the delay as it was a bit tricky to troubleshoot. Responding here even though you closed it as I still believe that this is a bug. I tried to progressively increase the get_home(x) value to see at which section if would fail.

Turns out to be related to the "Listen Again" category with albums which I believe are manual uploads including tracks which I believe have been deleted... Not sure why they are still getting recommended but turns out that some (albums 2 and 3 below) are empty have a link which is 'https://music.youtube.com/channel/undefined' and lead to nowhere essentially.

Screenshot From 2024-12-27 13-38-39

log below where I'm printing root[k]. First artist works fine, second one crashes:

- text - Listen again
----------------
- musicTwoRowItemRenderer - {'thumbnailRenderer': {'musicThumbnailRenderer': {'thumbnail': {'thumbnails': [{'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w544-h544-l90-rj', 'width': 544, 'height': 544}]}, 'thumbnailCrop': 'MUSIC_THUMBNAIL_CROP_CIRCLE', 'thumbnailScale': 'MUSIC_THUMBNAIL_SCALE_ASPECT_FILL', 'trackingParams': 'CI8DEIS_AiITCPyb49Wmx4oDFbmg2AUdzocMmA=='}}, 'aspectRatio': 'MUSIC_TWO_ROW_ITEM_THUMBNAIL_ASPECT_RATIO_SQUARE', 'title': {'runs': [{'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}]}, 'subtitle': {'runs': [{'text': '11.4K subscribers'}]}, 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}, 'trackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'menu': {'menuRenderer': {'items': [{'menuNavigationItemRenderer': {'text': {'runs': [{'text': 'Shuffle play'}]}, 'icon': {'iconType': 'MUSIC_SHUFFLE'}, 'navigationEndpoint': {'clickTrackingParams': 'CI4DEJrzBRgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'watchPlaylistEndpoint': {'playlistId': 'RDAO8B2oAM_ijCcc991Z5qQCdA', 'params': 'wAEB8gECGAE%3D'}}, 'trackingParams': 'CI4DEJrzBRgAIhMI_Jvj1abHigMVuaDYBR3OhwyY'}}, {'menuNavigationItemRenderer': {'text': {'runs': [{'text': 'Start radio'}]}, 'icon': {'iconType': 'MIX'}, 'navigationEndpoint': {'clickTrackingParams': 'CI0DEJvzBRgBIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'watchPlaylistEndpoint': {'playlistId': 'RDEM8B2oAM_ijCcc991Z5qQCdA', 'params': 'wAEB'}}, 'trackingParams': 'CI0DEJvzBRgBIhMI_Jvj1abHigMVuaDYBR3OhwyY'}}, {'toggleMenuServiceItemRenderer': {'defaultText': {'runs': [{'text': 'Subscribe'}]}, 'defaultIcon': {'iconType': 'SUBSCRIBE'}, 'defaultServiceEndpoint': {'clickTrackingParams': 'CIwDEKicBhgCIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'subscribeEndpoint': {'channelIds': ['UCs61FMW9q9MIJXa154PbcYQ'], 'params': 'GAA%3D'}}, 'toggledText': {'runs': [{'text': 'Unsubscribe'}]}, 'toggledIcon': {'iconType': 'SUBSCRIBE'}, 'toggledServiceEndpoint': {'clickTrackingParams': 'CIwDEKicBhgCIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'unsubscribeEndpoint': {'channelIds': ['UCs61FMW9q9MIJXa154PbcYQ'], 'params': 'GAA%3D'}}, 'trackingParams': 'CIwDEKicBhgCIhMI_Jvj1abHigMVuaDYBR3OhwyY'}}, {'menuNavigationItemRenderer': {'text': {'runs': [{'text': 'Share'}]}, 'icon': {'iconType': 'SHARE'}, 'navigationEndpoint': {'clickTrackingParams': 'CIsDEJH7BRgDIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'shareEntityEndpoint': {'serializedShareEntity': 'GhhVQ3ZrN2w5aU9oNE5fVHdGMEtGT21iUXc%3D', 'sharePanelType': 'SHARE_PANEL_TYPE_UNIFIED_SHARE_PANEL'}}, 'trackingParams': 'CIsDEJH7BRgDIhMI_Jvj1abHigMVuaDYBR3OhwyY'}}], 'trackingParams': 'CIoDEKc7IhMI_Jvj1abHigMVuaDYBR3OhwyY', 'accessibility': {'accessibilityData': {'label': 'Action menu'}}}}}
----------------
- title - {'runs': [{'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}]}
- runs - [{'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}]
- 0 - {'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}
- navigationEndpoint - {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}
- browseEndpoint - {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}
- browseEndpointContextSupportedConfigs - {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}
- browseEndpointContextMusicConfig - {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}
- pageType - MUSIC_PAGE_TYPE_ARTIST
----------------
- subtitle - {'runs': [{'text': '11.4K subscribers'}]}
- runs - [{'text': '11.4K subscribers'}]
- 0 - {'text': '11.4K subscribers'}
- text - 11.4K subscribers
----------------
- title - {'runs': [{'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}]}
- runs - [{'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}]
- 0 - {'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}
- text - Preston Reed
----------------
- title - {'runs': [{'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}]}
- runs - [{'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}]
- 0 - {'text': 'Preston Reed', 'navigationEndpoint': {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}}
- navigationEndpoint - {'clickTrackingParams': 'CIkDEKCzAhgAIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}
- browseEndpoint - {'browseId': 'UCvk7l9iOh4N_TwF0KFOmbQw', 'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}
- browseId - UCvk7l9iOh4N_TwF0KFOmbQw
----------------
- thumbnailRenderer - {'musicThumbnailRenderer': {'thumbnail': {'thumbnails': [{'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w544-h544-l90-rj', 'width': 544, 'height': 544}]}, 'thumbnailCrop': 'MUSIC_THUMBNAIL_CROP_CIRCLE', 'thumbnailScale': 'MUSIC_THUMBNAIL_SCALE_ASPECT_FILL', 'trackingParams': 'CI8DEIS_AiITCPyb49Wmx4oDFbmg2AUdzocMmA=='}}
- musicThumbnailRenderer - {'thumbnail': {'thumbnails': [{'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w544-h544-l90-rj', 'width': 544, 'height': 544}]}, 'thumbnailCrop': 'MUSIC_THUMBNAIL_CROP_CIRCLE', 'thumbnailScale': 'MUSIC_THUMBNAIL_SCALE_ASPECT_FILL', 'trackingParams': 'CI8DEIS_AiITCPyb49Wmx4oDFbmg2AUdzocMmA=='}
- thumbnail - {'thumbnails': [{'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w544-h544-l90-rj', 'width': 544, 'height': 544}]}
- thumbnails - [{'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w226-h226-l90-rj', 'width': 226, 'height': 226}, {'url': 'https://lh3.googleusercontent.com/a-/ALV-UjUbQGzZ2_DL08yAi4pQAupcyj2upWfEkLSFajn7LbQ8dDngP9y4cQ=w544-h544-l90-rj', 'width': 544, 'height': 544}]

The part right after the one above is where is starts with what looks like partial data, then fails:

- musicTwoRowItemRenderer - {'thumbnailRenderer': {'musicThumbnailRenderer': {'thumbnail': {'thumbnails': [{'url': 'https://i9.ytimg.com/vi_locker/8Xio21X5hLQ/locker.png?sqp=-oaymwEICOIBEOIBIAA&rs=AMzJL3m7hQC8IRXFqMaDJaLjVf4Cw77_iw', 'width': 226, 'height': 226}, {'url': 'https://i9.ytimg.com/vi_locker/8Xio21X5hLQ/locker.png?sqp=-oaymwEICKAEEKAEIAA&rs=AMzJL3kZS8e-k-iYR7QVCtQnQLuiGj3Vvg', 'width': 544, 'height': 544}]}, 'thumbnailCrop': 'MUSIC_THUMBNAIL_CROP_CIRCLE', 'thumbnailScale': 'MUSIC_THUMBNAIL_SCALE_ASPECT_FILL', 'trackingParams': 'CIgDEIS_AiITCPyb49Wmx4oDFbmg2AUdzocMmA=='}}, 'aspectRatio': 'MUSIC_TWO_ROW_ITEM_THUMBNAIL_ASPECT_RATIO_SQUARE', 'title': {'runs': [{'text': 'Anthony M'}]}, 'subtitle': {}, 'navigationEndpoint': {'clickTrackingParams': 'CIcDEKCzAhgBIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}, 'trackingParams': 'CIcDEKCzAhgBIhMI_Jvj1abHigMVuaDYBR3OhwyY'}
----------------
- title - {'runs': [{'text': 'Anthony M'}]}
- runs - [{'text': 'Anthony M'}]
- 0 - {'text': 'Anthony M'}
----------------
- navigationEndpoint - {'clickTrackingParams': 'CIcDEKCzAhgBIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}
----------------
- title - {'runs': [{'text': 'Anthony M'}]}
- runs - [{'text': 'Anthony M'}]
- 0 - {'text': 'Anthony M'}
- text - Anthony M
----------------
- navigationEndpoint - {'clickTrackingParams': 'CIcDEKCzAhgBIhMI_Jvj1abHigMVuaDYBR3OhwyY', 'browseEndpoint': {'browseEndpointContextSupportedConfigs': {'browseEndpointContextMusicConfig': {'pageType': 'MUSIC_PAGE_TYPE_ARTIST'}}}}

as you can see, it's missing a bunch of fields and eventually crashes, maybe because it's missing ['browseEndpoint']['browseId'] within navigationEndpoint in particular?

@sigma67 sigma67 reopened this Dec 27, 2024
@sigma67 sigma67 added the bug Something isn't working label Dec 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants