chore: add support for Python 3.13 #712

veenstrajelmer · 2024-10-23T11:32:25Z

Steps:

tried a lot since it was cumbersome to add python 3.13 support before merging chore: add support for numpy v2 #660 and chore: drop python 3.8 support #710
after merging these PR's, it just worked when I added python 3.13 to ci.yml
the python 3.13 wheels for windows for netcdf4 were published 1 day after the rest of the netcdf4 1.7.2 release. There was some inconvenient timing, so the lockfile had to be updated without using cache (so it would search for new netcdf4 wheels again) with poetry lock --no-cache
poetry version was updated, just because
still failed windows with py3.13 tests, but with SystemError: <method 'is_done' of '_thread._ThreadHandle' objects> returned a result with an exception set (details below). A local poetry install showed issues for numpy installation, which is resolved by specifying a different minimal numpy version for py 3.13 and higher. This also resolved this (apparently related) issue.
however, two tests are failing on tkinter errors (see below). Might be related to the fact that we use virtualenvs? win 10 Python-3.13.0 tkinter fails in venv python/cpython#125235. It seems that it can only be fixed with python 3.13.1 (expected in December 2024). >> update 23 december: all tests are green now.

SystemError details (resolved)

Windows fatal exception: access violation
Current thread 0x00000bb0 (most recent call first):
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\numpy\_core\getlimits.py", line 52 in __init__
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\numpy\_core\getlimits.py", line 232 in _register_known_types
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\numpy\__init__.py", line 295 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\meshkernel\py_structures.py", line 5 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\meshkernel\errors.py", line 3 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\meshkernel\__init__.py", line 3 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\net\models.py", line 7 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\net\__init__.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\mdu\models.py", line 24 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\mdu\__init__.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\hydrolib\core\dflowfm\__init__.py", line 9 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "D:\a\HYDROLIB-core\HYDROLIB-core\tests\dflowfm\ini\test_ini.py", line 9 in <module>
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\assertion\rewrite.py", line 184 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "C:\hostedtoolcache\windows\Python\3.13.0\x64\Lib\importlib\__init__.py", line 88 in import_module
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\pathlib.py", line 582 in import_path
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 493 in importtestmodule
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 546 in _getobj
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 284 in obj
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 562 in _register_setup_module_fixture
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\python.py", line 549 in collect
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\runner.py", line 389 in collect
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\runner.py", line 341 in from_call
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\runner.py", line 391 in pytest_make_collect_report
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_manager.py", line [12](https://github.com/Deltares/HYDROLIB-core/actions/runs/11601906361/job/32306114261#step:9:13)0 in _hookexec
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\runner.py", line 567 in collect_one_node
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 835 in _collect_one_node
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 970 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 975 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 975 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 975 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 975 in genitems
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 809 in perform_collect
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 347 in pytest_collection
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_hooks.py", line 5[13](https://github.com/Deltares/HYDROLIB-core/actions/runs/11601906361/job/32306114261#step:9:14) in __call__
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 336 in _main
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 283 in wrap_session
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\_pytest\main.py", line 330 in pytest_cmdline_main
  File "D:\a\HYDROLIB-core\HYDROLIB-core\.venv\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
Exception ignored on threading shutdown:
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.13.0\x64\Lib\threading.py", line [15](https://github.com/Deltares/HYDROLIB-core/actions/runs/11601906361/job/32306114261#step:9:16)24, in _shutdown
    if _main_thread._handle.is_done() and _is_main_interpreter():
SystemError: <method 'is_done' of '_thread._ThreadHandle' objects> returned a result with an exception set

Tkinter error details (resolved)

=========================== short test summary info ===========================
FAILED tests/dflowfm/test_net.py::test_create_1d_by_branch - _tkinter.TclError: Can't find a usable init.tcl in the following directories: 
    C:/hostedtoolcache/windows/Python/3.13.0/x64/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/.venv/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/.venv/library D:/a/HYDROLIB-core/HYDROLIB-core/library D:/a/HYDROLIB-core/HYDROLIB-core/tcl8.6.14/library D:/a/HYDROLIB-core/tcl8.6.14/library
This probably means that Tcl wasn't installed properly.
FAILED tests/dflowfm/test_net.py::test_create_1d_2d_1d2d - _tkinter.TclError: Can't find a usable init.tcl in the following directories: 
    C:/hostedtoolcache/windows/Python/3.13.0/x64/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/.venv/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/lib/tcl8.6 D:/a/HYDROLIB-core/HYDROLIB-core/.venv/library D:/a/HYDROLIB-core/HYDROLIB-core/library D:/a/HYDROLIB-core/HYDROLIB-core/tcl8.6.14/library D:/a/HYDROLIB-core/tcl8.6.14/library
This probably means that Tcl wasn't installed properly.
=========== 2 failed, 1426 passed, 10 skipped, 1 warning in 41.32s ============

sonarqubecloud · 2024-10-31T14:50:54Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

priscavdsluis

@veenstrajelmer, changes look good. Just a few additional requests:

Could you also update the sonar-project.properties?
Update the ci.yml to run the autoformatter with 3.13 instead of 3.12
Update the integration.yml to run with 3.13 instead of 3.12
Update the release.yml to run with 3.13 instead of 3.12
I think the docs.yml and docs_release.yml can be updated as well, but the docs generation seems to have been broken for a while already (> v0.5.2). So maybe don't bother with this. Created an issue for this: Doc generation seems to be broken since version 0.5.2 #725

sonarqubecloud · 2025-01-08T16:41:07Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

veenstrajelmer and others added 7 commits October 9, 2024 09:29

add python 3.13 to ci.yml

29db20e

allow lxml 5

713d71b

deleted lockfile

aaa5b0c

add minimal pandas version for py3.13

045f59c

add minimal pandas version for py3.13

5f83774

updated comments

470edc8

avoid conflicts

159f316

veenstrajelmer linked an issue Oct 23, 2024 that may be closed by this pull request

Investigate Python 3.13 support #707

Closed

8 tasks

veenstrajelmer changed the title ~~chore: add support for Python 313~~ chore: add support for Python 3.13 Oct 23, 2024

veenstrajelmer added 3 commits October 23, 2024 14:00

resolved conflicts

3190a32

updated lockfile

51c1ddf

comments

9906b84

veenstrajelmer requested a review from priscavdsluis October 23, 2024 12:07

veenstrajelmer added 4 commits October 23, 2024 14:18

lockfile

8d90ea7

updated poetry lockfile

bd148d2

updated poetry lockfile

bed1c37

updated numpy dependency and lockfile

8bb8627

veenstrajelmer mentioned this pull request Oct 31, 2024

Run Python 3.13 testbank Deltares/dfm_tools#559

Open

19 tasks

Merge branch 'main' into python-313

26be22e

priscavdsluis requested changes Jan 8, 2025

View reviewed changes

updated workflow files and sonar properties to python 3.13

69776b1

veenstrajelmer requested a review from priscavdsluis January 8, 2025 16:41

priscavdsluis approved these changes Jan 9, 2025

View reviewed changes

veenstrajelmer merged commit f4e77e8 into main Jan 9, 2025
15 checks passed

veenstrajelmer deleted the python-313 branch January 9, 2025 10:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add support for Python 3.13 #712

chore: add support for Python 3.13 #712

veenstrajelmer commented Oct 23, 2024 •

edited

Loading

sonarqubecloud bot commented Oct 31, 2024

priscavdsluis left a comment

sonarqubecloud bot commented Jan 8, 2025

chore: add support for Python 3.13 #712

chore: add support for Python 3.13 #712

Conversation

veenstrajelmer commented Oct 23, 2024 • edited Loading

sonarqubecloud bot commented Oct 31, 2024

Quality Gate passed

priscavdsluis left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 8, 2025

Quality Gate passed

veenstrajelmer commented Oct 23, 2024 •

edited

Loading