Support for abi3 Python packages with abi3audit checks

[jaimergp]

Proposal

Add support for abi3 Python packages to reduce the amount of artifacts produced for compatible projects (compliance checked with abi3audit).

Details

CPython has a "stable API" defined in PEP 384, which goes all the way back to Python 3.2. This allows compiled modules to run on newer Python versions than the one they were compiled against, as long as they only use the stable subset. Such packages are labeled as abi3.

conda-forge could package compatible packages like this, reducing the amount of artifacts (and build matrix entries) from V×P to P (V = Number of supported Python versions, P = Number of platforms).

In [this blog post] (recommended read!), authors reveal how this works in practice for PyPI, and how some packages opt-in to this label without actual compliance. To fix this issue, a tool abi3audit is developed, which can check for symbol usage and violations of the abi3 contract.

Implementation

If this sounds nice, we can think of how to implement it. Some ideas:

• Do it on conda-forge with some work. Hacky, but no new conda features needed:
  • Add checks or guidelines for staged-recipes to check for abi3 compatibility whenever possible.
  • Instrument conda-smithy with an abi3 mode, where Python in host is using the oldest supported Python, but run pins for >={oldest}.
  • Handle site-packages placement with symlinks, post-link scripts, patches in CPython default path.
• Add changes on conda and conda-build to support a new noarch variant (e.g. python-abi3) to handle this natively. This has the drawback of users needing to update to a newer conda to use this new package type.

Other notes

• Who wants to run (and publish?) an audit on conda-forge like the one in the blog post? 👀 Note some changes would be needed in abi3audit so it can scan non-abi3 objects.
• I am not sure about how well Windows is supported in this area.

References

• https://peps.python.org/pep-0384/
• https://blog.trailofbits.com/2022/11/15/python-wheels-abi-abi3audit/
• https://github.com/trailofbits/abi3audit

[jakirkham]

It seems like at a minimum would could add abi3 to the conda-forge Python matrix. Then add a migrator to add abi3 builds to packages and include an abi3audit run in test/commands. Admittedly this is adding one more thing to the the matrix, but it seems useful for people trying to get started with Python RCs (especially given the yearly release cycle of these)

[baszalmstra]

Another option would be to add a noarch python package together with a virtual package specifier (e.g. __linux). But this is very akward and the noarch package channel would grow with platform specific packages.

[jaimergp]

OS-specific noarch packages exist already today, so we are already in awkward territory :D

Splitting packages into different JSONs is decided only by conda-index, so we can technically instruct it to place the noarch + __linux packages in all linux-*/repodata.json files (not saying that we should, but we can) 😂
But then we would be breaking the assumption that all packages in a given JSON have the same subdir.

[jaimergp]

Some feedstocks are already looking into this. See: conda-forge/qiskit-terra-feedstock#41

[jaimergp]

And for those watching this space, check conda-forge/python-feedstock#669.

[wshanks]

It would be nice if abi3audit supported conda packages (I know there is an upstream issue linked above already). In my testing for now, I run it with explicit paths to the .so files of the installed package to check.

[jakirkham]

Just a note on Python 3.13, when using free-threading (so GIL disabled) in Python 3.13, it is not possible to use the Limited C API or Stable ABI

Of course the non-free-threading (so GIL enabled) Python 3.13, can continue to use the Limited C API and Stable ABI

More details in this doc

[jakirkham]

Thinking about the issue above, having some kind of virtual or metapackage that all abi3 packages depend on would be a good way to solve the free-threading problem

For example

package:
  name: python_abi3
  version: 1.0.0

build:
  number: 0
  skip: true  # [not is_freethreading]

requirements:
  host:
    - python
  run:
    - python

A consumer then might look like

{% set min_py_ver = "3.11" %}

package:
  name: mypkg
  version: 1.0.0

build:
  number: 0
  noarch: true  # [not is_freethreading]
  script:
    - python -m pip install .                                                       # [is_freethreading]
    - python -m pip install . --py-limited-api=cp{{ min_py_ver.replace(".", "") }}  # [not is_freethreading]

requirements:
  host:
    - python                   # [is_freethreading]
    - python {{ min_py_ver }}  # [not is_freethreading]
    - python_abi3              # [not is_freethreading]
  run:
    - python                   # [is_freethreading]
    - python {{ min_py_ver }}  # [not is_freethreading]
    - python_abi3              # [is_freethreading]

That way we still get a noarch package for the non-free-threading case. We just leverage the metapackage dependency (python_abi3 here) to ensure it can only be installed for python's where free-threading is disabled (as no python_abi3 packages are available in the free-threading case)