Skip to content

Commit

Permalink
Simplify Python modules installation (#5343)
Browse files Browse the repository at this point in the history 
* Simplify Python modules installation

* Remove miniconda environment setup for MacOS -- tensorflow-metal can be
  installed with plain pip now.
* Deduplicate requirements specifications to reduce need for copy-pasting.
* Stop installing modules inside a fresh virtualenv, since those are not
  relocatable. Use PYTHONPATH instead.
* Remove deactivation of outer virtualenv, since that will mess up the PATH.

* Fix shebang patcher

Since we're not using a virtualenv any more, shebangs for Python scripts most
likely refer to `$PYTHON_ROOT`, not `$PYTHON_MODULES_ROOT`.

* Fix binary path to match standard

Move binaries from `$INSTALLROOT/share/python-modules/bin` to
`$INSTALLROOT/bin`. This matches other packages, and lets us use the default
PATH entry added by alibuild. No libraries are installed in `.../lib` either
way, so we don't care about `LD_LIBRARY_PATH`.

* Fix Python libraries path to match standard

Install Python modules to a semi-standard path, to match other Python packages
in alidist (e.g. xjalienfs).

Updatable RPMs are no longer a concern, since we install these under
`/opt/alisw/<arch>/<package>` nowadays -- so there's no risk of conflict among
packages and with the base system.

* Fix cython binary path

* Update outdated Python packages on Python 3.11+

This should fix an ImportError when trying to use ipython on newer Python
versions.

* Reinstate virtualenv

* Merge dask packages from #5303

* Add comment about MacOS-shipped Python

This was encountered on one of the CI Macs and fixed by running "brew install
python". Homebrew doesn't link python@3.12 as bin/python3, only python@3.11.

* Fix tensorflow-metal version on Python 3.11
  • Loading branch information
@TimoWilken
TimoWilken authored Feb 9, 2024
1 parent 474abac commit 7ab6032
Show file tree
Hide file tree
Showing 6 changed files with 138 additions and 235 deletions.
5 changes: 4 additions & 1 deletion defaults-o2-dataflow.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,11 @@ overrides:
tag: "v12.2.0-alice1"
Python-modules-list:
env:
PIP39_REQUIREMENTS: |
PIP_BASE_REQUIREMENTS: |
pip==21.3.1
setuptools==65.5.1
wheel==0.37.1
PIP_REQUIREMENTS: ""
O2-customization:
env:
ENABLE_UPGRADES: "OFF" # Disable detector upgrades in O2
Expand Down
207 changes: 87 additions & 120 deletions python-modules-list.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,126 +6,93 @@ env:
setuptools==59.6.0
wheel==0.37.1
PIP_REQUIREMENTS: |
requests==2.27.1
ipykernel==5.1.0
ipython==7.4.0
ipywidgets==7.4.2
metakernel==0.20.14
mock==2.0.0
notebook==5.7.8
scons==4.1.0
PIP36_REQUIREMENTS: |
PyYAML==6.0.1
psutil==5.8.0
uproot==4.1.0
numpy==1.16.2
scipy==1.2.1
Cython==0.29.16
seaborn==0.9.0
sklearn-evaluation==0.4
Keras==2.2.4
tensorflow==1.13.1
xgboost==0.82
dryable==1.0.3
responses==0.10.6
pandas==0.24.2
scikit-learn==0.20.3
PIP38_REQUIREMENTS: |
PyYAML==6.0.1
psutil==5.8.0
uproot==4.1.0
scipy==1.6.1
Cython==0.29.21
seaborn==0.9.0
sklearn-evaluation==0.4
Keras==2.4.3
tensorflow==2.4.1
xgboost==1.3.3
numpy==1.19.5
dryable==1.0.3
responses==0.10.6
pandas==1.2.3
scikit-learn==0.24.1
PIP39_REQUIREMENTS: |
PyYAML==6.0.1
psutil==5.8.0
uproot==4.1.0
numpy==1.21.4
scipy==1.7.3
Cython==0.29.21
seaborn==0.11.0
scikit-learn==0.24.1
sklearn-evaluation==0.5.2
Keras==2.4.3
xgboost==1.2.0
dryable==1.0.5
responses==0.10.6
pandas==1.1.5
"PIP39_REQUIREMENTS_ubuntu2110_x86_64": |
PyYAML==6.0.1
psutil==5.8.0
uproot==4.1.0
numpy==1.21.4
scipy==1.7.3
Cython==0.29.21
seaborn==0.11.0
scikit-learn==0.24.1
sklearn-evaluation==0.5.2
Keras==2.4.3
xgboost==1.2.0
dryable==1.0.5
responses==0.10.6
pandas==1.1.5
# Keep the PIPxy version in sync with the Conda env we install in Python-modules!
# Everything but the first two lines copied from PIP39_REQUIREMENTS, but with versions
# adjusted such that wheels are available and for compatibility with tensorflow.
PIP39_REQUIREMENTS_osx_arm64: |
tensorflow-macos==2.13.0
tensorflow-metal==1.0.1
PyYAML==6.0.1
psutil==5.9.5
uproot==4.1.0
numpy==1.23.5
scipy==1.10.1
Cython==0.29.21
seaborn==0.11.0
scikit-learn==1.2.2
sklearn-evaluation==0.12.0
Keras==2.13.1
xgboost==1.7.5
dryable==1.0.5
responses==0.10.6
pandas==1.5.3
PIP310_REQUIREMENTS: |
PyYAML==6.0.1
psutil==5.9.0
uproot==4.1.0
numpy==1.23.4
scipy==1.9.3
Cython==0.29.21
seaborn==0.11.0
scikit-learn==0.24.1
sklearn-evaluation==0.8.1
Keras==2.4.3
xgboost==1.2.0
dryable==1.0.5
responses==0.10.6
pandas==1.1.5
PIP311_REQUIREMENTS: |
PyYAML==6.0.1
psutil==5.9.5
uproot==4.1.0
numpy==1.23.5
scipy==1.10.1
Cython==0.29.21
seaborn==0.11.0
scikit-learn==1.3.0
sklearn-evaluation==0.12.0
Keras==2.13.1
xgboost==1.7.5
dryable==1.0.5
responses==0.10.6
pandas==1.5.3
# This is a pip requirements file. For documentation see:
# https://pip.pypa.io/en/stable/reference/requirements-file-format/
# For valid environment markers (after the ';'), see:
# https://peps.python.org/pep-0508/#environment-markers

ipykernel == 5.1.0; python_version < '3.11'
ipykernel == 6.29.1; python_version >= '3.11'
ipython == 7.4.0; python_version < '3.11'
ipython == 8.21.0; python_version >= '3.11'
ipywidgets == 7.4.2; python_version < '3.11'
ipywidgets == 8.1.1; python_version >= '3.11'
metakernel == 0.20.14; python_version < '3.11'
metakernel == 0.30.1; python_version >= '3.11'
notebook == 5.7.8; python_version < '3.11'
notebook == 7.0.7; python_version >= '3.11'
scons == 4.1.0; python_version < '3.11'
scons == 4.6.0.post1; python_version >= '3.11'

requests == 2.27.1; python_version < '3.11'
requests == 2.31.0; python_version >= '3.11'
PyYAML == 6.0.1
uproot == 4.1.0

# Mock is included in the Python standard library as unittest.mock from
# 3.3 onwards.
mock == 2.0.0; python_version < '3.3'
responses == 0.10.6

psutil == 5.8.0; python_version < '3.10'
psutil == 5.9.0; python_version == '3.10'
psutil == 5.9.5; python_version >= '3.11'

numpy == 1.16.2; python_version < '3.8'
numpy == 1.19.5; python_version == '3.8'
numpy == 1.21.4; python_version == '3.9'
numpy == 1.23.4; python_version == '3.10'
numpy == 1.23.5; python_version >= '3.11'

scipy == 1.2.1; python_version < '3.8'
scipy == 1.6.1; python_version == '3.8'
scipy == 1.7.3; python_version == '3.9'
scipy == 1.9.3; python_version == '3.10'
scipy == 1.10.1; python_version >= '3.11'

Cython == 0.29.16; python_version < '3.8'
Cython == 0.29.21; python_version >= '3.8'

seaborn == 0.9.0; python_version < '3.9'
seaborn == 0.11.0; python_version >= '3.9'

scikit-learn == 0.20.3; python_version < '3.8'
scikit-learn == 0.24.1; python_version >= '3.8' and python_version < '3.11'
scikit-learn == 1.3.0; python_version >= '3.11'

sklearn-evaluation == 0.4; python_version < '3.9'
sklearn-evaluation == 0.5.2; python_version == '3.9'
sklearn-evaluation == 0.8.1; python_version == '3.10'
sklearn-evaluation == 0.12.0; python_version >= '3.11'

Keras == 2.2.4; python_version < '3.8'
Keras == 2.4.3; python_version >= '3.8' and python_version < '3.11'
Keras == 2.13.1; python_version >= '3.11'

tensorflow == 1.13.1; python_version < '3.8'
tensorflow == 2.4.1; python_version == '3.8'
tensorflow == 2.13.1; python_version >= '3.11'

# See version compatibility table at https://pypi.org/project/tensorflow-metal/
tensorflow-metal == 1.0.0; sys_platform == 'darwin' and python_version >= '3.11'

xgboost == 0.82; python_version < '3.8'
xgboost == 1.3.3; python_version == '3.8'
xgboost == 1.2.0; python_version >= '3.9' and python_version < '3.11'
xgboost == 1.7.5; python_version >= '3.11'

dryable == 1.0.3; python_version < '3.9'
dryable == 1.0.5; python_version >= '3.9'

pandas == 0.24.2; python_version < '3.8'
pandas == 1.2.3; python_version == '3.8'
pandas == 1.1.5; python_version >= '3.9' and python_version < '3.11'
pandas == 1.5.3; python_version >= '3.11'

dask[array,dataframe,distributed] == 2023.2.0; python_version < '3.11'
dask[array,dataframe,distributed] == 2023.12.1; python_version >= '3.11'
dask_jobqueue == 0.8.2

build_requires:
- alibuild-recipe-tools
---
Expand Down
129 changes: 38 additions & 91 deletions python-modules.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,109 +6,56 @@ requires:
- "FreeType:(?!osx)"
- libpng
build_requires:
- curl
- Python-modules-list
- alibuild-recipe-tools
prepend_path:
PATH: "$PYTHON_MODULES_ROOT/share/python-modules/bin"
LD_LIBRARY_PATH: "$PYTHON_MODULES_ROOT/share/python-modules/lib"
# If we need tensorflow to work on Mac, we must use lib/python$pyver, not lib/python here.
PYTHONPATH: $PYTHON_MODULES_ROOT/share/python-modules/lib/python/site-packages
# If we need tensorflow-metal to work on Mac during subsequent builds, we
# must use lib/python$pyver, not lib/python here.
PYTHONPATH: "$PYTHON_MODULES_ROOT/lib/python/site-packages"
---
#!/bin/bash -e
if [ -n "$VIRTUAL_ENV" ]; then
# Once more to get the deactivate
. "$VIRTUAL_ENV/bin/activate"
deactivate
fi
# A spurious PYTHONPATH can affect later commands
unset PYTHONPATH

# We use a different INSTALLROOT, so that we can build updatable RPMS which
# do not conflict with the underlying Python installation.
PYTHON_MODULES_INSTALLROOT=$INSTALLROOT/share/python-modules

case $ARCHITECTURE in
osx_arm64)
# On ARM Macs, we need to install Conda to get Tensorflow with hardware support.
# Available version list: https://repo.anaconda.com/miniconda/
# The Python version of this Conda env matters! Keep it in sync
# with the "PIPXY_REQUIREMENTS_osx_arm64" clause in Python-modules-list.
curl -fsSLo miniconda.sh 'https://repo.anaconda.com/miniconda/Miniconda3-py39_23.3.1-0-MacOSX-arm64.sh'
bash miniconda.sh -b -p "$PYTHON_MODULES_INSTALLROOT"
. "$PYTHON_MODULES_INSTALLROOT/bin/activate"
conda install -y -c apple tensorflow-deps ;;
*)
# On other platforms, just create a plain virtualenv.
python3 -m venv "$PYTHON_MODULES_INSTALLROOT"
. "$PYTHON_MODULES_INSTALLROOT/bin/activate" ;;
esac

# Major.minor version of Python
pyver="$(python3 -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_version())')"

# These are the basic requirements needed for all installation and platform
# and it should represent the common denominator (working) for all packages/platforms
echo "$PIP_BASE_REQUIREMENTS" | tr '[:space:]' '\n' > base-requirements.txt

# PIP*_REQUIREMENTS variables come from python-modules-list.sh.
case $ARCHITECTURE in
slc6_*) echo "$PIP_REQUIREMENTS" ;;
*)
echo "$PIP_REQUIREMENTS"
# Handle special lists for different platforms, e.g. $PIP39_REQUIREMENTS_osx_arm64.
this_pyver_requirements_var=PIP${pyver/.}_REQUIREMENTS
this_pyver_arch_requirements_var=PIP${pyver/.}_REQUIREMENTS_${ARCHITECTURE//-/_}
# Use $PIPxy_REQUIREMENTS_arch if set, falling back to $PIPxy_REQUIREMENTS.
echo "${!this_pyver_arch_requirements_var:-${!this_pyver_requirements_var}}" ;;
esac | tr -s '[:space:]' '\n' > requirements.txt
unset VIRTUAL_ENV

# Users might want to install more packages in the same environment. A venv
# provides a pip binary that will install packages into the same path.
# This copies the system python binary (or the one from PYTHON_ROOT) into the
# venv. We must not use symlinks, since those break if the package is uploaded
# to a remote store.
# NOTE: If you get an error saying "Error: This build of python cannot create
# venvs without using symlinks", then you are using the MacOS Python. You
# should be using the Homebrew Python instead, so run "brew install python".
python3 -m venv --copies "$INSTALLROOT"
. "$INSTALLROOT/bin/activate"
# From now on, we use the python3 binary copied into the venv. This makes pip
# install packages into the venv.

# Major.minor version of Python, needed for PYTHONPATH.
pyver=$(python3 -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_version())')

# Install pinned basic requirements for python infrastructure
echo "$PIP_BASE_REQUIREMENTS" > base-requirements.txt
python3 -m pip install -IU -r base-requirements.txt

# FIXME: required because of the newly introduced dependency on scikit-garden requires
# a numpy to be installed separately
# See also:
# https://github.com/scikit-garden/scikit-garden/issues/23
python3 -m pip install -IU numpy
# The above updates pip and setuptools, so install the rest of the packages separately.
echo "$PIP_REQUIREMENTS" > requirements.txt
python3 -m pip install -IU -r requirements.txt

# Find the proper Python lib library and export it
pushd "$PYTHON_MODULES_INSTALLROOT"
# let's remove any pre-existent symlinks to have a clean slate
[ -h lib64 ] && unlink lib64
[ -h lib ] && unlink lib
if [[ -d lib64 ]]; then
ln -nfs lib64 lib # creates lib pointing to lib64
elif [[ -d lib ]]; then
ln -nfs lib lib64 # creates lib64 pointing to lib
fi
ln -nfs "python$pyver" lib/python
popd

# Remove useless stuff
rm -rvf "$PYTHON_MODULES_INSTALLROOT"/share "$PYTHON_MODULES_INSTALLROOT"/lib/python*/test
find "$PYTHON_MODULES_INSTALLROOT"/lib/python* \
-mindepth 2 -maxdepth 2 -type d -and \( -name test -or -name tests \) \
-exec rm -rvf '{}' \;
rm -rvf "$INSTALLROOT/share"
find "$INSTALLROOT" -mindepth 2 -maxdepth 2 \
-type d -and \( -name test -or -name tests \) -exec rm -rvf '{}' \;

# Fix shebangs: remove hardcoded Python path. Scripts' shebangs will point at
# the venv's python using an absolute path by default, which we must change.
sed -r -i.deleteme -e "1s,^#!$INSTALLROOT/bin/,#!/usr/bin/env ," "$INSTALLROOT"/bin/*
rm -f "$INSTALLROOT"/bin/*.deleteme

case $ARCHITECTURE in
osx_arm64) ;;
*)
# Fix shebangs: remove hardcoded Python path
find "$PYTHON_MODULES_INSTALLROOT/bin" -type f -exec sed -i.deleteme -e "s|${PYTHON_MODULES_INSTALLROOT}|/usr|;s|python3|env python3|" '{}' \;
find "$PYTHON_MODULES_INSTALLROOT/bin" -name '*.deleteme' -delete ;;
esac
# Link python -> python$pyver, so we can refer to it in PYTHONPATH without knowing pyver.
ln -nsf "python$pyver" "$INSTALLROOT/lib/python"

# Modulefile
MODULEDIR="$INSTALLROOT/etc/modulefiles"
mkdir -p "$MODULEDIR"
alibuild-generate-module > "$MODULEDIR/$PKGNAME"
cat >> "$MODULEDIR/$PKGNAME" <<EoF
# Our environment
set PYTHON_MODULES_ROOT \$::env(BASEDIR)/$PKGNAME/\$version
prepend-path PATH \$PYTHON_MODULES_ROOT/share/python-modules/bin
prepend-path LD_LIBRARY_PATH \$PYTHON_MODULES_ROOT/share/python-modules/lib
mkdir -p "$INSTALLROOT/etc/modulefiles"
alibuild-generate-module --bin > "$INSTALLROOT/etc/modulefiles/$PKGNAME"
cat >> "$INSTALLROOT/etc/modulefiles/$PKGNAME" <<EOF
# We need to use lib/python$pyver, not lib/python here so that tensorflow-metal works on Mac.
prepend-path PYTHONPATH \$PYTHON_MODULES_ROOT/share/python-modules/lib/python$pyver/site-packages
EoF
prepend-path PYTHONPATH \$PKG_ROOT/lib/python$pyver/site-packages
EOF
2 changes: 1 addition & 1 deletion rivet.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ case $ARCHITECTURE in
--with-hepmc3="$HEPMC3_ROOT" \
--with-fastjet="$FASTJET_ROOT" \
LDFLAGS="${LOCAL_LDFLAGS}" \
CYTHON="$PYTHON_MODULES_ROOT/share/python-modules/bin/cython"
CYTHON="$PYTHON_MODULES_ROOT/bin/cython"
;;
esac

Expand Down
Loading

0 comments on commit 7ab6032

Please sign in to comment.