Skip to content

Commit

Permalink
[SPARK-50178][INFRA] Use PyArrow>=18.0.0 for Python 3.13
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to use `PyArrow>=18.0.0` for Python 3.13 to ensure the test coverage of `PyArrow 18 on Python 3.13`.

### Why are the changes needed?

PyArrow 18.0.0 is the first version which supports Python 3.13 officially.
- https://pypi.org/project/pyarrow/18.0.0/

### Does this PR introduce _any_ user-facing change?

No, this is an infra only change.

### How was this patch tested?

Pass the CI image building. The installed version will be used in Python 3.13 Daily CI.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #48709 from dongjoon-hyun/SPARK-50178.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Oct 31, 2024
1 parent 2ace2eb commit 6ba13ee
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion dev/infra/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ RUN apt-get update && apt-get install -y \
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.13
# TODO(SPARK-49862) Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python 3.13 image when it supports Python 3.13
RUN python3.13 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs this
RUN python3.13 -m pip install numpy six==1.16.0 pandas==2.2.3 scipy coverage matplotlib openpyxl grpcio==1.67.0 grpcio-status==1.67.0 lxml numpy>=2.1 && \
RUN python3.13 -m pip install numpy>=2.1 pyarrow>=18.0.0 six==1.16.0 pandas==2.2.3 scipy coverage matplotlib openpyxl grpcio==1.67.0 grpcio-status==1.67.0 lxml && \
python3.13 -m pip cache purge

# Remove unused installation packages to free up disk space
Expand Down

0 comments on commit 6ba13ee

Please sign in to comment.