Skip to content

Commit

Permalink
Update package versioning and readme (#9)
Browse files Browse the repository at this point in the history
  • Loading branch information
dimdi-y authored Dec 19, 2024
1 parent ebdea97 commit ab4683b
Show file tree
Hide file tree
Showing 3 changed files with 92 additions and 17 deletions.
37 changes: 20 additions & 17 deletions .circleci/scripts/binary_populate_env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,23 +50,26 @@ fi
PIP_UPLOAD_FOLDER='nightly/'
# We put this here so that OVERRIDE_PACKAGE_VERSION below can read from it
export DATE="$(date -u +%Y%m%d)"
BASE_BUILD_VERSION="$(cat ${PYTORCH_ROOT}/version.txt|cut -da -f1).dev${DATE}"

# Change BASE_BUILD_VERSION to git tag when on a git tag
# Use 'git -C' to make doubly sure we're in the correct directory for checking
# the git tag
if tagged_version >/dev/null; then
# Switch upload folder to 'test/' if we are on a tag
PIP_UPLOAD_FOLDER='test/'
# Grab git tag, remove prefixed v and remove everything after -
# Used to clean up tags that are for release candidates like v1.6.0-rc1
# Turns tag v1.6.0-rc1 -> v1.6.0
BASE_BUILD_VERSION="$(tagged_version | sed -e 's/^v//' -e 's/-.*$//')"
fi
if [[ "$(uname)" == 'Darwin' ]] || [[ "$PACKAGE_TYPE" == conda ]]; then
export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}"
else
export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}+poolside$DESIRED_CUDA"

if [[-z $PYTORCH_BUILD_VERSION ]]; then
BASE_BUILD_VERSION="$(cat ${PYTORCH_ROOT}/version.txt|cut -da -f1).dev${DATE}"

# Change BASE_BUILD_VERSION to git tag when on a git tag
# Use 'git -C' to make doubly sure we're in the correct directory for checking
# the git tag
if tagged_version >/dev/null; then
# Switch upload folder to 'test/' if we are on a tag
PIP_UPLOAD_FOLDER='test/'
# Grab git tag, remove prefixed v and remove everything after -
# Used to clean up tags that are for release candidates like v1.6.0-rc1
# Turns tag v1.6.0-rc1 -> v1.6.0
BASE_BUILD_VERSION="$(tagged_version | sed -e 's/^v//' -e 's/-.*$//')"
fi
if [[ "$(uname)" == 'Darwin' ]] || [[ "$PACKAGE_TYPE" == conda ]]; then
export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}"
else
export PYTORCH_BUILD_VERSION="${BASE_BUILD_VERSION}+$DESIRED_CUDA"
fi
fi

export PYTORCH_BUILD_NUMBER=1
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/poolside-nightly-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ on:
- true
- false
env:
# version for uploading to CodeArtifact
# current date will be added to it later as a suffix
PYTORCH_BUILD_VERSION_PREFIX: 2.6.0.dev20241210+cu126.poolside
# Needed for conda builds
ANACONDA_USER: pytorch
BINARY_ENV_FILE: /tmp/env
Expand Down Expand Up @@ -95,6 +98,7 @@ jobs:
echo "USE_SPLIT_BUILD=${{ env.use_split_build }}"
echo "MAX_JOBS=${{ env.MAX_JOBS }}"
echo "TORCH_CUDA_ARCH_LIST=${{ env.TORCH_CUDA_ARCH_LIST }}"
echo "PYTORCH_BUILD_VERSION=${{ env.PYTORCH_BUILD_VERSION_PREFIX }}.$(date -u +%Y%m%d)"
} >> "${GITHUB_ENV} }}"
- name: Checkout PyTorch
Expand Down Expand Up @@ -152,6 +156,7 @@ jobs:
-e USE_SPLIT_BUILD \
-e MAX_JOBS \
-e TORCH_CUDA_ARCH_LIST \
-e PYTORCH_BUILD_VERSION \
--tty \
--detach \
-v "${GITHUB_WORKSPACE}/pytorch:/pytorch" \
Expand Down
67 changes: 67 additions & 0 deletions poolside.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,70 @@ The changes we made should be summarized in the [poolside-changes.md](poolside-c

Feel free to contact Dmitrii Emelianenko or Vadim Markovtsev for any questions regarding this repo.

## Pulling upstream

Add the original repo as remote to fetch future changes. Make sure you also disable push on the remote (as you are not allowed to push to it anyway).

```shell
git remote add upstream https://github.com/pytorch/pytorch
git remote set-url --push upstream DISABLE
```

You can list all your remotes with `git remote -v`. You should see
```text
origin git@github.com:poolsideai/pytorch.git (fetch)
origin git@github.com:poolsideai/pytorch.git (push)
upstream https://github.com/pytorch/pytorch (fetch)
upstream DISABLE (push)
```

> When you push, do so on `origin` with `git push origin`.
When you want to pull changes from `upstream`, there are four steps:

### Select which nightly you want to rebase onto
We are making our changes over the bleeding edge nightly releases.

Visit the nightly branch in the upstream repo: https://github.com/pytorch/pytorch/commits/nightly/ .

The commit messages for nightly builds contain commit hash from the main branch.
*Important*: you need the commit hash from the commit message, not the commit hash from the nightly branch.

Let's assume you have selected a nightly with date `$NIGHTLY_DATE` and the corresponding commit hash `$COMMIT`.
The following instructions will assume that `$COMMIT` is an ancestor of our `main` in the `upstream/main` (i.e. we are rebasing onto a newer version).

### Reset the `main` branch to `$COMMIT`.

```shell
git fetch upstream main
# we should have fetched the $COMMIT from main by now
git checkout main
git reset --mixed $COMMIT
# This should work as `main` branch is an ancestor of $COMMIT originally
git push origin main
```

### Rebase `poolside-main` onto `main`

```shell
git checkout poolside-main
git rebase main
```

Now you need to fix the rebase conflicts if any.

You also need to Update the date in `PYTORCH_BUILD_VERSION_PREFIX` variable in our CI to `$DATE` [here](https://github.com/poolsideai/pytorch/blob/poolside-main/.github/workflows/poolside-nightly-build.yaml).
This is needed to represent that our fork is rebased over the specific pytorch nightly.

Now we're ready to push the updated version:

```shell
git push origin poolside-main --force-with-lease

```

# Publishing a new version to CodeArtifact:
Run the workflow defined here: https://github.com/poolsideai/pytorch/blob/poolside-main/.github/workflows/poolside-nightly-build.yaml
in the Actions tab.

Note: this only works on `poolside-main` branch (no runners will be found otherwise).

0 comments on commit ab4683b

Please sign in to comment.