Segmentation fault with parallel HDF5

[jklymak]

Hi,

Thanks for the fun looking model. I'm particularly excited to get running on GPU, if that turns out to be possible.

I have veros running fine in single-cpu mode, both on a login node and on my compute nodes. I've been testing on the 4-deg globe run..

As soon as I try and run in parallel, using

srun -n 2 -- veros run global_4deg.py -v trace --force-overwrite -b numpy -n 2 1

it fails trying to create the 4deg.averages.nc file, and throws a segmentation fault. A few logger.trace statements places it as failing at initialize_variable(state, dim, var, ncfile) in netcdf.py. The odd thing is that it appears to work for the first couple of dim/var pairs (xt, xu), but fails for yt.

I'm assuming this is a problem somewhere in my h5py netcdf-mpi setup?

My module list looks like

  1) CCconfig          6) python/3.8.10   (t)     11) flexiblas/3.0.4       16) hdf5-mpi/1.10.6   (io)
  2) gentoo/2020 (S)   7) ipykernel/2021a         12) blis/0.8.1            17) mpi4py/3.1.2      (t)
  3) StdEnv/2020 (S)   8) imkl/2020.1.217 (math)  13) ucx/1.8.0             18) netcdf-mpi/4.7.4  (io)
  4) mii/1.1.1         9) gcccore/.9.3.0  (H)     14) libfabric/1.10.1      19) scipy-stack/2021a (math)
  5) libffi/3.3       10) gcc/9.3.0       (t)     15) openmpi/4.0.3    (m)

and pip list:

...
h5netcdf                           0.11.0                              
h5py                               3.1.0   
...
mpi4py                             3.1.2 
...
netCDF4                            1.5.6+computecanada 
...
veros                              1.4.2+21.gd6b51dc                   /lustre06/project/6001902/jklymak/veros

Of course I can supply the rest of the environment.

Anyway, any suggested things I should be looking for would be much appreciated. Also if there is a better mailing list for this, or forum... Cheers, Jody

[jklymak]

As a sanity check I ran https://github.com/Unidata/netcdf4-python/blob/master/examples/mpi_example.py and this works fine in parallel mode (mpiexec -np 4 python testMPI.py). Is there a higher level sanity check I should be running to be equivalent of what veros is doing? Thanks....

[jklymak]

Also, https://gist.github.com/apdavison/36126ee26067592ee69bf51b57fd3f31, which tests h5py, works fine in parallel.

[dionhaefner]

Hey Jody, happy to hear that you are playing with Veros.

This is absolutely the right place to ask for help. Stuff like this tends to be pretty hard to debug, unfortunately. Some thoughts:

• You don't need to worry about the netCDF4 library since we're not using it (all netCDF files are written directly through HDF5 / h5py).

• It sounds like you did this correctly since your sanity check works, but just to be sure: you should install h5py via something like

   $ CC=mpicc HDF5_MPI="ON" pip install h5py --no-binary h5py

  to make sure it is linked to the correct HDF5 and MPI libraries.

• This is just a hunch, but it sure seems weird that xt and xu work while yt fails. You are running on 2 processes where only the x direction is chunked, so yt is actually the same data on both processes. Maybe this is the root of the problem? Could you try running with -n 1 2 and -n 2 2 and see what happens?

[jklymak]

That was perfect. Of course it crashed, which means it is a good test of the hdf5 setup. I must have somehow crossed my installs, largely because I probably do not understand python envs very well (I usually use conda). To get it to work I did:

module load StdEnv  gcc  openmpi mpi4py netcdf-mpi scipy-stack hdf5-mpi
pip install click entrypoints requests pillow loguru tqdm h5netcdf
pip install -e .

and that cleaned things up so that the test script

mpiexec -np 4 python testh5netcdf.py

and veros itself:

mpiexec -np 4 -- veros run global_4deg.py -v trace --force-overwrite -b numpy -n 2 2

worked fine.

(OK, now onto getting jax to work ;-)

I would be a little abashed, but I think that test script is going to be really useful for many folks. Thanks for putting it together!

[jklymak]

jax works great, except I get cuda not available errors.

On my machine if I do module load cuda, it unloads hdf5-mpi and netcdf-mpi however I assume I need those, so I need to ask my sysadmins if there is a way to load all of them...

[dionhaefner]

Happy to hear that you got it to work.

For the record, I usually use a conda environment on our HPC cluster and just pip install everything I need into it. So something like

$ conda create -n "veros" "python<3.9"
$ conda activate veros
$ module load openmpi_cuda petsc hdf5-parallel  # only load system libraries
$ pip install mpi4py
$ CC=mpicc HDF5_MPI="ON" pip install h5py --no-binary h5py
$ pip install petsc4py==<petsc_version>

I like this because Python packages become outdated so fast, and this way you have full control over everything and can upgrade on demand. But it's probably easier to let your administration handle this.

Regarding JAX, I assume you installed the GPU version?

If you just want to run on a single GPU for now you can also use sequential HDF5. You only need MPI (and mpi4jax) for multi-GPU contexts.

[dionhaefner]

Perhaps we should have a script somewhere that installs a full HPC environment with mpi4py, petsc4py, h5py, mpi4jax, and JAX... even if it doesn't work out of the box for everyone, it would at least be a good template to adapt.

[jklymak]

Yeah, we are strongly discouraged from using conda on the computecanada nodes. I guess they don't want 1000 copies of big libraries clogging up space, and I think they do try to optimize the shared libraries for the cluster. So I was trying to do this by the book. But if that doesn't work I may try to use conda. I'll see what the system admins say.... Thanks again, I think I have enough to play with for now.