MPI-Operator run example failed

[q443048756]

I setup the mpi-operator v0.4.0

and try to deploy the example:
mpi-operator-0.4.0/examples/v2beta1/tensorflow-benchmarks/tensorflow-benchmarks.yaml

My k8s has three node, each node has a 3060 graphics card

but it seem can not run it correctly:
1、Using the default configuration, I don't see any pods starting, it should be a failure

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: tensorflow-benchmarks
spec:
slotsPerWorker: 1
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
command:
- mpirun
- --allow-run-as-root
- -np
- "2"
- -bind-to
- none
- -map-by
- slot
- -x
- NCCL_DEBUG=INFO
- -x
- LD_LIBRARY_PATH
- -x
- PATH
- -mca
- pml
- ob1
- -mca
- btl
- ^openib
- python
- scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
- --model=resnet101
- --batch_size=64
- --variable_update=horovod
Worker:
replicas: 2
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
resources:
limits:
nvidia.com/gpu: 1
2、When replicas: 1, the pod starts normally，I suspect that the task cannot call the GPU across nodes.

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: tensorflow-benchmarks
spec:
slotsPerWorker: 1
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
command:
- mpirun
- --allow-run-as-root
- -np
- "1"
- -bind-to
- none
- -map-by
- slot
- -x
- NCCL_DEBUG=INFO
- -x
- LD_LIBRARY_PATH
- -x
- PATH
- -mca
- pml
- ob1
- -mca
- btl
- ^openib
- python
- scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
- --model=resnet101
- --batch_size=64
- --variable_update=horovod
Worker:
replicas: 1
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
resources:
limits:
nvidia.com/gpu: 1

3、After the pod is started, the launcher reports an error
2023-10-25 09:53:08.464568: E tensorflow/c/c_api.cc:2184] Internal: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid
Traceback (most recent call last):
File "scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py", line 73, in
app.run(main) # Raises error on invalid flags, unlike tf.app.run()
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py", line 61, in main
params = benchmark_cnn.setup(params)
File "/tensorflow/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 3538, in setup
with tf.Session(config=create_config_proto(params)) as sess:
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1586, in init
super(Session, self).init(target, graph, config=config)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 701, in init
self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

---

mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[12892,1],0]
Exit code: 1

[tenzen-y]

Does the example without GPUs work fine?

https://github.com/kubeflow/mpi-operator/blob/master/examples/v2beta1/pi/pi.yaml

[alculquicondor]

Alternatively, did you install the nvidia drivers on the nodes?

[q443048756]

没有 GPU 的示例可以正常工作吗？

https://github.com/kubeflow/mpi-operator/blob/master/examples/v2beta1/pi/pi.yaml
At the beginning, I have used other methods to test the GPU availability. The first point I mentioned is that the task cannot use multi-node GPU to work. When replicas: 2, the pod cannot be started. When replicas: 1, the pod can work normally. start up

https://github.com/kubeflow/mpi-operator/blob/master/examples/v2beta1/pi/pi.yaml:This example is successful

[q443048756]

或者，您是否在节点上安装了 nvidia 驱动程序？

After installing it and I have tested it, the GPU is working normally.he first point I mentioned is that the task cannot use multi-node GPU to work. When replicas: 2, the pod cannot be started. When replicas: 1, the pod can work normally. start up

[alculquicondor]

Uhm... interesting. Although it sounds like some networking problems that you need to work out with your provider. I don't think it's related to mpi-operator.

[kuizhiqing]

@q443048756 It would be the cuda thing in the default image mpioperator/tensorflow-benchmarks:latest DO NOT compatible with your local environment, I suggest you to find the right base image from here https://hub.docker.com/r/nvidia/cuda and then build your own one.

[tenzen-y]

@q443048756 It would be the cuda thing in the default image mpioperator/tensorflow-benchmarks:latest DO NOT compatible with your local environment, I suggest you to find the right base image from here https://hub.docker.com/r/nvidia/cuda and then build your own one.

It sounds reasonable. Thank you for the helping.

[wang-mask]

It is the same failure, maybe it is time to update the example.