CUDA12.6, gcc9.4, but "There was an error before creating cudnn handle (500): cudaErrorSymbolNotFound : named symbol not found"

[Huilin-Li]

Caution: Please only report your issue related to the installation on your local PC or macOS. If you can get the help message by colabfold_batch --help or run a test prediction successfully, your installation is successful. Requests or questions regarding ColabFold features should be directed to ColabFold repo's issues.

---

What is your installation issue?
I firstly executed: colabfold_batch myfa0.fa myfa0_out --msa-only works. However, then

(/storage/shenhuaizhongLab/lihuilin/mycolabfold/localcolabfold/colabfold-conda) [lihuilin@ga40q08 apgfastas]$ colabfold_batch myfa0.fa myfa0_out
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1730616792.448210 2934203 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1730616792.452606 2934203 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-03 14:53:24,193 Running colabfold 1.5.5 (c21e1768d18e3608e6e6d99c97134317e7e41c75)

WARNING: You are welcome to use the default MSA server, however keep in mind that it's a
limited shared resource only capable of processing a few thousand MSAs per day. Please
submit jobs only from a single IP address. We reserve the right to limit access to the
server case-by-case when usage exceeds fair use. If you require more MSAs: You can
precompute all MSAs with `colabfold_search` or host your own API and pass it to `--host-url`

2024-11-03 14:53:24,471 Running on GPU
2024-11-03 14:53:26,236 Found 5 citations for tools or databases
2024-11-03 14:53:26,236 Query 1/30: aaalA (length 309)
2024-11-03 14:53:26,262 Loaded myfa0_out/aaalA.pickle
E1103 14:53:29.987652 2934203 cuda_dnn.cc:502] There was an error before creating cudnn handle (500): cudaErrorSymbolNotFound : named symbol not found
E1103 14:53:29.988284 2934203 cuda_dnn.cc:502] There was an error before creating cudnn handle (500): cudaErrorSymbolNotFound : named symbol not found
2024-11-03 14:53:30,205 Could not predict aaalA. Not Enough GPU memory? FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
2024-11-03 14:53:30,206 Query 2/30: aaavA (length 418)
2024-11-03 14:53:30,642 Loaded myfa0_out/aaavA.pickle
^Z
[2]+  Stopped                 colabfold_batch myfa0.fa myfa0_out

Computational environment

• OS: [e.g. Ubuntu 22.04, Windows10 & WSL2, macOS...]
• CUDA version if Linux (Show the output of /usr/local/cuda/bin/nvcc --version.)

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

A clear and concise description of what you expected to happen.

[punit-jha123]

I am getting the same error! did you manage to fix this?

I have nvcc -V as 12.1 and gcc 10.2.1-6 Debian

[Huilin-Li]

I am getting the same error! did you manage to fix this?

I have nvcc -V as 12.1 and gcc 10.2.1-6 Debian

I remember the problem in my case is I didn't activate CUDA environment in my HPC environment when I installed the localcolabfold.