failed to freeze DPA-2 model in version 3.0.0rc0

[96XinHe]

I successfully pre-trained a DPA-2 model with out any erro information, but then I got an error whether I ran dp --pt freeze or dp freeze.

To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
[2024-11-19 16:32:31,383] DEEPMD INFO DeePMD version: 3.0.0rc0
/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Traceback (most recent call last):
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/bin/dp", line 10, in
sys.exit(main())
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/deepmd/main.py", line 927, in main
deepmd_main(args)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/deepmd/pt/entrypoints/main.py", line 547, in main
freeze(model=FLAGS.model, output=FLAGS.output, head=FLAGS.head)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/deepmd/pt/entrypoints/main.py", line 370, in freeze
model = torch.jit.script(model)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_script.py", line 1284, in script
return torch.jit._recursive.create_script_module(
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 477, in create_script_module
concrete_type = get_module_concrete_type(nn_module, share_types)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 428, in get_module_concrete_type
concrete_type = concrete_type_store.get_or_create_concrete_type(nn_module)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 369, in get_or_create_concrete_type
concrete_type_builder = infer_concrete_type_builder(nn_module)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 234, in infer_concrete_type_builder
sub_concrete_type = get_module_concrete_type(item, share_types)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 428, in get_module_concrete_type
concrete_type = concrete_type_store.get_or_create_concrete_type(nn_module)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 369, in get_or_create_concrete_type
concrete_type_builder = infer_concrete_type_builder(nn_module)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 234, in infer_concrete_type_builder
sub_concrete_type = get_module_concrete_type(item, share_types)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 428, in get_module_concrete_type
concrete_type = concrete_type_store.get_or_create_concrete_type(nn_module)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 369, in get_or_create_concrete_type
concrete_type_builder = infer_concrete_type_builder(nn_module)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 333, in infer_concrete_type_builder
attr_type, inferred = infer_type(name, value)
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/_recursive.py", line 178, in infer_type
ann_to_type = torch.jit.annotations.ann_to_type(class_annotations[name], fake_range())
File "/data/run01/scw6dyq/software/DP_series/deepmd-kit_3.0.0rc0/lib/python3.10/site-packages/torch/jit/annotations.py", line 422, in ann_to_type
raise ValueError(f"Unknown type annotation: '{ann}' at {loc.highlight()}")
ValueError: Unknown type annotation: 'list[int]' at

I don't understand how this error occurs. If I use the same input file to pre_train and then run dp freeze in version 3.0.0b4 I don't get this problem.

By the way, I would like to know that it is not possible to use dp convert-backend to convert .pth to .pb in 3.0.0b4 right?

I appreciate everyone's willingness to help answer my questions.

[njzjz]

xref: pytorch/pytorch#98521
It's fixed in PyTorch 2.1. We can drop the PyTorch 2.0 support.

[96XinHe]

Can I now upgrade pytorch on this version(3.0.0rc0)?
I tried conda install pytorch==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia
but the installation fails.

[96XinHe]

• package horovod-0.28.1-cuda_mpi_openmpi_hb6cfc05_101 has constraint pytorch >=2.0.0,<2.1.0a0 cuda*

It looks like I'm being limited by Horovod's demands.