Questions about DPA2 zero shot test #3606
-
|
When using the data and pre-trained Model.pt file publicly available in Zenodo (https://zenodo.org/records/10483908) for downstream testing tasks, there is a significant numerical deviation in the Energy RMSE values across multiple datasets, while the Force prediction values are largely consistent with those in the paper. Taking the H2O-PBE0TS-MD task as an example, my code is as follows: Below is the result log I obtained: 2024-03-26 11:08:12,891 [inference.py:223] INFO # ----------weighted average of errors----------- Why is this happening? If you need a more detailed log file, you can refer to the attached document. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 8 replies
-
|
When conducting a zero-shot test, it is important to adjust the energy bias due to the differences in DFT settings between the pretraining and downstream datasets. Failing to do so will result in a significant energy shift. To automate this process, you can set This will generate a 1-step fine-tuned model ( Hope this will help you. @duduRUHappy |
Beta Was this translation helpful? Give feedback.
-
|
According to your advice, I implemented a 1-step fine-tune to obtain the model-1.pt, and then used it for zero-shot test. The rmse_e (0.1796) is somewhat better than before (150.5802), but there is still a significant gap compared to the results (rmse_e = 0.0006) in the paper. Besides that, the rmse_f (0.6234) is much worse than before (0.0444). By the way, the standard deviations of energy and force labels in the test set are 0.0033 and 0.8161. And the json file used for finetune and the finetune result log file are attached. 2024-03-27 16:20:27,811 [inference.py:211] INFO # ----------weighted average of errors----------- |
Beta Was this translation helpful? Give feedback.
-
|
Yes, for the default initial learning rate 0.0002, even 1 step is too much to retain the accuracy. In practice, we use a sufficiently small learning rate (e.g. start_lr=stop_lr=1e-10) to finetune 1 step. Perhaps we should support 0 step to avoid this @iProzd. Currently 0 step will raise an error that numb_steps must be larger than warmup_steps. |
Beta Was this translation helpful? Give feedback.
-
|
Does testing dpa2 model require deepmd v3 or we can do it using previous deepmd version? Please let me know. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Can you please share an example of how you proceeded with testing the model with this source code? |
Beta Was this translation helpful? Give feedback.
-
|
Maybe you can get some help from this notebook https://bohrium.dp.tech/notebooks/49345212168 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I am looking into it. It shows me that model extension can be either .pb or .pth and I see your model's extension is .pt -m MODEL, --model MODEL |
Beta Was this translation helpful? Give feedback.
-
|
Also, is it possible to load your own test dataset for testing using the dpa2 checkpoint on this notebook? |
Beta Was this translation helpful? Give feedback.
-
|
If you wanna build the source code (https://bohrium.dp.tech/notebooks/49345212168), you can refer to the readme file for specific steps and requirements. Or you can just try the pre-release 3.0 version（https://github.com/deepmodeling/deepmd-kit/releases）. By the way, I'm also the new user of this platform, if you have more question, plz ask help from the tech team. Tks. |
Beta Was this translation helpful? Give feedback.
When conducting a zero-shot test, it is important to adjust the energy bias due to the differences in DFT settings between the pretraining and downstream datasets. Failing to do so will result in a significant energy shift. To automate this process, you can set
numb_steps=1(or maybe 0 is also ok) in theinput.jsonfile (containing downstream data systems, such as H2O-PBE0TS-MD, placed intraining/training_data) and perform a 1-step fine-tuning to update the energy bias based on the test dataset:This will generate a 1-step fine-tuned model (
model-1.pt) with an updated energy bias, which you can then use for zero-shot t…