“Convex hull and its distance” in the paper "Matbench Discovery An evaluation framework for machine learning crystal stability prediction"

[Claudia-Hello]

Hi,
Sorry but I would like to ask, in the paper "Matbench Discovery: An evaluation framework for machine learning crystal stability prediction", how is the distance between a particular structure and a convex hull calculated? And how is the distance between a group of structures and a convex hull obtained? If I understand correctly, this is called convex hull distance, right?

Thank you so much!!

[janosh]

a group of structures in the same chemical system with their respective computed energies is what spans the convex hull. see here for how we compute the distance to that energy surface: #39 (comment). you may find that whole thread instructive. let me know if that doesn't help.

[Claudia-Hello]

So, what you're saying is that we can use this line of code to calculate how far the model predicts formation energy relative to the convex hull? But the .py file no longer exists in the project. In this case, then how should we calculate the convex hull distance now?

In addition, I am a little confused, for some of the regression metrics in the table below, such as mae, are these metrics for the formation energy predicted by the model? I mean, when the model does the regression prediction, does it predict the crystal formation energy? If so, then the convex hull distance is calculated by the code as you answered earlier, right?

Thank you!

[janosh]

I mean, when the model does the regression prediction, does it predict the crystal formation energy?

correct

If so, then the convex hull distance is calculated by the code as you answered earlier, right?

yes

such as mae, are these metrics for the formation energy predicted by the model

yes, but also have a look at this code comment. the MAE for formation energy and hull distance are actually the same

matbench-discovery/matbench_discovery/preds.py

Lines 220 to 230 in 663b4d1

	# To avoid confusion for anyone reading this code, we calculate the formation energy MAE
	# here and report it as the MAE for the energy above the convex hull prediction. The
	# former is more easily calculated but the two quantities are the same. The formation
	# energy of a material is the difference in energy between a material and its
	# constituent elements in their standard states. The distance to the convex hull is
	# defined as the difference between a material's formation energy and the minimum
	# formation energy of all possible stable materials made from the same elements. Since
	# the formation energy of a material is used to calculate the distance to the convex
	# hull, the error of a formation energy prediction directly determines the error in the
	# distance to the convex hull prediction.

But the .py file no longer exists in the project

the file was renamed but it's still there

matbench-discovery/data/wbm/compile_wbm_test_set.py

Line 555 in 663b4d1

e_above_hull = ppd_mp.get_e_above_hull(cse, allow_negative=True)

[Claudia-Hello]

I got questions about the definition of convex hull and its construction, so let's use an example to make this clear.

Suppose we have a dataset containing three structures that are composed of two elements, which are hydrogen (H) and oxygen (O).

Dataset
Our dataset contains the following 3 structures:
Water (H2O)
Oxygen (O2)
Hydroxide (H2O2)

1. Here comes the first question:
   There are still many hydroxides that are not included in our dataset (let's say there are 20), so do we need to consider these 20 compounds when defining and constructing the convex hull? If so, should we consider only the hydroxides? I mean, do we need to consider othe forms like ionic forms?

2. Then, how should we construct the convex hull?

Thank you~

[janosh]

in practice, the convex hull can only contain the compounds we know about. as such it's almost guaranteed to be incomplete unless the chemical space has already been extremely well sampled. for famous/common materials, this may be the case but in the context of materials discovery, it's rarely the case. the true convex hull will be lower in energy than the one spanned by the points we've so far sampled in that chemical space, meaning the materials we think are thermodynamically stable may not be with respect to the true hull

[Claudia-Hello]

Thank you very much!