materials-toolkits/lematerial

LeMaterial

Crystal structures from LeMaterial (PBE), in two independent forms:

The HDF5 files and the CIF files come from different releases and selections of LeMaterial: one is not a conversion of the other.

Datasets curated from LeMaterial/LeMatBulkUnique - unique_pbe version 1.1

This repository is aimed to contain a curated version of the PBE calculated part of LeMaterial dataset created by Entalpic and available at "https://huggingface.co/datasets/LeMaterial/LeMat-BulkUnique" as "unique_pbe".

The curation was done using Pymatgen version 2024.10.3.

The curated dataset is given directly in CIF format for easy readability by usual crystal data reader softwares and python packages. The CIFs were generated by pymatgen's CifWriter class, but the header containing the unique "materialid" key and the entalpic_fingerprint line containing the hash value generated by Entalpic's hash algorithm were added by us using a script modifying the already CifWriter-generated data.

Curation steps

Were filtered out:

• —All structures containing at least one rare gas element (group 18 of periodic table, i.e. He (Z = 2), Ne (Z = 10), Ar (Z = 18), Kr (Z = 36), Xe (Z = 54), Rn (Z = 86) and Og (Z = 118)).

• —All structures containing at least one of the 28 elements with f-type orbital valence (from La (Z = 57) to Yb (Z = 69), and from Ac (Z = 89) to No (Z = 102)).

NOTE: Lu (Z = 90) and Lr (Z = 103) were not touched because their valence electron is actually in a d-type orbital, they are d1 elements just like Sc (Z = 21) and Y (Z = 39).

• —All unphysical structures having at least a pair of atoms closer than 0.5 angströms.

• —All structures considered equivalent by the StructureMatcher with default parameters of fractional length tolerance ltol = 0.2, site tolerance stol = 0.3, and angle tolerance angle_tol = 5.0 degrees.

NOTE: Even if we exctracted the initial data from the "Unique" dataset which passed Entalpic's hash algorithm, a lot of duplicates remained for the StructureMatcher. We put an emphasis on this to underline the statement from Entalpic's own page that their hash method is not perfect yet, but it is still a very useful approach by its efficiency in order to do a pre-filtering task.

Dataset files specifications

• —The file simply named "lematerial.cif" contains all 2,838,953 structures that passed all curation steps mentioned above.
• —The files whose name contain a crystal system name are crystal system specific subsets of the full dataset. The classification relies on the SpacegroupAnalyzer class of Pymatgen with default fractional coordinate tolerance of 0.01 and angle tolerance of 5.0 degrees.
• —The data in the subsets is not symmetrized in order to have all true atomic positions as given in the original dataset and not ones estimated by symmetry operations.
• —The file named "lematerial_metastables.cif" is a subset of "lematerial.cif" containing only materials with DFT hull energies under 0.1 eV/atom (energies extracted from the LeMatBulk-DFT-Hull-All dataset).

Subsets specifications

• —"lematerial_triclinics.cif" contains 101,024 triclinic structures
• —"lematerial_monoclinics.cif" contains 441,922 monoclinic structures
• —"lematerial_orthorhombics.cif" contains 449,888 orthorhombic structures
• —"lematerial_tetragonals.cif" contains 905,269 tetragonal structures
• —"lematerial_trigonals.cif" contains 272,257 trigonal structures
• —"lematerial_hexagonals.cif" contains 97,285 hexagonal structures
• —"lematerial_cubics.cif" contains 571,308 cubic structures
• —"lematerial_metastables.cif" contains 506,853 metastable (Ehull <= 0.1 eV/atom) structures from any crystal system
• —"lematerialmetastableswenergies.cif" is equivalent to "lematerialmetastables.cif" but total energies and energies above hull are added to data under the 'totalenergypbe' and 'abovehullenergy_pbe' keys, respectively.

Complete LeMat-Bulk PBE in HDF5 (all.hdf5, unique.hdf5)

These two files are a separate, unfiltered export of LeMaterial/LeMat-Bulk (compatible_pbe, revision 0dc17eea), preprocessed with pycrystalgen. They do not replace the curated CIF files described above and do not apply their curation steps: rare gases, f-block elements, close atoms and StructureMatcher duplicates are all kept.

Structures have 1 to 444 atoms (median 6). They come from Alexandria (4,628,422), OQMD (567,945) and the Materials Project (138,931).

unique.hdf5 keeps, among the structures sharing the same entalpic_fingerprint, the one with the lowest energy per atom (the first one when equal). The 46 structures without fingerprint are kept. 408,261 structures are removed this way.

Labels

34 structures of all.hdf5 (31 of unique.hdf5) have no value in LeMat-Bulk-DFT-Hull-All. Their energy above the hull was computed from the total energies of the file itself, with the convex hull of each chemical system. The same computation reproduces the published values of all the other structures within 1e-6 eV/atom, except two structures of the As-H-Hg-O-Zn system which differ by 0.0003 eV/atom: mp-24216 is on the hull but has no published value, so the published hull of this system was built without it. Published values are stored unchanged.

The energies of the structures from OQMD are shifted for nine elements (Gd, Cu, Ni, Pu, Bi, Pa, Fe, Np, Re) compared with the two other sources, and the published energy above the hull mixes the three: see the convex hull, built without OQMD, before using energy_above_hull to select stable structures.

Labels are not cleaned: 152 structures of all.hdf5 have a positive energy per atom (up to +648 eV/atom) and the energy above the hull goes up to 651 eV/atom. Filter them for training.

The number of atoms is the length of species_at_sites; the column nsites of LeMat-Bulk disagrees with it for one structure (oqmd-2969647: 13 atoms, nsites 8).

Format

Flat concatenated arrays:

The lattice is rebuilt from cell lengths and angles, so it is in the standard orientation (first vector along x). Fractional coordinates are wrapped into the cell. The batch array is an artefact of the writer and is not used when reading. Structures are in the order of LeMat-Bulk; row i of an index file describes structure i (one structure of LeMat-Bulk has no immutable_id).

Usage

from pycrystalgen.data import load_dataset

dataset = load_dataset("lematerial", "unique")  # or "all"
structures = dataset[:32]  # batched Structures

Processing and checks

Built with pycrystalgen (scripts/build_lematerial.py): structures are written to CIF then preprocessed to HDF5. Every structure of both files was compared with its row of LeMat-Bulk: number of atoms, elements, fractional coordinates, lattice, total energy and energy above the hull.

LeMat-Bulk is distributed under CC-BY-4.0; refer to it for the terms of use and the references of its sources.

Convex hull of LeMat-Bulk PBE (hull.hdf5)

hull.hdf5 holds the composition and the energy of the 102,910 structures on the convex hull of LeMat-Bulk compatible_pbe, built from its structures of the Materials Project and Alexandria only (4,767,353 structures). It is enough to compute the energy above the hull of any other structure: the hull of a chemical system only depends on its stable structures and the ones of its sub-systems.

Why the structures from OQMD are left out

LeMat-Bulk gathers structures from the Materials Project, Alexandria and OQMD, and a convex hull compares their total energies. For nine elements, the total energies of the structures from OQMD are not on the scale of the two other sources:

The shift is the difference between the energy of a structure from OQMD and the energy the same structure would have in the two other sources. It was measured with a pretrained energy model (see the estimator below): the difference between its predictions and the energies of the dataset, per atom of each element, is the same for the Materials Project and Alexandria and differs by this amount for OQMD. For Gd, Cu and Ni it is also the difference between the lowest energy of the element alone in OQMD and in the two other sources. 209,531 of the 567,945 structures from OQMD (37%) contain one of these elements.

A hull mixing the three sources is wrong for these elements:

• —with a positive shift, structures from OQMD look less stable than they are;
• —with a negative shift (Bi), structures from OQMD look more stable than they are. In the hull of the three sources, 1,316 of the 3,044 stable structures with bismuth came from OQMD and hid the stable structures of the two other sources: 1,221 structures with bismuth of the Materials Project and Alexandria are on this hull and were not on the mixed one.

Compared with the hull of the three sources (the first version of this file, 109,112 structures of which 13,685 from OQMD), for the structures of the Materials Project and Alexandria:

• —7,487 more are on the hull;
• —the energy above the hull is unchanged (within 1e-4 eV/atom) for 89.9%, lower by more than 0.01 eV/atom for 69,148 and by more than 0.05 eV/atom for 18,070.

What is lost. Structures from OQMD are also left out for the other elements, where their energies are comparable: a phase which is stable and only known from OQMD is not on this hull. Compared to this hull, 18,162 structures from OQMD are below it: 3,465 contain a shifted element (median 0.036 eV/atom below, the shift for most of them) and 14,693 do not (median 0.002 eV/atom below; 2,589 by more than 0.01 and 213 by more than 0.05). In the chemical systems of these last ones the hull is too high by this amount.

`all.hdf5` and `unique.hdf5` are not changed. Their energy_above_hull is the published one (LeMat-Bulk-DFT-Hull-All), computed with the three sources. It differs from the energy above this hull by more than 0.001 eV/atom for 231,949 structures of the Materials Project and Alexandria (4.9%), and it is not reliable for the structures from OQMD with a shifted element. To get the energy above this hull, use energy_above_reference below with the total_energy of the files.

Format

Unlike all.hdf5, ptr has one more entry than the number of structures, and the energy is per atom, in 64 bits, from the energies of LeMat-Bulk. There is no lattice and no atomic position: hull_index.parquet gives the immutable_id of each row, to find the structure in all.hdf5 or in LeMat-Bulk.

The hull covers 89 elements and has stable compounds in 58,671 chemical systems: 90 structures of one element (two structures of the same element have exactly the same energy), 6,763 binary, 64,759 ternary, 30,204 quaternary and 1,094 with five to seven elements. 88,047 come from Alexandria and 14,863 from the Materials Project.

Usage

from huggingface_hub import hf_hub_download
from pycrystalgen.data.hull import energy_above_reference, load_hull

hull = load_hull(hf_hub_download("materials-toolkits/lematerial", "hull.hdf5", repo_type="dataset"))

# z: atomic numbers of all atoms, num_atoms and total_energy (eV): one value per structure
above = energy_above_reference(hull, z, num_atoms, total_energy)

The result is in eV/atom: 0 on the hull, positive above, negative for a structure more stable than the hull, NaN for a structure with an element absent from the hull.

Energies must be comparable with the ones of the Materials Project and Alexandria in LeMat-Bulk `compatible_pbe` (PBE, same calculation settings). The energy of a structure relaxed with another functional gives a number that is not an energy above this hull. For an energy predicted by a machine learning model, see the estimator below.

Checks

• —Comparing the 4,767,353 structures of the Materials Project and Alexandria to the saved hull gives back the energy above the hull computed on all of them within 1e-14 eV/atom, with the same 102,910 structures at zero.
• —The same computation on the three sources reproduces the published values of LeMat-Bulk-DFT-Hull-All within 1e-6 eV/atom for all structures but two (see above).

Estimated total energy (energy_correction_orb.hdf5)

pycrystalgen.estimator estimates the PBE total energy of a structure, on the scale of the hull above, with a pretrained graph network: orb-v3 conservative (orb-v3-conservative-inf-mpa) of Orbital Materials (orb-models, Apache-2.0, arXiv:2504.06231). The energy prediction of the network is rewritten in pycrystalgen without other dependency than PyTorch; it loads the official weights and gives the energies of orb-models 0.7.0 within 2e-5 eV/atom (5e-4 for cells with more than 120 atoms closer than 6 Å of an atom).

energy_correction_orb.hdf5 is a correction of the predicted energy: an energy per atom of each element (correction, shape (128,), eV/atom, indexed by atomic number; counts is the number of fitted structures with the element), fitted to minimize the absolute error on 300,000 structures of the Materials Project and Alexandria of LeMat-Bulk. It is small: 0.003 eV/atom on average, at most 0.024 (Pu). This checkpoint is trained on energies computed like the ones of the Materials Project, and is already on their scale.

from pycrystalgen.data import HDF5Dataset
from pycrystalgen.estimator import EnergyEstimator

# orb-v3-conservative-inf-mpa, energy_correction_orb.hdf5 and hull.hdf5 of this repository:
# the hull is the one of the Materials Project and Alexandria structures, without OQMD
estimator = EnergyEstimator.from_lematerial(device="cuda")
print(estimator.hull_source)

# dataset: total energy in eV and energy above the hull in eV/atom
energy, above = estimator.predict_energy_above_hull(HDF5Dataset("structures.hdf5"))

# batch of Structures on the device of the model
above = estimator.energy_above_hull(structures)

The energy above the hull is negative for a structure estimated more stable than the hull, and NaN for a structure with an element absent from the hull.

Accuracy

Structures of all.hdf5 which were not used to fit the correction (energy_splits.parquet lists them and the fitted ones; fitted and tested structures never share a fingerprint). The network is trained on the Materials Project and Alexandria, not on OQMD: sets B and C are structures from OQMD with a fingerprint (B) or a composition (C) absent from both.

Energies above the hull are computed with hull.hdf5, for DFT and for the estimator; a structure is stable when it is on or below the hull. Without the correction the errors of A, B and C are 0.0100, 0.0104 and 0.0122 eV/atom, and 27% of the stable structures of A are found. energy_validation.json holds these numbers.

• —The error is about 0.01 eV/atom, the same for structures and compositions outside of the training data of the network. It tells if a structure is within 0.1 eV/atom of the hull; it does not tell reliably if it is exactly on the hull.
• —Set D is the same as B with the nine elements whose energies in OQMD are shifted: the error is the shift of the reference energies, not of the estimator, which gives the energy on the scale of the Materials Project and Alexandria.
• —The energy is the one of the structure as given: it is not relaxed.
• —The network is not exactly invariant to rotations: the energy changes by about 2e-4 eV/atom on average (up to 8e-3) with the orientation of the cell.
• —A structure with two atoms at the same position has no energy (NaN); one structure of LeMat-Bulk is in this case (oqmd-2969647).