Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA

Michał Kulczykowski; Rafał Łabędzki

Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA

3–5 minutes

read

•

2 September, 2026

Read full paper

Open PDF

Abstract

Self-supervised pretraining has transformed language and vision, but its value for molecular graph neural networks remains contested. We ask whether pretraining on a large unlabelled corpus improves molecular property prediction. We adapt LeJEPA, a predictor-free joint-embedding predictive architecture regularised by Sketched Isotropic Gaussian Regularisation (SIGReg), to molecular graphs, evaluating GPS and Chemprop-style D-MPNN encoders on the Wong et al. [1] antibiotic-activity dataset and ogbg-molhiv using a multi-seed, bootstrap-based protocol. Pretraining improves learned representations but does not robustly improve finetuning. A frozen probe on pretrained embeddings exceeds random initialisation on both tasks (ogbg-molhiv ROC-AUC 0.788 vs 0.665; +0.123), reaching the published self-supervised band, but this does not translate into finetuning gains. On the antibiotic scaffold split, a canonical partition is significant (delta AUPRC +0.041, p = 0.010), but the effect vanishes across five partitions (pooled +0.013, p = 0.095). Finetuning is null on the random split, ogbg-molhiv, and D-MPNN. The representational edge is nevertheless recoverable. Embeddings saturate at ~16-32 effective dimensions, whereas Morgan fingerprints improve to 1024 bits. At matched dimensionality, fingerprints lead validation (0.799 vs 0.782 at 128 dimensions) but trail shifted test scaffolds (0.759 vs 0.788). Truncating embeddings and combining them with a 1024-bit Morgan fingerprint raises ogbg-molhiv ROC-AUC from 0.805 to 0.832 (delta +0.027; 95% CI [+0.003, +0.054]; p = 0.014); an untrained encoder gains nothing (delta -0.003). Thus, pretraining supplies complementary information best realised through feature-level combination, while finetuning gains are weak and partition-dependent.

Authors: Michał Kulczykowski, Rafał Łabędzki

References

Wong, F. et al. Discovery of a structural class of antibiotics with explainable deep learning. Nature 2024, 626, 177–185, DOI: 10.1038/s41586-023-06887-8.
Gilmer, J.; Schoenholz, S. S.; Riley, P. F.; Vinyals, O.; Dahl, G. E. In International conference on machine learning, 2017, pp 1263–1272.
Irwin, J. J.; Tang, K. G.; Young, J.; Dandarchuluun, C.; Wong, B. R.; Khurelbaatar, M.; Moroz, Y. S.; Mayfield, J.; Sayle, R. A. ZINC20—A Free Ultralarge-Scale Chemical Database for Ligand Discovery. Journal of Chemical Information and Modeling 2020, 60, 6065–6073, DOI: 10 . 1021 / acs . jcim .0c00675.
Hou, Z.; Liu, X.; Cen, Y.; Dong, Y.; Yang, H.; Wang, C.; Tang, J. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, 2022, pp 594–604.
Hu, W.; Liu, B.; Gomes, J.; Zitnik, M.; Liang, P.; Pande, V.; Leskovec, J. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 2019.
Wang, Y.; Wang, J.; Cao, Z.; Barati Farimani, A. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence 2022, 4, 279–287.
LeCun, Y. et al. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review 2022, 62, 1–62.
Maes, L.; Le Lidec, Q.; Scieur, D.; LeCun, Y.; Balestriero, R. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv preprint arXiv:2603.19312 2026.
Skenderi, G.; Li, H.; Tang, J.; Cristani, M. Graph-level Representation Learning with Joint-Embedding Predictive Architectures. arXiv preprint arXiv:2309.16014 2023.
Balestriero, R.; LeCun, Y. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics, 2025.
O’neill, J. Antimicrobial resistance: tackling a crisis for the health and wealth of nations. Rev. Antimicrob. Resist. 2014.
Tang, K. W. K.; Millar, B. C.; Moore, J. E. Antimicrobial Resistance (AMR). British Journal of Biomedical Science 2023, Volume 80 – 2023, DOI: 10.3389/bjbs.2023.11387.
Aslam, B.; Asghar, R.; Muzammil, S.; Shafique, M.; Siddique, A. B.; Khurshid, M.; Ijaz, M.; Rasool, M. H.; Chaudhry, T. H.; Aamir, A.; Baloch, Z. AMR and Sustainable Development Goals: at a crossroads. Globalization and Health 2024, 20, 73, DOI: 10.1186/s12992-024-01046-8.
Amann, S.; Neef, K.; Kohl, S. Antimicrobial resistance (AMR). European Journal of Hospital Pharmacy 2019, 26, 175–177, DOI: 10.1136/ejhpharm-2018-001820.
Walsh, C. Where will new antibiotics come from? Nature Reviews Microbiology 2003, 1, 65–70.
Stokes, J. M.; Yang, K.; Swanson, K.; Jin, W.; Cubillos-Ruiz, A.; Donghia, N. M.; MacNair, C. R.; French, S.; Carfrae, L. A.; Bloom-Ackermann, Z., et al. A deep learning approach to antibiotic discovery. Cell 2020, 180, 688–702.18
Hu, W.; Fey, M.; Zitnik, M.; Dong, Y.; Ren, H.; Liu, B.; Catasta, M.; Leskovec, J. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Rampášek, L.; Galkin, M.; Dwivedi, V. P.; Luu, A. T.; Wolf, G.; Beaini, D. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems 2022, 35, 14501–14515.
Dwivedi, V. P.; Luu, A. T.; Laurent, T.; Bengio, Y.; Bresson, X. In International Conference on Learning Representations (ICLR), 2022.
Yang, K.; Swanson, K.; Jin, W.; Coley, C.; Eiden, P.; Gao, H.; Guzman-Perez, A.; Hopper, T.; Kelley, B.; Mathea, M., et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling 2019, 59, 3370–3388.
Xu, K.; Hu, W.; Leskovec, J.; Jegelka, S. In International Conference on Learning Representations (ICLR), 2019.
Ying, C.; Cai, T.; Luo, S.; Zheng, S.; Ke, G.; He, D.; Shen, Y.; Liu, T.-Y. In Advances in Neural Information Processing Systems (NeurIPS), 2021.