Machine Learning for Molecules and Materials NeurIPS 2018 Workshop
Organizers:José Miguel Hernández-Lobato, Klaus-Robert Müller, Brooks Paige, Matt J. Kusner, Stefan Chmiela, Kristof T. Schütt
December 8, 2018 - Palais des Congrès de Montréal, Montréal, CANADA - Room 519
Abstract
The success of machine learning has been demonstrated time and time again in classification, generative modelling, and reinforcement learning. This revolution in machine learning has largely been in domains with at least one of two key properties: (1) the input space is continuous, and thus classifiers and generative models are able to smoothly model unseen data that is ‘similar’ to the training distribution, or (2) it is trivial to generate data, such as in controlled reinforcement learning settings such as Atari or Go games, where agents can re-play the game millions of times. Unfortunately there are many important learning problems in chemistry, physics, materials science, and biology that do not share these attractive properties, problems where the input is molecular or material data.
Accurate prediction of atomistic properties is a crucial ingredient toward rational compound design in chemical and pharmaceutical industries. Many discoveries in chemistry can be guided by screening large databases of computational molecular structures and properties, but high level quantum-chemical calculations can take up to several days per molecule or material at the required accuracy, placing the ultimate achievement of in silico design out of reach for the foreseeable future. In large part the current state of the art for such problems is the expertise of individual researchers or at best highly-specific rule-based heuristic systems. Efficient methods in machine learning, applied to the prediction of atomistic properties as well as compound design and crystal structure prediction, can therefore have pivotal impact in enabling chemical discovery and foster fundamental insights.
Because of this, in the past few years there has been a flurry of recent work towards designing machine learning techniques for molecule and material data [1-39]. These works have drawn inspiration from and made significant contributions to areas of machine learning as diverse as learning on graphs to models in natural language processing. Recent advances enabled the acceleration of molecular dynamics simulations, contributed to a better understanding of interactions within quantum many-body system and increased the efficiency of density based quantum mechanical modeling methods. This young field offers unique opportunities for machine learning researchers and practitioners, as it presents a wide spectrum of challenges and open questions, including but not limited to representations of physical systems, physically constrained models, manifold learning, interpretability, model bias, and causality.
The goal of this workshop is to bring together researchers and industrial practitioners in the fields of computer science, chemistry, physics, materials science, and biology all working to innovate and apply machine learning to tackle the challenges involving molecules and materials. In a highly interactive format, we will outline the current frontiers and present emerging research directions. We aim to use this workshop as an opportunity to establish a common language between all communities, to actively discuss new research problems, and also to collect datasets by which novel machine learning models can be benchmarked. The program is a collection of invited talks, alongside contributed posters. A panel discussion will provide different perspectives and experiences of influential researchers from both fields and also engage open participant conversation. An expected outcome of this workshop is the interdisciplinary exchange of ideas and initiation of collaboration.
Covariant neural network architectures for learning physics
Risi Kondor
16:30
Contributed talk
Learning protein structure with a differentiable simulator
16:40
Contributed talk
Generating equilibrium molecules with deep neural networks
16:50
Contributed talk
Molecular Transformer for Chemical Reaction Prediction and Uncertainty Estimation
17:00
Contributed talk
Steerable Wavelet Scattering for 3D Atomic Systems with Application to Li-Si Energy Prediction
17:10
Closing remarks
Brooks Paige
17:20
Poster session
Boltzmann Generators – Sampling Equilibrium States of Many-Body Systems with Deep Learning
Computing equilibrium states in condensed-matter many-body systems, such as solvated proteins, is a long-standing challenge. Lacking methods for generating statistically independent equilibrium samples directly, vast computational effort is invested for simulating these system in small steps, e.g., using Molecular Dynamics. Combining deep learning and statistical mechanics, we here develop Boltzmann Generators, that are shown to generate statistically independent samples of equilibrium states of representative condensed matter systems and complex polymers. Boltzmann Generators use neural networks to learn a coordinate transformation of the complex configurational equilibrium distribution to a distribution that can be easily sampled. Accurate computation of free energy differences, and discovery of new system states are demonstrated, providing a new statistical mechanics tool that performs orders of magnitude faster than standard simulation methods.
Predicting Electron-Ionization Mass Spectrometry using Neural Networks
When confronted with a substance of unknown identity, researchers often perform mass spectrometry on the sample and compare the observed spectrum to a library of previously-collected spectra to identify the molecule. While popular, this approach will fail to identify molecules that are not in the existing library. In response, we propose to improve the library’s coverage by augmenting it with synthetic spectra that are predicted using machine learning. We contribute a lightweight neural network model that quickly predicts mass spectra for small molecules. Achieving high accuracy predictions requires a novel neural network architecture that is designed to capture typical fragmentation patterns from electron ionization. We analyze the effects of our modeling innovations on library matching performance and compare our models to prior machine learning-based work on spectrum prediction.
Tensor Field Networks: rotation-, translation-, and permutation-equivariant convolutional neural networks for 3D points
Atomic systems (molecules, crystals, proteins, nanoclusters, etc.) are naturally represented by a set of coordinates in 3D space labeled by atom type. This is a challenging representation to use for neural networks because the coordinates are sensitive to 3D rotations and translations and there is no canonical orientation or position for these systems. We present a general neural network architecture that naturally handles 3D geometry and operates on the scalar, vector, and tensor fields that characterize physical systems. Our networks are locally equivariant to 3D rotations and translations at every layer. In this talk, we describe how the network achieves these equivariances and demonstrate the capabilities of our network using simple tasks: for example, given a small organic molecule with an atom removed, it can replace the correct element at the correct location in space.
Design of Coarse-grained Molecular Models with Machine Learning
High-resolution atomistic molecular dynamics (MD) simulations are
widely used to predict thermodynamics and kinetics of biomolecular
systems. However, the study of large macromolecular complexes on
biological timescales at atomistic resolution is still out of reach. A
common approach to go beyond the time- and length-scales accessible
with computationally expensive high-resolution MD is the definition of
coarse-grained (CG) models. Existing CG approaches define an effective
interaction potential to reproduce defined properties of
high-resolution models or experimental data. We reformulate the
definition of a CG model of a molecular system as a supervised
learning problem and illustrate the approach on test systems for which
the atomistic simulations can be fully sampled and compared with the
results of the CG model.
A translation approach to molecular graph optimization
Drug discovery is particularly challenging since the chemical space is vast and difficult to navigate. Recently, deep molecular generative models have been introduced to generate or optimize compounds with desirable properties. In this talk, we formulate molecular optimization as a graph-to-graph translation problem. The goal is to learn to transform one molecular graph to another with better properties based on a corpus of paired molecules. To this end, we introduce junction tree encoder-decoder to learn graph-to-graph mappings, and augment the model with latent variables to capture the diversity of outputs. Our model is evaluated on multiple molecule optimization tasks and it outperforms previous state-of-the-art baselines by a significant margin.
Application of graph neural networks in molecule design
There has been considerable excitement in the development of neural networks that handle graph structured data, and the application of these methods to problems in chemistry is gaining momentum. These methods have shown promise in regression of molecular properties from structure, and now there is growing interest in building graph structured generative models that can sample valid chemical structures and optimize those structures for a particular application. This talk will give a short overview of graph structured neural networks and challenges in applying these methods for building generative models of molecules.
Covariant neural network architectures for learning physics
Deep neural networks have proved to be extremely effective in image recognition, machine translation, and a variety of other data centered engineering tasks. However, generalizing neural networks to learning physical systems requires a careful examination of how they reflect symmetries. In this talk we give an overview of recent developments in the field of covariant/equivariant neural networks. Specifically, we focus on three applications: learning properties of chemical compounds from their molecular structure, image recognition on the sphere, and learning force fields for molecular dynamics. The work presented in this talk was done in collaboration with Brandon Anderson, Zhen Lin, Truong Son Hy, Horace Pan, and Shubhendu Trivedi.
Deep Reinforcement Learning for de-novo Drug Design
In this talk we will computational strategy for de-novo design of molecules with desired properties termed ReLeaSE (Reinforcement Learning for Structural Evolution). Based on deep and reinforcement learning approaches, ReLeaSE integrates two deep neural networks – generative and predictive – that are trained separately but employed jointly to generate novel targeted chemical libraries. ReLeaSE employs simple representation of molecules by their SMILES strings only. Generative models are trained with stack-augmented memory network to produce chemically feasible SMILES strings, and predictive models are derived to forecast the desired properties of the de novo generated compounds. In the first phase of the method, generative and predictive models are trained separately with a supervised learning algorithm. In the second phase, both models are trained jointly with the reinforcement learning approach to bias the generation of new chemical structures towards those with the desired physical and/or biological properties. In the proof-of-concept study, we have employed the ReLeaSE method to design chemical libraries with a bias toward structural complexity or biased toward compounds with either maximal, minimal, or specific range of physical properties such as melting point or hydrophobicity, as well as to develop novel inhibitors of JAK2 and EGFR kinases. The approach proposed herein can find a general use for generating targeted chemical libraries of novel compounds optimized for either a single desired property or multiple properties.
Deep Generative Models for Knowledge-Free Molecular Geometry
In this talk, I will present how we design a graph neural network based conditional autoencoder for predicting the coordinates of atoms in a molecule starting from its description. The proposed approach, which is free of domain knowledge, is empirically shown to perform similarly to the existing approaches based on the combination of the distance geometry (DG) and force field minimization however with a significantly lower variance.
Accepted Papers
Graph-Based Network using Attention Mechanism for Predicting Molecular Properties
Amir H. K. Ahmadi, Parsa Moradi, Babak H. Khalaj
Efficient prediction of 3D electron densities using machine learning [arXiv]
Mihail Bogojeski, Felix Brockherde, Leslie Vogt-Maranto, Li Li, Mark E. Tuckerman, Kieron Burke, Klaus-Robert Müller
Spotlight Talk Generating equilibrium molecules with deep neural networks [arXiv]
Niklas W. A. Gebauer, Michael Gastegger, Kristof T. Schütt
Spotlight Talk Powerful, transferable representations for molecules through intelligent task selection in deep multitask networks [arXiv]
Clyde Fare, Lukas Turcani, Edward O. Pyzer-Knapp
Spotlight Talk Incomplete Conditional Density Estimation for Fast Materials Discovery [GitHub]
Phuoc Nguyen, Truyen Tran, Sunil Gupta, Santu Rana, Svetha Venkatesh
Spotlight Talk Learning protein structure with a differentiable simulator
John Ingraham, Adam Riesselman, Chris Sander, Debora Marks
Spotlight Talk Predicting Electron-Ionization Mass Spectrometry using Neural Networks
Jennifer N. Wei, David Belanger, Ryan P. Adams, D. Sculley
Spotlight Talk Band gap prediction for large organic crystal structures with machine learning [arXiv]
Bart Olsthoorn, R. Matthias Geilhufe, Stanislav S. Borysov, Alexander V. Balatsky
Spotlight Talk Uncertainty quantification of molecular property prediction using Bayesian neural network models [arXiv]
Seongok Ryu, Yongchan Kwon, Woo Youn Kim
Batched Stochastic Bayesian Optimization via Combinatorial Constraints Design
Kevin K. Yang, Yuxin Chen, Alycia Lee, Yisong Yue
Spotlight Talk Steerable Wavelet Scattering for 3D Atomic Systems with Application to Li-Si Energy Prediction [arXiv]
Xavier Brumwell, Paul Sinz, Kwang Jin Kim, Yue Qi, Matthew Hirn
Chemical Structure Elucidation from Mass Spectrometry by Matching Substructures [arXiv]
Jing Lim, Joshua Wong, Minn Xuan Wong, Lee Han Eric Tan, Hai Leong Chieu, Davin Choo, Neng Kai Nigel Neo
Design by Adaptive Sampling
David H. Brookes, Jennifer Listgarten
Spotlight Talk Molecular Transformer for Chemical Reaction Prediction and Uncertainty Estimation [chemRxiv]
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Costas Bekas, Alpha A. Lee
Descriptor for Separating Base-material and Additive in Machine Learning of Thermoelectric Material Property Prediction
Dataset Bias in the Natural Sciences: A Case Study in Chemical Reaction Prediction and Synthesis Design [chemRxiv]
Ryan-Rhys Griffiths, Philippe Schwaller, Alpha A. Lee
Bayesian Optimization of High Transparency, Low Haze, and High Oil Contact Angle Rigid and Flexible Optoelectronic Substrates
Sajad Haghanifar, Sooraj Sharma, Luke M. Tomasovic, and Paul. W. Leu, Bolong Cheng
Fast classification of small X-ray diffraction datasets using physics-based data augmentation and deep neural networks
Felipe Oviedo, Zekun Ren, Shijing Sun, Charlie Settens, Zhe Liu, Giuseppe Romano, Tonio Buonassisi, Ramasamy Savitha, Siyu I.P. Tian, Brian L. DeCost, Aaron Gilad Kusne
PaccMann: Prediction of anticancer compound sensitivity with multi-modal attention-based neural networks [arXiv]
Ali Oskooei, Jannis Born, Matteo Manica, Vigneshwari Subramanian, Julio Sáez-Rodríguez, María Rodríguez Martínez
Multiple-objective Reinforcement Learning for Inverse Design and Identification
Haoran Wei, Mariefel Olarte, Garrett B. Goh
Efficient nonmyopic active search with applications in drug and materials discovery [arXiv]
Shali Jiang, Gustavo Malkomes, Benjamin Moseley, Roman Garnett
Inference of the three-dimensional chromatin structure and its temporal behavior [arXiv]
Bianca-Cristina Cristescu, Zalán Borsos, John Lygeros, María Rodríguez Martínez, Maria Anna Rapsomaniki
Neural Message Passing with Edge Updates for Predicting Properties of Molecules and Materials [arXiv]
Peter Bjørn Jørgensen, Karsten Wedel Jacobsen, Mikkel N. Schmidt
Independent Vector Analysis for Data Fusion Prior to Molecular Property Prediction with Machine Learning [arXiv]
Zois Boukouvalas, Daniel C. Elton, Peter W. Chung, Mark D. Fuge
Optimizing Interface/Surface Roughness for Thermal Transport
Shenghong Ju, Thaer M. Dieb, Koji Tsuda, Junichiro Shiomi
Graph Convolutional Neural Networks for Polymers Property Prediction [arXiv]
Behler, J., Lorenz, S., Reuter, K. (2007). Representing molecule-surface interactions with symmetry-adapted neural networks. J. Chem. Phys., 127(1), 07B603.
[2]
Behler, J., Parrinello, M. (2007). Generalized neural-network representation of high-dimensional potential-energy surfaces. Phys. Rev. Lett., 98(14), 146401.
[3]
Kang, B., Ceder, G. (2009). Battery materials for ultrafast charging and discharging. Nature, 458(7235), 190.
[4]
Bartók, A. P., Payne, M. C., Kondor, R., Csányi, G. (2010). Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Phys. Rev. Lett., 104(13), 136403.
[5]
Behler, J. (2011). Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys, 134(7), 074106.
[6]
Behler, J. (2011). Neural network potential-energy surfaces in chemistry: a tool for large-scale simulations. Phys. Chem. Chem. Phys., 13(40), 17930-17955.
[7]
Rupp, M., Tkatchenko, A., Müller, K.-R., von Lilienfeld, O. A. (2012). Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett., 108(5), 058301.
[8]
Snyder, J. C., Rupp, M., Hansen, K., Müller, K.-R., Burke, K. (2012). Finding density functionals with machine learning. Phys. Rev. Lett., 108(25), 253002.
[9]
Montavon, G., Rupp, M., Gobre, V., Vazquez-Mayagoitia, A., Hansen, K., Tkatchenko, A., Müller, K.-R., von Lilienfeld, O. A. (2013). Machine learning of molecular electronic properties in chemical compound space. New J. Phys., 15(9), 095003.
[10]
Hansen, K., Montavon, G., Biegler, F., Fazli, S., Rupp, M., Scheffler, M., Tkatchenko, A., Müller, K.-R. (2013). Assessment and validation of machine learning methods for predicting molecular atomization energies. J. Chem. Theory Comput., 9(8), 3404-3419.
[11]
Bartók, A. P., Kondor, R., Csányi, G. (2013). On representing chemical environments. Phys. Rev. B, 87(18), 184115.
[12]
Schütt K. T., Glawe, H., Brockherde F., Sanna A., Müller K.-R., Gross E. K. U. (2014). How to represent crystal structures for machine learning: towards fast prediction of electronic properties. Phys. Rev. B., 89(20), 205118.
[13]
Ramsundar, B., Kearnes, S., Riley, P., Webster, D., Konerding, D., Pande, V. (2015). Massively multitask networks for drug discovery. arXiv preprint arXiv:1502.02072.
[14]
Rupp, M., Ramakrishnan, R., & von Lilienfeld, O. A. (2015). Machine learning for quantum mechanical properties of atoms in molecules. J. Phys. Chem. Lett., 6(16), 3309-3313.
[15]
V. Botu, R. Ramprasad (2015). Learning scheme to predict atomic forces and accelerate materials simulations., Phys. Rev. B, 92(9), 094306.
[16]
Hansen, K., Biegler, F., Ramakrishnan, R., Pronobis, W., von Lilienfeld, O. A., Müller, K.-R., Tkatchenko, A. (2015). Machine learning predictions of molecular properties: Accurate many-body potentials and nonlocality in chemical space. J. Phys. Chem. Lett, 6(12), 2326-2331.
[17]
Alipanahi, B., Delong, A., Weirauch, M. T., Frey, B. J. (2015). Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning. Nat. Biotechnol., 33(8), 831-838.
[18]
Duvenaud, D. K., Maclaurin, D., Aguilera-Iparraguirre, J., Gomez-Bombarelli, R., Hirzel, T., Aspuru-Guzik, A., Adams, R. P. (2015). Convolutional networks on graphs for learning molecular fingerprints. NeurIPS, 2224-2232.
[19]
Faber F. A., Lindmaa A., von Lilienfeld, O. A., Armiento, R. (2016). Machine learning energies of 2 million elpasolite (A B C 2 D 6) crystals. Phys. Rev. Lett., 117(13), 135502.
[20]
Gomez-Bombarelli, R., Duvenaud, D., Hernandez-Lobato, J. M., Aguilera-Iparraguirre, J., Hirzel, T. D., Adams, R. P., Aspuru-Guzik, A. (2016). Automatic chemical design using a data-driven continuous representation of molecules. arXiv preprint arXiv:1610.02415.
[21]
Wei, J. N., Duvenaud, D, Aspuru-Guzik, A. (2016). Neural networks for the prediction of organic chemistry reactions. ACS Cent. Sci., 2(10), 725-732.
[22]
Sadowski, P., Fooshee, D., Subrahmanya, N., Baldi, P. (2016). Synergies between quantum mechanics and machine learning in reaction prediction. J. Chem. Inf. Model., 56(11), 2125-2128.
[23]
Lee, A. A., Brenner, M. P., Colwell L. J. (2016). Predicting protein-ligand affinity with a random matrix framework. Proc. Natl. Acad. Sci., 113(48), 13564-13569.
[24]
Behler, J. (2016). Perspective: Machine learning potentials for atomistic simulations. J. Chem. Phys., 145(17), 170901.
[25]
De, S., Bartók, A. P., Csányi, G., Ceriotti, M. (2016). Comparing molecules and solids across structural and alchemical space. Phys. Chem. Chem. Phys., 18(20), 13754-13769.
[26]
Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K.-R., Tkatchenko, A. (2017). Quantum-chemical insights from deep tensor neural networks. Nat. Commun., 8, 13890.
[27]
Segler, M. H., Waller, M. P. (2017). Neural‐symbolic machine learning for retrosynthesis and reaction prediction. Chem. Eur. J., 23(25), 5966-5971.
[28]
Kusner, M. J., Paige, B., Hernández-Lobato, J. M. (2017). Grammar variational autoencoder. arXiv preprint arXiv:1703.01925.
[29]
Coley, C. W., Barzilay, R., Jaakkola, T. S., Green, W. H., Jensen K. F. (2017). Prediction of organic reaction outcomes using machine learning. ACS Cent. Sci., 3(5), 434-443.
[30]
Altae-Tran, H., Ramsundar, B., Pappu, A. S., Pande, V. (2017). Low data drug discovery with one-shot learning. ACS Cent. Sci., 3(4), 283-293.
[31]
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., Dahl, G. E. (2017). Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212.
[32]
Chmiela, S., Tkatchenko, A., Sauceda, H. E., Poltavsky, Igor, Schütt, K. T., Müller, K.-R. (2017). Machine learning of accurate energy-conserving molecular force fields. Sci. Adv., 3(5), e1603015.
[33]
Ju, S., Shiga T., Feng L., Hou Z., Tsuda, K., Shiomi J. (2017). Designing nanostructures for phonon transport via bayesian optimization. Phys. Rev. X, 7(2), 021024.
[34]
Ramakrishnan, R, von Lilienfeld, A. (2017). Machine learning, quantum chemistry, and chemical space. Reviews in Computational Chemistry, 225-256.
[35]
Hernandez-Lobato, J. M., Requeima, J., Pyzer-Knapp, E. O., Aspuru-Guzik, A. (2017). Parallel and distributed Thompson sampling for large-scale accelerated exploration of chemical space. arXiv preprint arXiv:1706.01825.
[36]
Smith, J., Isayev, O., Roitberg, A. E. (2017). ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost. Chem. Sci., 8(4), 3192-3203.
[37]
Brockherde, F., Li, L., Burke, K., Müller, K.-R. By-passing the Kohn-Sham equations with machine learning. Nat. Commun., 8, 872.
[38]
Schütt, K. T., Kindermans, P. J., Sauceda, H. E., Chmiela, S., Tkatchenko, A., Müller, K. R. (2017). MolecuLeNet: A continuous-filter convolutional neural network for modeling quantum interactions. NeurIPS 2017.
[39]
Chmiela, S., Sauceda, H. E., Müller, K.-R., Tkatchenko, A. Towards Exact Molecular Dynamics Simulations with Machine-Learned Force Fields. Nat. Commun., 9(1), 3887.