Media and Talks
AI Safety Research at MATS
Understanding the internals of large language models
Mechanistic Interpretability (mech interp) is the field of study of reverse engineering neural networks from the learned weights down to human-interpretable algorithms. Sparse Autoencoders (SAEs) help provide a useful basis for this exploration. As part of the Blue Dot AI Safety Fundamentals Spring 2024 course, I played with the SAE features of GPT2 small.
You can look at my exploration findings here.
Frontier Development Lab 2022 (Forecasting Earthquakes around CO2 sequestration sites with DoE)
In the summer of 2022 McKinsey and company sponsored the Frontier Development Lab by staffing a Data Scientist, Constantin, to be the machine learning and process lead for the CO2 project. Here is an overview of its motivations and goals:
Climate change from greenhouse gas emissions pose serious sustainability challenges for humanity. Achieving carbon neutrality by 2050 mandates capturing and storing over 30 billion metric tons of CO2, in addition to reducing emissions. The most promising storage solution is geologic carbon sequestration: injecting CO2 into the earth’s surface. However, previous studies have shown that excessive CO2 injections can cause earthquakes. Forecasting these earthquakes from pressure and past seismological data can allow sequestration operators to decide when to slow down or pause operations.
Current state-of-the-art physical methods are either inaccurate or have suboptimal software implementations that are prohibitively slow and require seismology-PhD-level expertise to train (e.g. CRS model). For each new sequestration site, seismologists have to wait for sufficient data before making useful predictions. Hence, the community is eager for outside approaches. The aim of this engagement is to tackle these shortcomings while retaining trust of all stakeholders. Supported by more induced seismological data than any other team has had access to and endorsed by seismology and ML experts, this project’s long term improvement targets and stages are:
- 1. Accessibility and Training Speed: Produce a streamlined implementation of the CRS model
- 2. Accuracy: Adapt state-of-the-art ML time series methods like SCInet to be used alongside CRS predictions
- 3. Trust and Transfer Learning: Build a hypernetwork that can predict the optimal parameters on the CRS model using less data than required to perform a CRS model fit
Start 39:20 mins in: Panel on AI for National Challenges with Pamela Isom (Director, Artificial Intelligence and Technology - U.S. Department of Energy), Madhulika Guhathakurta (Lead Astonomist, SETI Institute), Constantin Weisser
Frontier Development Lab 2021 (Coastal Digital Twin with NASA)
Joined the collaboration between NASA Ames, the SETI Institute, and commercial AI partners like Google Cloud to minimize the deleterious effects of climate change. Our four-person team's approach to this problem is to speed up flood predictions by building a machine learning surrogate model for an established inundation height model like NEMO (Nucleus for European Modelling of the Ocean). We were the first to apply the physics-informed state-of-the-art Fourier Neural Operator models to a real-world setting and predicted inundation height better than the physical and ML baselines. Furthermore, we provide a package called "coastal twin" that documents and automates our work and is a platform for further collaboration for earth and ML scientists. Our work is a step in the direction of making practical, high-precision flooding predictions a possibility, which would inform evacuation orders and save lives.
PhD Defense
On March 30th 2021 I successfully defended my thesis. Hence, I became the first person to receive a PhD in Physics, Statistics, and Machine Learning from MIT.
MIT Consulting Club Interview
CERN & The Sense of Beauty
Moment Decomposition
At the NeurIPS 2020 "Machine Learning and the Physical Sciences" workshop, my colaborator presented our work on Moment Decomposition (MoDe). MoDe is a fast and easy general method to flexibly control ML classifier dependence on a continuous feature. While this approach is crucial to prevent biasing of particle physics analyses while retaining classification power, it could be applied to constrain the COVID vaccine distribution fractions to be at most quadratic in age.
Autoencoders for Compression and Simulation in Particle Physics
At the ICLR 2020 "Fundamental Science in the era of AI" workshop, I presented a poster about using generative models to simulate particle identification features. Without a fundamentally more CPU-efficient simulation generation procedure like the one proposed, particle physics experiments like LHCb will not be able to benefit sufficiently from the 100-fold increase in data rate. Research conducted after this presentation showed that normalizing flow based architectures, especially those based on Unconstrained Monotonic Neural Networks, yield even more high-fidelity simulation samples.
For video click here
Dark Photons at LHCb
One of the largest open questions in particle physics is finding out about the nature of Dark Matter - an unknown substance that constitutes 85% of the matter of our universe and is needed to explain galaxy dynamics amongst other phenomena. The dark photon model has become increasingly popular as a steppinng stone towards a more complete understanding. We test this model with the LHCb experiment, set world-leading limits, and help shed light on this mystery of dark matter.