CMSA

Toolformer: Language Models Can Teach Themselves to Use Tools
2023-04-26 14:00 - 15:00 (America/New_York) | Virtual
New Technologies in Mathematics Seminar
Speaker: Timo Schick, Meta AI
Title: Toolformer: Language Models Can Teach Themselves to Use Tools
Abstract: Language models exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel. In this talk, we show how these limitations can be overcome by letting language models teach themselves to use external tools via simple APIs. We discuss Toolformer, a model trained to independently decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. Through this, it achieves substantially improved zero-shot performance across a variety of downstream tasks without sacrificing its core language modeling abilities.

Toroidal Positive Mass Theorem
2023-05-02 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Aghil Alaee
Title: Toroidal Positive Mass Theorem
Abstract: In this talk, we review the positive mass conjecture in general relativity and prove a toroidal version of this conjecture in an asymptotically hyperbolic setting.

Generative Adversarial Networks (GANs): An Analytical Perspective
2023-05-03 12:30 - 13:30 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Speaker: Xin Guo, UC Berkeley
Title: Generative Adversarial Networks (GANs): An Analytical Perspective
Abstract: Generative models have attracted intense interests recently. In this talk, I will discuss one class of generative models, Generative Adversarial Networks (GANs). I will first provide a gentle review of the mathematical framework behind GANs. I will then proceed to discuss a few challenges in GANs training from an analytical perspective. I will finally report some recent progress for GANs training in terms of its stability and convergence analysis.

Modern Hopfield Networks for Novel Transformer Architectures
2023-05-10 14:00 - 15:00 | Virtual
New Technologies in Mathematics Seminar
Speaker: Dmitry Krotov, IBM Research – Cambridge
Title: Modern Hopfield Networks for Novel Transformer Architectures
Abstract: Modern Hopfield Networks or Dense Associative Memories are recurrent neural networks with fixed point attractor states that are described by an energy function. In contrast to conventional Hopfield Networks, which were popular in the 1980s, their modern versions have a very large memory storage capacity, which makes them appealing tools for many problems in machine learning and cognitive and neurosciences. In this talk, I will introduce an intuition and a mathematical formulation of this class of models and will give examples of problems in AI that can be tackled using these new ideas. Particularly, I will introduce an architecture called Energy Transformer, which replaces the conventional attention mechanism with a recurrent Dense Associative Memory model. I will explain the theoretical principles behind this architectural choice and show promising empirical results on challenging computer vision and graph network tasks.

Quantum information and extended topological quantum field theory
2023-05-15 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Gabriel Wong
Title: Quantum information and extended topological quantum field theory
Abstract: Recently, ideas from quantum information theory have played an important role in condensed matter and quantum gravity research. Most of these applications focus on the entanglement structure of quantum states, and the computation of entanglement measures such as entanglement entropy has been an essential part of the story. In this talk, we will address some subtleties that arise when trying to define entanglement entropy in quantum field theory and quantum gravity. In particular, we will explain why extended topological field theory provides a useful framework to define and compute entanglement entropy in a continuous system. Time permitting, we will explain some recent applications of these ideas in low dimensional quantum gravity and to topological string theory.

Quantum UV-IR map and curve counts in skeins
2023-09-15 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Sunghyuk Park
Title: Quantum UV-IR map and curve counts in skeins
Abstract: Quantum UV-IR map (a.k.a. q-nonabelianization map), introduced by Neitzke and Yan, is a map from UV line defects in a 4d N=2 theory of class S to those of the IR. Mathematically, it can be described as a map between skein modules and is a close cousin of quantum trace map of Bonahon and Wong.
In this talk, I will discuss how quantum UV-IR map can be generalized to a map between HOMFLYPT skein modules, using skein-valued curve counts of Ekholm and Shende.

CMSA Q and A Seminar 9/19/2023
2023-09-19 12:30 - 13:30 | Common Room, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Speakers: Dan Freed (Harvard Math and CMSA), Denis Auroux (Harvard Math)

The TinyStories Dataset: How Small Can Language Models Be And Still Speak Coherent
2023-09-20 14:00 - 15:00 | Virtual
New Technologies in Mathematics Seminar
Speaker: Ronen Eldan, Microsoft Research
Title: The TinyStories Dataset: How Small Can Language Models Be And Still Speak Coherent
Abstract: While generative language models exhibit powerful capabilities at large scale, when either the model or the number of training steps is too small, they struggle to produce coherent and fluent text: Existing models whose size is below a few billion parameters often do not generate coherent text beyond a few sentences. Hypothesizing that one of the main reasons for the strong reliance on size is the vast breadth and abundance of patterns in the datasets used to train those models, this motivates the following question: Can we design a dataset that preserves the essential elements of natural language, such as grammar, vocabulary, facts, and reasoning, but that is much smaller and more refined in terms of its breadth and diversity?
In this talk, we introduce TinyStories, a synthetic dataset of short stories that only contain words that 3 to 4-year-olds typically understand, generated by GPT-3.5/4. We show that TinyStories can be used to train and analyze language models that are much smaller than the state-of-the-art models (below 10 million parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate certain reasoning capabilities. We also show that the trained models are substantially more interpretable than larger ones, as we can visualize and analyze the attention and activation patterns of the models, and show how they relate to the generation process and the story content. We hope that TinyStories can facilitate the development, analysis and research of language models, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.

Modularity of Landau-Ginzburg Models
2023-09-22 12:00 - 13:00 | Common Room, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Chuck Doran
Title: Modularity of Landau-Ginzburg Models
Abstract: Fano varieties are the basic building blocks of algebraic varieties. Smooth Fano varieties have been classified in dimensions one (the projective line), two (del Pezzo surfaces), and three (Mori-Mukai classification). What does Mirror Symmetry have to say about such classifications? By studying the Landau-Ginzburg models mirror to smooth Fano threefolds we can transform the Mori-Mukai classification into an effective uniruledness result for moduli spaces of certain K3 and abelian surfaces. This is joint work with Andrew Harder, Ludmil Katzarkov, Mikhail Ovcharenko, and Victor Przjalkowski (arXiv:2307.15607).

CMSA Q and A Seminar 9/26/2023
2023-09-26 12:30 - 13:30 | Common Room, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
CMSA Q and A Seminar
Speakers: Michael Douglas (CMSA) and Mayuko Yamashita (Kyoto University)
Topics:
Michael Douglas: “What is non-commutative field theory?”

Mayuko Yamashita: “What is differential cohomology?”

Transformers for maths, and maths for transformers
2023-09-27 14:00 - 15:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
New Technologies in Mathematics Seminar
Speaker: François Charton, Meta AI
Title: Transformers for maths, and maths for transformers
Abstract: Transformers can be trained to solve problems of mathematics. I present two recent applications, in mathematics and physics: predicting integer sequences, and discovering the properties of scattering amplitudes in a close relative of Quantum ChromoDynamics.
Problems of mathematics can also help understand transformers. Using two examples from linear algebra and integer arithmetic, I show that model predictions can be explained, that trained models do not confabulate, and that carefully choosing the training distributions can help achieve better, and more robust, performance.

Moduli of vector bundles on curve and semiorthogonal decomposition
2023-09-29 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Kai Xu (CMSA)
Title: Moduli of vector bundles on curve and semiorthogonal decomposition
Abstract: In this talk we construct semiorthogonal decompositions of moduli of vector bundles on a curve into its symmetric powers. The essential ingredients in the proof include Borel-Weil-Bott theory for loop groups, derived Schur-Weyl duality for current groups and derived Θ-stratification.

Gravitational Instantons
2023-10-02 16:30 - 17:30 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Speaker: Yu-Shen Lin (Boston University)
Title: Gravitational Instantons
Abstract: Gravitational instantons were introduced by Hawking as building blocks of his Euclidean quantum gravity theory back in the 1970s. These are non-compact Calabi-Yau surfaces with L2 curvature and thus can be viewed as the non-compact analogue of K3 surfaces. K3 surfaces are 2-dimensional Calabi-Yau manifolds and are usually the testing stone before conquering the general Calabi-Yau problems. The moduli space of K3 surfaces and its compactification on their own form important problems in various branches in geometry. In this talk, we will discuss the Torelli theorem of gravitational instantons, how the cohomological invariants of a gravitational instanton determine them. As a consequence, this leads to a description of the moduli space of gravitational instantons.

CMSA Q and A Seminar 10/3/2023
2023-10-03 12:30 - 13:30 | Common Room, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
CMSA Q and A Seminar
Speakers: Dan Freed (Harvard Math & CMSA) and Dan Berwick-Evans (University of Illinois at Urbana-Champaign)

Topics:
Dan Freed: What is framing anomaly? How is it different from other anomalies?
Dan Berwick-Evans: What is Atiyah-Singer index theorem?

Random matrices and large deviations
2023-10-06 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Benjamin McKenna
Title: Random matrices and large deviations
Abstract: We give a generalist overview of random matrices and their (a)typical behaviors. In recent years, classical results have been complemented by a variety of new ones, in both the math and physics literatures, whose proofs leverage connections with special integrals over matrix groups. Some of these models exhibit interesting transition points, whose motivating relationships to eigenvector (de)localization are not yet fully understood. Based on joint work with Jonathan Husson.

CMSA Q and A Seminar 10/10/23
2023-10-10 12:30 - 13:30 | Common Room, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
CMSA Q and A Seminar
Speakers: Dan Freed (Harvard Math and CMSA) and Sunghyuk Park (CMSA)
Topics:
Dan Freed: What is Dijkgraaf-Witten theory?
Sunghyuk Park: What happened at the Clay Math Institute Workshop?

LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
2023-10-11 14:00 - 15:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
New Technologies in Mathematics Seminar
Speaker: Alex Gu, MIT Dept. of EE&CS
Title: LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
Abstract: Large language models (LLMs) have shown promise in proving formal theorems using proof assistants such as Lean. However, existing methods are difficult to reproduce or build on, due to private code, data, and large compute requirements. This has created substantial barriers to research on machine learning methods for theorem proving. We introduce LeanDojo: an open-source Lean playground consisting of toolkits, data, models, and benchmarks. LeanDojo extracts data from Lean and enables interaction with the proof environment programmatically. It contains fine-grained annotations of premises in proofs, providing valuable data for premise selection: a key bottleneck in theorem proving. Using this data, we develop ReProver (Retrieval-Augmented Prover): the first LLM-based prover that is augmented with retrieval for selecting premises from a vast math library. It is inexpensive and needs only one GPU week of training. Our retriever leverages LeanDojo’s program analysis capability to identify accessible premises and hard negative examples, which makes retrieval much more effective. Furthermore, we construct a new benchmark consisting of 96,962 theorems and proofs extracted from Lean’s math library. It features a challenging data split requiring the prover to generalize to theorems relying on novel premises that are never used in training. We use this benchmark for training and evaluation, and experimental results demonstrate the effectiveness of ReProver over non-retrieval baselines and GPT-4. We thus provide the first set of open-source LLM-based theorem provers without any proprietary datasets and release it under a permissive MIT license to facilitate further research.

On the Breakdown of Einstein's Gravity
2023-10-13 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Puskar Mondal (CMSA)
Title: On the Breakdown of Einstein’s Gravity
Abstract: It is important to understand under which conditions, the solutions of non-linear hyperbolic PDEs break down in finite time. In the context of Einstein’s gravity, this is very closely tied to naked singularity formation and Penrose’s weak cosmic censorship conjecture. In this talk, I will give sharp estimates on the relevant geometric entities that allow one to continue the solutions of Einstein’s equations indefinitely in the future in a ‘time’ direction without forming a naked singularity.

An exploration of infinite games—infinite Wordle and the Mastermind numbers
2023-10-16 16:30 - 17:30 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Speaker: Joel D. Hamkins (Notre Dame and Oxford)
Title: An exploration of infinite games—infinite Wordle and the Mastermind numbers
Abstract: Let us explore the nature of strategic reasoning in infinite games, focusing on the cases of infinite Wordle and infinite Mastermind. The familiar game of Wordle extends naturally to longer words or even infinite words in an idealized language, and Mastermind similarly has natural infinitary analogues. What is the nature of play in these infinite games? Can the codebreaker play so as to win always at a finite stage of play? The analysis emerges gradually, and in the talk I shall begin slowly with some easy elementary observations. By the end, however, we shall engage with sophisticated ideas in descriptive set theory, a kind of infinitary information theory. Some assertions about the minimal size of winning sets of guesses, for example, turn out to be independent of the Zermelo-Fraenkel ZFC axioms of set theory. Some questions remain open.

Physics of Language Models: Knowledge Storage, Extraction, and Manipulation
2023-10-18 14:00 - 15:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
New Technologies in Mathematics Seminar
Speaker: Yuanzhi Li, CMU Dept. of Machine Learning and Microsoft Research
Title: Physics of Language Models: Knowledge Storage, Extraction, and Manipulation
Abstract: Large language models (LLMs) can memorize a massive amount of knowledge during pre-training, but can they effectively use this knowledge at inference time? In this work, we show several striking results about this question. Using a synthetic biography dataset, we first show that even if an LLM achieves zero training loss when pretraining on the biography dataset, it sometimes can not be finetuned to answer questions as simple as “What is the birthday of XXX” at all. We show that sufficient data augmentation during pre-training, such as rewriting the same biography multiple times or simply using the person’s full name in every sentence, can mitigate this issue. Using linear probing, we unravel that such augmentation forces the model to store knowledge about a person in the token embeddings of their name rather than other locations.
We then show that LLMs are very bad at manipulating knowledge they learn during pre-training unless a chain of thought is used at inference time. We pretrained an LLM on the synthetic biography dataset, so that it could answer “What is the birthday of XXX” with 100% accuracy. Even so, it could not be further fine-tuned to answer questions like “Is the birthday of XXX even or odd?” directly. Even using Chain of Thought training data only helps the model answer such questions in a CoT manner, not directly.
We will also discuss preliminary progress on understanding the scaling law of how large a language model needs to be to store X pieces of knowledge and extract them efficiently. For example, is a 1B parameter language model enough to store all the knowledge of a middle school student?

Black Holes as Quantum Systems
2023-10-20 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Daniel Kapec (CMSA)
Title: Black Holes as Quantum Systems
Abstract: To an outside observer, a black hole appears to be an ordinary quantum mechanical system with finite entropy and highly chaotic internal dynamics. Nevertheless, the low-temperature thermodynamics of the Kerr black hole presents several puzzles. For instance, the leading order semiclassical approximation to the black hole density of states predicts a surprisingly large ground state degeneracy, while poorly understood quantum corrections are known to become increasingly important at low temperatures. I will review the modern picture of black holes as quantum systems and then discuss a recent result on the leading correction to the low-temperature thermodynamics of the Kerr black hole that resolves many of the old puzzles.

On Provable Copyright Protection for Generative Model
2023-10-23 16:30 - 17:30 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Speaker: Boaz Barak (Harvard)
Title: On Provable Copyright Protection for Generative Model
Abstract: There is a growing concern that learned conditional generative models may output samples that are substantially similar to some copyrighted data C that was in their training set. We give a formal definition of near access-freeness (NAF) and prove bounds on the probability that a model satisfying this definition outputs a sample similar to C, even if C is included in its training set.
Roughly speaking, a generative model p is k-NAF if for every potentially copyrighted data C, the output of p diverges by at most k-bits from the output of a model q that did not access C at all. We also give generative model learning algorithms, which efficiently modify the original generative model learning algorithm in a black box manner, that output generative models with strong bounds on the probability of sampling protected content. Furthermore, we provide promising experiments for both language (transformers) and image (diffusion) generative models, showing minimal degradation in output quality while ensuring strong protections against sampling protected content.
Joint work with Nikhil Vyas and Sham Kakade. Paper appeared in ICML 2023 and is on https://arxiv.org/abs/2302.10870

CMSA Q and A Seminar 10/24/23
2023-10-24 12:30 - 13:30 | Common Room, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
CMSA Q and A Seminar
Speakers: Xi Yin (Harvard Physics) and Peter Kronheimer (Harvard Math)

Topics:
Xi Yin: What is string field theory?
Peter Kronheimer: What is Seiberg-Witten theory?

Llemma: an open language model for mathematics
2023-10-25 14:00 - 15:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
New Technologies in Mathematics Seminar
Speaker: Sean Welleck, CMU, Language Technologies Institute
Title: Llemma: an open language model for mathematics
Abstract: We present Llemma: 7 billion and 34 billion parameter language models for mathematics. The Llemma models are initialized with Code Llama weights, then trained on the Proof-Pile II, a 55 billion token dataset of mathematical web data, code, and scientific papers. The resulting models show improved mathematical capabilities, and can be adapted to various tasks. For instance, Llemma outperforms the unreleased Minerva model suite on an equi-parameter basis, and is capable of tool use and formal theorem proving without any further fine-tuning. We openly release all artifacts, including the Llemma models, the Proof-Pile II, and code to replicate our experiments. We hope that Llemma serves as a platform for new research and tools at the intersection of generative models and mathematics.

Extension of pluricanonical forms in positive and mixed characteristics
2023-10-27 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Iacopo Brivio (CMSA)
Title: Extension of pluricanonical forms in positive and mixed characteristics
Abstract: The geometry of a complex manifold $X$ is to a large extent determined by its pluricanonical forms, i.e. global sections of $(Omega^{dim X}_X)^{otimes m}$ for $mgeq 0$. A famous theorem of Siu states that when $Xto D$ is a smooth projective family of complex manifolds, then every pluricanonical form on $X_0$ extends to the whole of $X$. Both this theorem and the tools used in its proof had a deep impact in higher dimensional birational geometry and moduli theory. In this talk I am going to give an overview of the extension problem for pluricanonical forms when $D$ is the spectrum of a positive or mixed characteristic discrete valuation ring.

Homotopy categories of rings: some properties and consequences in module categories
2023-10-30 16:30 - 17:30 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Speaker: Manuel Cortés-Izurdiaga (University of Malaga)
Title: Homotopy categories of rings: some properties and consequences in module categories
Abstract: Given a non-necessarily commutative ring with unit and an additive subcategory of the category of right modules, one can consider complexes of modules in the subcategory and the corresponding homotopy category. Sometimes, these homotopy categories are the first step in studying other (algebraic) homotopy categories, such as those associated to a scheme. To study these categories, one can use results from the category of modules or the category of complexes. In the first part of the talk, we will see how some results of homotopy categories of complexes extend to homotopy categories of N-complexes, for a natural number N greater than or equal to 2, using some techniques from module categories, such us the deconstruction of a class of modules.
Another approximation is to use other methods for studying homotopy categories, like those coming from triangulated categories. In some cases, the results obtained in homotopy categories imply some consequences in the category of modules. In the second part of the talk, we will see how to prove the existence of Gorenstein-projective precovers for some specific rings using this approach.

Deformations of Landau-Ginzburg models and their fibers
2023-11-03 12:00 - 13:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Member Seminar
Speaker: Zi Yang Kang
Title: Deformations of Landau-Ginzburg models and their fibers
Abstract: In mirror symmetry, the dual object to a Fano variety is a Landau-Ginzburg model. Broadly, a Landau-Ginzburg model is quasi-projective variety Y with a superpotential function w, but not all such pairs correspond to Fano varieties under mirror symmetry, so a very natural question to ask is: Which Landau-Ginzburg models are mirror to Fano varieties? In this talk, I will discuss a cohomological characterization of mirrors of (semi-)Fano varieties, focusing on the case of threefolds. I’ll discuss how this characterization relates to the deformation and Hodge theory of (Y,w), and in particular, how the classification of (semi-)Fano threefolds is related to questions about moduli spaces of lattice polarized K3 surfaces.

Impossibility results in classical dynamical systems
2023-11-06 16:30 - 17:30 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
Speaker: Matthew Foreman (UC Irvine)
Title: Impossibility results in classical dynamical systems
Abstract: In 1932, motivated by questions in statistical and celestial mechanics, von Neumann proposed classifying the statistical behavior of dynamical systems. In the 1960’s, motivated by work of Poincaré, Smale proposed classifying the qualitative behavior of dynamical systems. These questions laid the groundwork for enormous amounts of work, but the fundamental questions remain open. This talk shows that they are impossible to answer in a rigorous sense. The talk will discuss various kinds of impossibility results and describe how they apply to von Neumann’s program and Smale’s program.

CMSA Q and A Seminar 11/7/2023
2023-11-07 12:30 - 13:30 | Common Room, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
CMSA Q and A Seminar
Speakers: Michael Hopkins (Harvard Math) and Minhea Popa (Harvard Math)
Topics:
Mike Hopkins: What are topological modular forms?
Mihnea Popa: What is the abundance conjecture and why should it to be true?

Peano: Learning Formal Mathematical Reasoning Without Human Data
2023-11-08 14:00 - 15:00 | CMSA Room G10, CMSA, 20 Garden Street, Cambridge, MA, 02138, United States
New Technologies in Mathematics Seminar
Speaker: Gabriel Poesia, Dept. of Computer Science, Stanford University
Title: Peano: Learning Formal Mathematical Reasoning Without Human Data
Abstract: Peano is a theorem proving environment in which a computational agent can start tabula rasa in a new domain, learn to solve problems through curiosity-driven exploration, and create its own higher level actions. Gabriel will describe the system, present case studies on learning to solve simple algebra problems from the Khan Academy platform, and describe work on progress on learning the Natural Number Game, a popular introduction to theorem proving in Lean for mathematicians.
