Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks

11/16/2022 12:30 pm - 1:30 pm
Alejandro Poveda Faidra Monachou Jie Wang Michael Simkin
CMSA Room G10
Address: CMSA, 20 Garden Street, Cambridge, MA 02138 USA

Colloquium

Speaker: Hidenori Tanaka (NTT Research at Harvard)

Title: Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks

Abstract: In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle, but the role of symmetry breaking is not well understood. Here, we develop a Lagrangian formulation to study the geometry of learning dynamics in neural networks and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. Then, we generalize Noether’s theorem known in physics to describe a unique symmetry breaking mechanism in learning and derive the resulting motion of the Noether charge: Noether’s Learning Dynamics (NLD). Finally, we apply NLD to neural networks with normalization layers and discuss practical insights. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.