Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks

2022-11-16 12:30 - 13:30
CMSA Room G10
Address: CMSA, 20 Garden Street, Cambridge, MA 02138 USA

Colloquium

Speaker: Hidenori Tanaka (NTT Research at Harvard)

Title: Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks

Abstract: In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle, but the role of symmetry breaking is not well understood. Here, we develop a Lagrangian formulation to study the geometry of learning dynamics in neural networks and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. Then, we generalize Noether’s theorem known in physics to describe a unique symmetry breaking mechanism in learning and derive the resulting motion of the Noether charge: Noether’s Learning Dynamics (NLD). Finally, we apply NLD to neural networks with normalization layers and discuss practical insights. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.