Loading Events

« All Events

  • This event has passed.

High-dimensional learning of narrow neural networks

October 4, 2024 @ 12:00 pm - 1:00 pm

Member Seminar

Speaker: Hugo Cui, CMSA

Title: High-dimensional learning of narrow neural networks

Abstract: This talk explores the interplay between neural network architectures and data structure through the lens of high-dimensional asymptotics. We focus on a class of narrow neural networks, namely networks possessing a finite number of hidden units, while operating in high dimensions. In the limit of large data dimension and comparably large number of samples, we derive a tight asymptotic characterization of the learning of these architectures. As an illustration, we discuss how this characterization enables the analysis of a solvable model of dot-product attention. We show how the latter can learn to implement either a positional attention mechanism (with tokens attending to each other based on their respective positions), or a semantic attention mechanism (with tokens attending to each other based on their meaning), and evidence a phase transition with sample complexity from positional to semantic learning.

Details

Date:
October 4, 2024
Time:
12:00 pm - 1:00 pm
Event Category:

Venue

CMSA Room G10
CMSA, 20 Garden Street
Cambridge, MA 02138 United States
+ Google Map
Phone:
6174967132