BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CMSA - ECPv6.16.3//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:CMSA
X-ORIGINAL-URL:https://cmsa.fas.harvard.edu
X-WR-CALDESC:Events for CMSA
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20230312T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20231105T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20240310T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20241103T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20250309T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20251102T060000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20240925T140000
DTEND;TZID=America/New_York:20240925T150000
DTSTAMP:20260725T171758
CREATED:20240907T180716Z
LAST-MODIFIED:20241002T144226Z
UID:10003454-1727272800-1727276400@cmsa.fas.harvard.edu
SUMMARY:Infinite Limits and Scaling Laws for Deep Neural Networks
DESCRIPTION:New Technologies in Mathematics Seminar \nSpeaker: Blake Bordelon \nTitle: Infinite Limits and Scaling Laws for Deep Neural Networks \nAbstract: Scaling up the size and training horizon of deep learning models has enabled breakthroughs in computer vision and natural language processing. Empirical evidence suggests that these neural network models are described by regular scaling laws where performance of finite parameter models improves as model size increases\, eventually approaching a limit described by the performance of an infinite parameter model. In this talk\, we will first examine certain infinite parameter limits of deep neural networks which preserve representation learning and then describe how quickly finite models converge to these limits. Using dynamical mean field theory methods\, we provide an asymptotic description of the learning dynamics of randomly initialized infinite width and depth networks. Next\, we will empirically investigate how close the training dynamics of finite networks are to these idealized limits. Lastly\, we will provide a theoretical model of neural scaling laws which describes how generalization depends on three computational resources: training time\, model size and data quantity. This theory allows analysis of compute optimal scaling strategies and predicts how model size and training time should be scaled together in terms of spectral properties of the limiting kernel. The theory also predicts how representation learning can improve neural scaling laws in certain regimes. For very hard tasks\, the theory predicts that representation learning can approximately double the training-time exponent compared to the static kernel limit.
URL:https://cmsa.fas.harvard.edu/event/newtech_92524/
LOCATION:CMSA Room G10\, CMSA\, 20 Garden Street\, Cambridge\, MA\, 02138\, United States
CATEGORIES:New Technologies in Mathematics Seminar
ATTACH;FMTTYPE=image/png:https://cmsa.fas.harvard.edu/media/CMSA-NTM-Seminar-9.25.24.docx-1.png
END:VEVENT
END:VCALENDAR