BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CMSA - ECPv6.17.1//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-ORIGINAL-URL:https://cmsa.fas.harvard.edu
X-WR-CALDESC:Events for CMSA
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20210314T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20211107T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20220313T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20221106T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20230312T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20231105T060000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20220928T140000
DTEND;TZID=America/New_York:20220928T150000
DTSTAMP:20240214T110335Z
CREATED:20230808T184138Z
LAST-MODIFIED:20240214T110335Z
UID:10001211-1664373600-1664377200@cmsa.fas.harvard.edu
SUMMARY:Statistical mechanics of neural networks: From the geometry of high dimensional error landscapes to beating power law neural scaling
DESCRIPTION:New Technologies in Mathematics \nSpeaker: Surya Ganguli\, Stanford University \n\nTitle: Statistical mechanics of neural networks: From the geometry of high dimensional error landscapes to beating power law neural scaling\n\n\n\n\nAbstract: Statistical mechanics and neural network theory have long enjoyed fruitful interactions.  We will review some of our recent work in this area and then focus on two vignettes. First we will analyze the high dimensional geometry of neural network error landscapes that happen to arise as the classical limit of a dissipative many-body quantum optimizer.  In particular\, we will be able to use the Kac-Rice formula and the replica method to calculate the number\, location\, energy levels\, and Hessian eigenspectra of all critical points of any index.  Second we will review recent work on neural power laws\, which reveal that the error of many neural networks falls off as a power law with network size or dataset size.  Such power laws have motivated significant societal investments in large scale model training and data collection efforts.  Inspired by statistical mechanics calculations\, we show both in theory and in practice how we can beat neural power law scaling with respect to dataset size\, sometimes achieving exponential scaling\, by collecting small carefully curated datasets rather than large random ones.\n\n\n\nReferences: Y. Bahri\, J. Kadmon\, J. Pennington\, S. Schoenholz\, J. Sohl-Dickstein\, and S. Ganguli\, Statistical mechanics of deep learning\, Annual Reviews of Condensed Matter Physics\, 2020.\n\nSorscher\, Ben\, Robert Geirhos\, Shashank Shekhar\, Surya Ganguli\, and Ari S. Morcos. 2022. Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning https://arxiv.org/abs/2206.14486 (NeurIPS 2022).
URL:https://cmsa.fas.harvard.edu/event/8303/
LOCATION:CMSA Room G10\, CMSA\, 20 Garden Street\, Cambridge\, MA\, 02138\, United States
CATEGORIES:New Technologies in Mathematics Seminar
ATTACH;FMTTYPE=image/png:https://cmsa.fas.harvard.edu/media/CMSA-NTM-Seminar-09.28.2022.png
END:VEVENT
END:VCALENDAR