Seminar Series: Yingbin Liang

October 17, 2024

3:00PM - 4:00PM

EA170

Add to Calendar 2024-10-17 15:00:00 2024-10-17 16:00:00 Seminar Series: Yingbin Liang Speaker: Yingbin Liang, Department of Electrical and Computer Engineering, OSUTitle: Demystifying In-Context Learning of Transformers: A Theoretical PerspectiveAbstract: Transformers, as foundational models, have recently revolutionized many machine learning (ML) applications, including natural language processing, computer vision, and robotics. A particularly striking discovery is their remarkable in-context learning capability, where models can capture new, unseen tasks by leveraging task-specific prompts without the need for further parameter fine-tuning. This intriguing empirical finding naturally leads to a compelling theoretical question: Why can transformers, trained through the standard practice of gradient descent, acquire such in-context learning abilities?In this talk, Liang will present their recent work on characterizing the training dynamics of a one-layer transformer designed to in-context learn regression tasks, under both linear regression data and data with feature representations. Liang will then discuss the optimality of attention models upon convergence, along with their ability to generalize to unseen tasks and data samples, which explains the in-context learning capability of transformers. Additionally, Liang will discuss the analytical techniques they have developed, which may prove useful for a broader range of problems. Liang will conclude with some thoughts on potential future research directions.Short Bio: Dr. Yingbin Liang is currently a Professor at the Department of Electrical and Computer Engineering at the Ohio State University (OSU), and a core faculty of the Ohio State Translational Data Analytics Institute (TDAI). She also serves as the Deputy Director of the AI-EDGE Institute at OSU. Dr. Liang received the Ph.D. degree in Electrical Engineering from the University of Illinois at Urbana-Champaign in 2005, and served on the faculty of University of Hawaii and Syracuse University before she joined OSU. Dr. Liang's research interests include machine learning, optimization, information theory, and statistical signal processing. Dr. Liang received the National Science Foundation CAREER Award and the State of Hawaii Governor Innovation Award in 2009. She also received EURASIP Best Paper Award in 2014. She is an IEEE fellow. EA170 OSU ASC Drupal 8 ascwebservices@osu.edu America/New_York public

2024-10-17 15:00:00 2024-10-17 16:00:00 Seminar Series: Yingbin Liang Speaker: Yingbin Liang, Department of Electrical and Computer Engineering, OSUTitle: Demystifying In-Context Learning of Transformers: A Theoretical PerspectiveAbstract: Transformers, as foundational models, have recently revolutionized many machine learning (ML) applications, including natural language processing, computer vision, and robotics. A particularly striking discovery is their remarkable in-context learning capability, where models can capture new, unseen tasks by leveraging task-specific prompts without the need for further parameter fine-tuning. This intriguing empirical finding naturally leads to a compelling theoretical question: Why can transformers, trained through the standard practice of gradient descent, acquire such in-context learning abilities?In this talk, Liang will present their recent work on characterizing the training dynamics of a one-layer transformer designed to in-context learn regression tasks, under both linear regression data and data with feature representations. Liang will then discuss the optimality of attention models upon convergence, along with their ability to generalize to unseen tasks and data samples, which explains the in-context learning capability of transformers. Additionally, Liang will discuss the analytical techniques they have developed, which may prove useful for a broader range of problems. Liang will conclude with some thoughts on potential future research directions.Short Bio: Dr. Yingbin Liang is currently a Professor at the Department of Electrical and Computer Engineering at the Ohio State University (OSU), and a core faculty of the Ohio State Translational Data Analytics Institute (TDAI). She also serves as the Deputy Director of the AI-EDGE Institute at OSU. Dr. Liang received the Ph.D. degree in Electrical Engineering from the University of Illinois at Urbana-Champaign in 2005, and served on the faculty of University of Hawaii and Syracuse University before she joined OSU. Dr. Liang's research interests include machine learning, optimization, information theory, and statistical signal processing. Dr. Liang received the National Science Foundation CAREER Award and the State of Hawaii Governor Innovation Award in 2009. She also received EURASIP Best Paper Award in 2014. She is an IEEE fellow.  EA170 Department of Statistics stat@osu.edu America/New_York public

Speaker: Yingbin Liang, Department of Electrical and Computer Engineering, OSU

Title: Demystifying In-Context Learning of Transformers: A Theoretical Perspective

Abstract: Transformers, as foundational models, have recently revolutionized many machine learning (ML) applications, including natural language processing, computer vision, and robotics. A particularly striking discovery is their remarkable in-context learning capability, where models can capture new, unseen tasks by leveraging task-specific prompts without the need for further parameter fine-tuning. This intriguing empirical finding naturally leads to a compelling theoretical question: Why can transformers, trained through the standard practice of gradient descent, acquire such in-context learning abilities?

In this talk, Liang will present their recent work on characterizing the training dynamics of a one-layer transformer designed to in-context learn regression tasks, under both linear regression data and data with feature representations. Liang will then discuss the optimality of attention models upon convergence, along with their ability to generalize to unseen tasks and data samples, which explains the in-context learning capability of transformers. Additionally, Liang will discuss the analytical techniques they have developed, which may prove useful for a broader range of problems. Liang will conclude with some thoughts on potential future research directions.

Short Bio: Dr. Yingbin Liang is currently a Professor at the Department of Electrical and Computer Engineering at the Ohio State University (OSU), and a core faculty of the Ohio State Translational Data Analytics Institute (TDAI). She also serves as the Deputy Director of the AI-EDGE Institute at OSU. Dr. Liang received the Ph.D. degree in Electrical Engineering from the University of Illinois at Urbana-Champaign in 2005, and served on the faculty of University of Hawaii and Syracuse University before she joined OSU. Dr. Liang's research interests include machine learning, optimization, information theory, and statistical signal processing. Dr. Liang received the National Science Foundation CAREER Award and the State of Hawaii Governor Innovation Award in 2009. She also received EURASIP Best Paper Award in 2014. She is an IEEE fellow.