Speaker: Oh-Ran Kwon
Title: Semi-supervised Learning for Heterogenous Data
Abstract: Many contemporary prediction problems have two prominent features: (1) data are heterogeneous and (2) unlabeled data are abundant, while labeled data are difficult to obtain. This talk introduces an approach that tackles the challenge of heterogeneity by leveraging the opportunity presented by abundant unlabeled data. The Mixture of Experts (MoE) model is a versatile framework for predictive modeling that has gained renewed interest in the age of large language models. A collection of predictive "experts" is learned along with a "gating function" that controls how much influence each expert is given when a prediction is made. This structure allows relatively simple models to excel in complex, heterogeneous data settings.
This talk begins by introducing an oceanography application problem where data exemplify these two prominent features. Oceanographers have widely used a self-developed algorithm without a clear understanding of its implied data-generating model. We specify the model implicitly assumed by this algorithm, which turns out to be based on the MoE framework. However, this model makes the strong assumption that the latent clustering structure in unlabeled data maps directly to the influence that the gating function assigns to each expert in the supervised task. We relax this assumption, imagining a noisy connection between the two, which gives rise to a novel method for semi-supervised learning of MoE models. The proposed algorithm is based on least trimmed squares, which succeeds even in the presence of misaligned data. Our theoretical analysis characterizes the conditions under which our approach yields estimators with a parametric rate of convergence. Simulated and real data examples demonstrate the method's efficacy.
