Title: Bagging regularized M-estimators: Precise asymptotics and cross-validation
Abstract: Ensemble methods can improve model stability by averaging predictions from multiple base learners. A canonical ensemble method is bagging [bootstrap aggregating] (which trains models on resampled datasets and averages their outputs), and its sibling, subagging [subsampled bootstrap aggregating] (which uses subsampled datasets). Beyond computational benefits, subagging can also improve generalization, especially in overparameterized regimes near interpolation thresholds (e.g., by smoothing out the double descent peaks). This talk will present theoretical results on subagging of regularized M-estimators under proportional asymptotics, where the sample size, feature size, and subsample sizes all grow with fixed limiting ratios.
- Risk asymptotics: Precise risk formulas for heterogeneous ensembles (each component may have a different subsample size, loss, and regularizer), derived via a provably contractible nonlinear system that captures limiting correlations between estimator errors and residuals on overlapping subsamples. {Theorem 2 of [1]}
- Optimal (oracle) tuning: In the homogeneous case (common loss, penalty, and subsample size), optimally tuning the subsample size k results in monotonic risk behavior in n, and as the number of ensembles M \to \infty, the optimal k^\star lies in the overparameterized regime (k^\star \le \min(n, p)), when bagging estimators with vanishing explicit regularization. {Proposition 7 of [1]}
- Data-dependent tuning: A corrected GCV (CGCV) procedure, adding a simple degrees-of-freedom adjustment, yields consistent risk estimates for any finite ensemble without sample splitting or refitting. {Theorem 2 of [2]}
Of independent interest, in the non-ensemble setting (M = 1), our analysis also establishes convergence of trace‐based degrees-of-freedom functionals, extending previous results for square loss and ridge, lasso regularizers. {Table 2 of [1]}
This is joint work with the following collaborators (in alphabetical order): Pierre Bellec, Jin-Hong Du, Takuya Koriyama, and Kai Tan, and will feature the following papers (in presentation order): [1] subagging asymptotics (https://pratikpatil.io/papers/subagging-asymptotics.pdf); [2] corrected generalized cross-validation (https://pratikpatil.io/papers/cgcv.pdf).
Website: https://pratikpatil.io/
