From Anisotropy to Anomaly: Online Geometric Diagnostics during Transformer Training

We propose an online monitor for Transformer training that combines latent-space statistical correlation with geometric anisotropy to jointly characterize geometric drift and representation anomalies. The framework enables real-time perturbation detection across multiple scales — NanoGPT, ViT, and Pythia-2.8B — and provides early-warning signals during pre-training. Collaboration with Huawei.

Authors: Zhengyang Hu, Wenyi Fang, Yang Zheng, Yanchao Yang.