Zhenyi Shen 沈振翼

Research Highlights

I study Efficient Scaling: how algorithmic and architectural advances can push the Pareto frontier of model capability and computational cost. My goal is to make efficiency an engine for stronger reasoning and further scaling. Across attention, latent representations, and model collaboration, I ask: how can each unit of computation unlock greater capability?

• How can attention scale to longer contexts? SSA aligns sparse and full attention outputs during training to reduce their performance gap and support flexible sparsity at inference [SSA, ICML 2026].
• Can reasoning move beyond natural-language tokens? CODI uses self-distillation to compress chains of thought into continuous latent representations [CODI, EMNLP 2025]. Related work on ComprExIT extends this focus on compact information to context compression through explicit information transmission [ComprExIT, NeurIPS 2026].
• Can we separate understanding from searching? Our intuition is that understanding a problem and searching for its solution can benefit from different model capacities. PLatour is an early realization of this idea through efficient prefill–decode (PD) separation: a large model encodes the problem during prefill, and lightweight learned bridges pass its representations to a smaller model for autoregressive search and generation. This opens a path toward scaling understanding and searching independently. PACT-Check tests whether the gains depend on transferred information [PLatour, Preprint].