e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs

The provided text introduces "e3," a new training methodology for Large Language Models (LLMs) designed to improve their reasoning capabilities and enable extrapolation of test-time compute. This means LLMs can continue to enhance performance even when given more processing time than they were trained on. The core of e3 lies in three key components: leveraging asymmetries in LLM competence, where models are better at verifying answers than generating them; utilizing negative gradients in reinforcement learning to encourage exploration and chain these asymmetric operations; and employing a coupled curriculum that aligns task difficulty with training budget to structure this exploration effectively. Experiments demonstrate that e3 significantly boosts performance on complex mathematical reasoning tasks like AIME and HMMT, outperforming other models within its size class and showing robust scaling with increased test-time compute.keepSave to notecopy_alldocsAdd noteaudio_magic_eraserAudio OverviewflowchartMind Map