Understanding how Large Language Models (LLMs) "think" remains one of AI's most significant challenges. While explicit Chain-of-Thought (CoT) provides transparency, modern latent reasoning methods such as CODI and COCONUT face a fundamental interpretability problem. They maintain multiple, superimposed candidate traces within a hidden representation space, making it difficult to track how reasoning evolves over time.
Dynamical Systems Approach
According to recent research published on ArXiv, scientists are proposing to model these latent token sequences as trajectories within a representation space. Rather than examining isolated steps, they apply dynamical systems analysis to characterize the evolution of the model's logic. The study utilized quantitative measures, including Lyapunov sensitivity, direction consistency, and step-to-step changes, alongside qualitative projections like UMAP and DMD/PHATE.
Stable Attractors vs. Unstable Systems
The analysis revealed that latent CoT reasoning is not random but exhibits structured dynamics falling into two distinct stability classes:
- CODI: Functions as a "stable attractor," gravitating toward specific equilibrium states.
- COCONUT: Behaves as an "unstable expanding system."
Furthermore, the research found that SIM-CoT supervision tightens both behaviors without altering their underlying dynamics. This framework provides actionable insights into the interpretability of latent reasoning and offers a path toward improving the performance of these complex AI systems.
