SwarmEvolve: Latent Agent Swarms for Evolutionary Discovery and Optimization
AI-assisted program search is becoming a practical way to explore mathematical constructions, algorithms, and numerical methods. The basic idea is evolutionary: start with a program, ask an AI system to modify or combine it, evaluate the result, and repeat. The challenge is that language-model agents often spend more time explaining their proposed changes in text than producing useful candidates.
In SwarmEvolve: Latent Agent Swarms for Evolutionary Discovery and Optimization, Qile Jiang and George Em Karniadakis propose a different design. Instead of asking a small number of agents to generate long textual reasoning traces, their framework runs hundreds of low-cost agents in latent space. The agents communicate through transformer key-value caches, or KV caches, and only a final summarizer decodes a textual program.
The problem with text-heavy evolutionary search
LLM-guided evolutionary systems need both exploration and refinement. If every agent repeatedly edits the same strong candidate, the search may converge prematurely. If the system explores too broadly, it may waste its budget on invalid or redundant programs. Existing systems also pay a substantial inference cost when each agent writes a chain of thought, receives feedback, and communicates through increasingly long prompts.
The authors frame SwarmEvolve as an attempt to increase the number and diversity of proposals evaluated within a fixed time budget. The method is designed for problems with an automatic evaluator, such as packing, combinatorial optimization, mathematical inequalities, and scientific-computing tasks with measurable numerical error.
How SwarmEvolve works
Each evolutionary round creates 10 swarms containing 45 latent agents each. The swarms have different jobs: refining one of the best archived programs, hybridizing several promising programs, exploring less-used directions, or starting from the initial solution. Agents also receive varied focus prompts, such as questioning assumptions, examining numerical parameters, or considering a textbook method.
Each agent reads the problem, its focus, and an assigned parent program, then performs 40 latent reasoning steps. Rather than sampling tokens, the model feeds a transformed hidden state back as the next input embedding. This produces a sequence of internal states without decoding a visible reasoning passage.
SwarmEvolve then applies a novelty filter. It represents each agent by the average of its hidden states and selects the 30 most diverse agents in each swarm using cosine distance and greedy farthest-point selection. The retained agents’ KV caches are concatenated and passed to a summarizer. The summarizer reads these latent traces, combines ideas from different parents, and writes one complete program. New programs are evaluated, checked for duplicates and syntax problems, and added to an archive. A separate parameter-tuning agent can also propose small numerical edits.
What the experiments found
The authors evaluated SwarmEvolve on 21 problems spanning packing, geometry, analysis, combinatorics, and scientific computing. They compared it with OpenEvolve, ShinkaEvolve, and CodeEvolve using Qwen3 8B, 14B, and 32B-FP8 models. Each run lasted 60 minutes on one GPU, used eight CPU cores for evaluation, and was repeated with three seeds per problem and model.
SwarmEvolve achieved the highest average score for all three model sizes. The reported averages were 0.804 with Qwen3 8B, 0.837 with Qwen3 14B, and 0.869 with Qwen3 32B-FP8, ahead of the corresponding baseline averages. The authors also report that the 8B version outperformed the baselines running the larger 32B model on average.
The throughput difference was substantial in the reported setup. SwarmEvolve evaluated an average of 734 programs in 60 minutes with Qwen3 8B, compared with 158 for OpenEvolve, 99 for ShinkaEvolve, and 81 for CodeEvolve. Depending on the baseline, SwarmEvolve reached that baseline’s 60-minute score in a median of roughly 3 to 8 minutes.
The results were not limited to a higher number of candidates. With Qwen3 32B-FP8, the method solved all five sparse linear-system tasks to full accuracy in the authors’ benchmark, which no baseline run achieved. With Qwen3 8B, it also improved on AlphaEvolve’s published constructions on three problems, including a 26-circle packing result and two autocorrelation inequalities. These are benchmark outcomes, not evidence that the system independently establishes new mathematical theory.
Why latent communication matters
The ablation study supports the central engineering choice. Replacing 40 latent steps with text reasoning produced lower average scores: 0.796 with up to 40 text tokens and 0.767 with up to 256 tokens, compared with 0.804 for latent reasoning. The authors report that a round’s latent reasoning took about four seconds, while 256-token text reasoning took about 32 seconds before the summarizer’s additional encoding cost.
These findings suggest that latent processing helps mainly by reallocating compute. More of the fixed time budget goes toward generating and evaluating candidate programs rather than producing intermediate prose. The latent hidden states also provide a direct numerical representation for novelty filtering, avoiding a separate embedding model and judge for that step.
Limitations and practical implications
Several constraints temper the claims. The experiments use one model family for both agents and summarizers, because direct KV-cache sharing requires compatible internal representations. The implementation also depends on standard attention; models based on linear attention or state-space mechanisms would need another way to merge agent states. KV-cache memory grows with the number of agents, latent steps, and model layers, making larger swarms or models hardware-intensive.
There is also a broader methodological limitation: performance depends on the quality of the evaluator. SwarmEvolve is most applicable when candidate programs can be tested automatically and cheaply enough to support repeated search. High benchmark scores do not by themselves guarantee robustness outside the evaluator’s cases, mathematical validity beyond the encoded objective, or useful scientific explanations.
For practitioners, the paper’s clearest lesson is to treat agent scaling as an inference-systems problem, not only a prompting problem. If a task has a reliable evaluator, replacing verbose intermediate communication with latent computation may allow substantially more search per GPU-hour. The authors’ results make SwarmEvolve a promising design for numerical optimization and scientific code discovery, while leaving open how well latent swarms transfer across models, evaluators, and open-ended research problems.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.