Now for a different kind of LLM benchmark!

Of course this caught my attention: LLMs and Swarm Intelligence! According to this study by a team from Renmin University of China, LLMs differ significantly in decentralized swarm scenarios based on their ability to coordinate under strict constraints of local perception and communication.

• Task-Specific Performance: LLMs exhibit varying strengths across different swarm tasks. For example:

• Flocking generally yields the highest scores, indicating better performance in maintaining group cohesion and alignment.

• Synchronization shows greater divergence in performance, highlighting challenges in achieving consensus.

• Models like gemini-2.0-flash and o4-mini excel in spatial tasks like Pursuit and Foraging, while claude-3.7-sonnet performs well in Synchronization.

• Emergent Coordination: Some LLMs demonstrate basic coordination abilities, but struggle with robust planning and strategy formation under uncertainty. For instance:

• Models like deepseek-v3 and gpt-4.1 show moderate success in tasks requiring spatial reasoning.

• Others, such as deepseek-r1 and claude-3.5-haiku, perform poorly across most tasks, indicating limitations in adapting to decentralized constraints.