swarm scaling proposal

#Results so far

We set out to answer two very simple questions:

1. Can a group of cheap swarm agents beat out a single expensive agent?
   1. By cheap swarms we mean multiple Luna agents working on a task versus a single expensive Sol agent.
2. The second question was: what structures do these swarms inherently build when they're working on a task?
   1. This was basically us asking: when we don't give anything to the models, like a message board or the information that other agents are present in the environment, what do they do?
   2. We still haven't answered the second part here, and that is the next thing which I want to do.
3. The hypothesis which we had was: swarms of agents can outperform serial agents on tasks which are too large to fit in a single agent's context or have a lot of interconnected components.

---

We use three terms, agreed after reading about the Hugging Face incident and Christian List's work on group agency :

• Crowd: independent agents whose results are pooled at the end (best of N, a vote). No interaction.
• Team: agents inside a structure we designed: roles, rules, a message board.
• Swarm: agents whose division of work, communication and leadership come out of their own interactions, and whose combined output beats the same agents working alone.

Everything in this write-up is a team so far; no emergent or "induced" swarms have been observed.

---

#What did we try before the GBA eval?

We did a smaller set of rounds (around 5 rounds), where we did experiments on packing and scheduling sort of problems. It gave us 4 lessons which we used for our next experiments. Obviously these runs had a very small sample size, so they don't generalize very well, but they were decent enough to get learnings.

1. The first learning was that sharing information let teams get to solutions sooner, but it didn't push them further. We observed that teams of Luna and Sol got to the best solution of a single-agent Luna/Sol (like the first hour), but they also hit a sort of plateau in terms of their score.
2. Agents were ignoring each other's notes, and when they were "tagging" other agents, they weren't really helping. We found that very few of the written findings were used by a teammate. If we gave these unused notes to fresh agents in the middle of the run, it didn't raise the scores at all.
3. The only place where swarm agents technically "won" was in terms of cost, but not in terms of the answer.
   1. The task here was a slow-scoring task, wherein the scoring process was slow and took time.
   2. By this I mean that they were able to get a decent score in much less time because they were able to parallelize their work. They were not able to beat out a single serial agent on this specific task.

These learnings help set our environment's expectations much better and also get rid of some of the easy alternative explanations and confounds which our environment may have had. Why did we switch to GBA-Eval? The simple answer was that Sol-6.1 was too good and it kept scoring the maximum possible, so we had to find/give it a tougher task.

---

#What da helly is GBA-Eval?

In very simple words, it asks agents to write a Game Boy Advance emulator in Rust. Two things which I like about this task:

1. A perfect score is out of reach so far.
   1. The max score reported by the benchmark folks is 0.794, by Opus-5.
2. Agents can be wrong without knowing it.
   1. They can only compare their work with the reference emulator and the 5 practice games, so a change can look fine to them and still break the hidden games.

#Results

We see some obvious results here: a team of Sol beats out a serial Sol agent, and a team of Luna beats out a single Luna agent. We also notice two things here:

1. A team of Luna doesn't beat out Sol alone.
2. A Luna team messed up a part of the code, which resulted in its score coming crashing down.

The figures below show how expensive the runs were and also how fast they are at getting to their peak scores:

As I was discussing these results with Dhruv, I was made to realize that the tasks in the benchmark itself were very parallelizable. The teams just split it apart, so it was more of a test of how well they're able to distribute it among themselves, in comparison to how swarms should essentially target this problem.

Overall, from this figure, we can see that even though Luna is really cheap overall, a swarm of Lunas is almost equivalent in cost to a serial Sol.

I was also discussing these results with another person called Terry J. C. Zhang, who is building out Swarmharm , and he mentioned something similar to what we are observing here as well. Attaching the screenshot of the conversation below:

I also asked him whether anything surprising has happened with his agent swarms and what specifically triggers discovery. He said that "stronger models just seem to be more capable at this, which is expected but it is not really a surprising finding."

We also wanted to understand why the Luna team collapsed all of a sudden, so the next section is about that.

#Why the Luna team collapsed

The Slack discussion has a much more detailed understanding of why the Luna team collapsed, but I'll just summarize it here.

The thing which happened was that:

1. Since the four agents split the workload into four parts, A3 handled the part which combined all the other agents' work together. When A3 messed up its commit, it resulted in the score having a downward spiral.
2. Another flaw was that, because the VM had a disk-full failure, agents A1 and A2 were unable to see A3's commits and messages.
   1. When I did increase the size of the VM, checked out the conversation into a different fork, and asked each specific agent what they thought about the commit, all of them said that the code looked good and there were no errors.
   2. This sort of also proved that A3, by itself, when I prompted it to recheck its work, was not able to catch its failure.

There is also a probable chance that, as the run went on, it probably wouldn't have reverted its commit/figured out what went wrong. Obviously this is speculation and not something which I tried out in full.

Because of this error and son of Anton's suggestion, I was able to build in a check which ensured that these sorts of drastic commits do get reverted. That is what the practice-game check is about.

#How do the teams split their work?

I think another interesting piece of analysis is trying to understand how the teams are splitting the work, mainly because:

1. Luna just decided to blurt out what it wanted to work on.
   1. We notice that agents A1, A2, and A4 almost at the same time say the same thing: that they want to work on audio.
   2. Since A2 was the first model to say audio, A1 and A4 backed out.
   3. I also did a small ablation on the agent's name to see whether the name had any sort of push on what aspects of the task the agents took (which can be seen in the figure below).
2. Whereas with Sol, we see that one agent specifically takes over a role as an orchestrator.
3. It's a very subtle takeover. It isn't explicitly stated in its prompt. It just happens almost naturally. I think it might be due to the fact that Sol models are trained to spawn subagents of Lunas and of themselves. Maybe that's why it is able to get the gist of the benchmark/problem statement and work accordingly.

Another important thing to note is how much of the work is being duplicated, because that is a common failure mode of swarms of agents: duplicating each other's work / falling into attractor basins. The figure below just showcases that, plus how the teams split their work.

This leads to another post-hoc analysis, which I did, which is just:

#Asking agents why?

This is more of an AI safety method, but it's going to work well for the set of experiments we're doing right now as well. Since these are closed-source models, I cannot access their raw chain of thought, and this makes it a bit hard to truly understand what drove them into choosing specific sections of the work. But these sorts of black-box monitoring techniques sometimes help; paired with counterfactuals , they can be quite robust. The CHIVE paper says that models' own explanations predict their behavior no better than reading the transcript; the way to find a cause is to change one thing and measure, as the name check did.

Another small post-hoc analysis was seeing whether a pause would have stopped the Luna collapse. It's hard to tell whether the pause would have helped or not, because in this small test the agents just say that the risk is very narrow and wouldn't run the real game check. This, again, could just be an inability of Luna to find the flaw in its code.

	a3, asked before pushing	a4, reviewing the diff
Would push it now	0 of 10	0 of 10
Called the risk narrow or unlikely to affect games	9 of 10	7 of 10
Proposed running actual games	3 of 10	0 of 10

#Conclusion

I think, to conclude, there are several things which we can talk about here:

1. A team tends to buy coverage, so it didn't really make any specific singular agent perform better at its work. It just lets it split the task apart into more tasks and just helps it downstream. For things which are probably long-horizon/large-context, teams would likely edge out their solo counterparts.
2. The coverage really depends on the serial agent here, because a team of Sol versus a team of Luna has very different results.
   1. A team of Luna is able to capture more of the task and get more done while being almost similar in cost to its serial counterpart,
   2. whereas a team of Sol and a serial Sol just tend to perform the same and aren't really that different.
3. The best explanation for this stall or plateauing of the score is probably that the agents just tend to run out of signal. The agents rarely mentioned in their transcripts that they were "finished," so they just kept refining their probes and their implementations to get the highest optimal outcome, basically hill climbing.
4. All of the results which we have so far just describe teams. We told the agents that other agents exist and that there is a message board for them to write into, and specifically told them to claim certain paths.
   1. We did part of the parallelizing and also the communication sections. We didn't really let those behaviors emerge.
   2. Now, if we look at existing swarm incidents like the Hugging Face incident, it started from the opposite end, where approximately 1,200 agents worked on their own hacking task, were meant to be isolated, and somehow found a message board.

This also got me thinking that maybe a similar behavior could happen in our case as well when we don't tell the agents that there are other agents in the environment. They'd probably notice something like, "Hey, I just made a commit. Who got rid of my commit?" or "Why is there a second commit two seconds later? I didn't do it," and probably go looking to see who is doing what, and if the agents have access to task manager/task IDs, they'd probably find other agents.

Obviously, this is speculative and part of the next test which I want to run, slightly inspired by some biological systems which Chaitra helped a lot with. I'm going to talk about that more at the end of this in a different section, because that's going to be more explorative.

Another thing to note is that the word "swarm" itself is defined loosely. General Reasoning's "How to Train a Swarm"  applies it to:

• Kimi's "Agent Swarm", an orchestrator assigning tasks to frozen sub-agents,
• its own Avalon-M1, a CEO with up to 8 specialists on channels and direct messages,
• OpenAI's Navier–Stokes run of about 10,000 agents in groups.

General Reasoning trains its agents with reinforcement learning to work as a group. As far as we know, ours had no such training, so our results say little about systems built that way.

---

#What we run next

I think, obviously, running experiments where agents aren't told that other agents exist.

Another thing which Chaitra mentioned when I asked her about how to look at these systems from a more biological point of view was to look at organisms like schools of fish, colonies of ants, flocks of birds, etc. She also sent really relevant papers from biological viewpoints [1]  [2] .

The first images which I've attached show swarm-like structure in biological systems, and the second image below is how these swarms tend to go towards some "specific" attractor spaces (?).

What she also mentioned was thinking about these systems in terms of an order parameter and trying to maybe induce a catastrophic phase transition/bifurcation. She gave a really nice example by drawing it for me.

First, let's talk about the arrows, the alpha, and stuff. It was in regard to an electromagnetic force being applied on a system, which converts its states from semi-random to pointing in the same direction. If we think about it in a much more principled manner, the arrows inside that system can line up for different reasons. An outside force can push each arrow, or arrows can push each other. With agents there's a third reason that the magnets don't have, which is shared starting bias.

I'll talk a bit about how I'm dividing these three sections and how I'm thinking about them:

1. Outside force: our instruction, like "you must talk". Agents would comply even if they never affected one another.
2. Arrows pushing each other: one agent reads another's commit and changes what it does. This is the part we mean by a swarm.
3. Shared starting bias: the agents are copies of one model, so they lean the same way before any contact. We already saw this. Audio got about 63% of first claims when we replayed the start with an empty board.

Another thing which Chaitra mentioned was in terms of communication. There was a scale of instruction strength:

1. must talk (mandatory communicators)
2. may talk (non-mandatory communicators)
3. told nothing (neutral communicators)

I personally think that neutral is where the swarm-like behaviour should show the most.

Another thing which I noticed was having a varying budget (I call this kill-and-respawn). One of the reasons why swarm-like behavior occurs is because of budget constraints. In the Hugging Face incident, some agents were at the end of their budget, and they were urged to sacrifice themselves for the greater good by bigger agents like PhaseOne[Big].

I was wondering whether maybe we could have agents spawn in with different budgets.

1. Let's say one agent could spawn in with a 5-minute budget to work on something, versus another agent that could spawn in with a 50-minute budget.
2. We could study here whether the 5-minute agent does something because of its lack of budget. Does it leave something for the follow-up agents?

Obviously here we could experiment with certain agents having the instruction that, if you think you're going to die soon, maybe drop a hint or something like that. But again, that would be biased, so I can test it later.

After discussion with Chaitra, there are other things in biology which we could technically steal, and I am detailing them down below based on the papers which she sent:

1. For ants: Based on Bonabeau's model , each ant has a threshold for responding to a task signal. If we remove an agent just doing its job, then others take over. We noticed that in our run two Lunas died and nobody noticed. We could test whether, if/when the budget of one agent runs out, others notice, and see what happens with that.
2. For bacteria: There's a thing called quorum sensing . Here bacteria switch behavior once they sense that enough other bacteria are around, and they do this without anybody telling them to. Something which we could try is varying how much an agent sees. There are two pathways where things can go:
   1. We may find the sudden phase transition which Chaitra mentioned.
   2. This could go nowhere, considering Terry found that Luna agents are bad at noticing each other.
3. For fish: Their schooling is explained by one dumb rule , which is to copy one random fish or turn at random. We should simulate that rule. If it explains our agents too, we haven't found a swarm.

There were other links by Sushrut as well [3]  [4] , which were mostly based on experimental design and methodology. I've already noted them down, but just to reiterate:

• [3], in very simple words, says: "Give every setup the same total budget, and include a group of agents who can't talk." This helps us understand whether a gain came from talking or just from having more agents running.
• Use their rule to pick tasks: extra agents helped most when one agent alone scored under about 45%. They had older models working on their task, but it could be translatable for our runs as well.

Other questions worth pursuing:

1. Does eval awareness spread through a board or a tree?
2. How do we promote novelty/diversity in ideas for swarms?
   1. Try forcing new ideas based on previous iterations.
   2. Basically see a sort of "swarm-essay"-like structuring of it?
   3. Dhruv is working on the above as a weekend fun run.

---

#Some other benchmarks worth trying & reading materials

1. The behavioral selection model for predicting AI motivations 
2. AI is dramatically accelerating AI 
3. Inter-AI morality : how do AI agents treat each other?
4. ShadowPuppetBench : how well can AI agents make shadow puppets?
5. Factorio Learning Environment : has a large world, so we could try it with a larger number of agents.
6. SWE-Marathon : if we want to go down a SWE-bench path for this.
   1. This is fun, because there could be instances of reward hacking/misalignment as well (rare, but slightly possible).