How Does AI Control a Server Player?

The answer: atomic actions that can be freely combined.

Acknowledgements

This chapter owes quite a few people as well, so first, a deep bow of thanks!

• @murasaki44: In #109, pointed out that progressive loading in 0.1.3 had made things worse. That one piece of feedback made me tear down half of 0.1.4.
• Claude Code: It really did show me how important bash, a single Turing-complete entry point, is.
• Voyager, Mindcraft: They chose to control their agents with scripts long ago. It took me a long detour to understand why the pioneers did it that way.
• CC: Tweaked, Super Factory Manager: Two pioneers that drive the game with scripts, and the inspiration behind Numen choosing a Turing-complete scripting language.

Why Not Just Give the AI a Keyboard?

Early on, a friend commented on one of my demo videos and suggested letting the AI press WASD directly. Setting aside the token explosion, at the speed an AI reacts, the results would be bad. Imagine walking from A to B to mine something: for the AI, that's a long string of tool calls, W W W W W A A A A S S S D D D, then holding down the left mouse button… Nobody's wallet can afford that, and nobody can put up with how slowly the AI would think.

What is an "atomic action"?

Let's call actions like WASD atomic actions: actions that cannot be broken down any further, as defined by the system.

Note the "as defined by the system" part. If you really insist on splitting "press W", you can of course break it down into: neurons fire in the brain, the signal travels to the synapses in your middle finger, the middle finger responds, and W gets pressed. (Why the middle finger? Because I play LoL, and my middle finger lives on W~)

So handing atomic actions like these straight to the AI clearly doesn't work: the results are poor, and it doesn't make economic sense either.

Combining Atomic Actions

So could we combine atomic actions first and then give them to the AI? For example, define a "mine" tool: when the AI calls it, Numen automatically carries out the whole sequence for the AI, "walk up to the target block and break it".

A sequence of actions made up of several atomic actions is what we'll call a workflow.

This approach is clearly great. Of course, I didn't come up with it all by myself: I looked at excellent open-source projects like Voyager and Mindcraft, and everyone does it this way. Nobody hands the AI a keyboard.

At this point, giving the AI sequence after sequence of actions, everything looked really, really wonderful. Little did I know that fate's gifts always come with a price tag, quietly attached, and this left a piece of technical debt for Numen's future.

Supporting the Mod Ecosystem: A Double-Edged Sword

With the AI now controlling a server-side player, the next question came right after: can we support the huge mod ecosystem of the Minecraft universe? We clearly couldn't let Numen become an island.

My galaxy brain kicked in: just extract an API layer and require every tool to register with it, even the tools Numen itself provides. Everyone's equal!

Wow, what an elegant architecture! It looked like Numen would soon support all kinds of mods. And yet, fate's gifts once again came with a quiet price tag, leaving another piece of technical debt for later (that's two for two).

There is no absolutely correct architecture, only architecture that fits the job. Seen from back then, this architecture was without doubt a success:

• We chose server-side fake players, getting nearly native Minecraft compatibility and an easy way to support other mods;
• We chose workflows for the AI to act through, so the AI could genuinely operate mods.

Fine, that was the early version of Numen. Before 0.1.4, Numen used a tool-registration architecture: any tool you wanted the AI to call just had to register with the API.

Once the architecture was settled, the coding began. Very quickly, Claude and I finished the early workflows: mining, building houses, fighting, crafting… a whole set of workflows for the AI to call.

At this point, I thought Numen had a good architecture and was a proper production-grade mod.

The Tools Kept Piling Up

The first to come knocking was the second piece of debt, the one that made it two for two.

Before long, I released the first version of the mod and quickly started supporting the mod ecosystem. Right away I ran into a serious problem: Numen's tool definitions kept piling up.

Adding a feature meant registering yet another tool. And tools combine with each other heavily: say tools A, B, C and D combine into a workflow. I still had to register an extra tool E = A + B + C + D so the model would know E is a dedicated workflow; if the model called A, B, C and D one by one, the results would be much worse. Atomic tools could perfectly well be combined into a workflow, yet I had to register the combined workflow all over again, with its parameters and description written out once more.

And every extra tool pushed token usage up with it. Tool definitions didn't cost many tokens back then, but with Minecraft's enormous mod ecosystem, every mod I supported forced me to add a few tools. Sooner or later Numen's tool list would become hugely bloated, possibly with hundreds of definitions, and that is catastrophic for a model's attention. Soon enough, Numen's 30-odd tools were about to drown the AI's attention.

From Progressive Loading to a Command Line

To deal with "every new tool costs more tokens", I introduced progressive tool loading in 0.1.3: give the model only the tool names, hold back the definitions, and let the model look up a definition when it needs one.

But the mechanism wasn't done well, and players soon reported that it worked badly (#109); it also tended to break prompt caching. On top of that, I felt progressive loading still wasn't enough: with hundreds of tools in the future, even listing just the names would be too much.

After a few days of thinking, my galaxy brain had an idea: a command line. Turn every tool into a command, and load tool definitions per namespace through --help, such as numen --help and ysm --help. This is really progressive loading at the level of mods, and it did cut the token usage.

Workflows Became Black Boxes

Then the first piece of debt came knocking too.

I thought through "atomic tools" and "workflows" again. Are atomic tools wrong? No. Sometimes the AI really does need an atomic tool like "right-click". Are workflows always right? Not that either. Sometimes the model needs fine control over the path it walks, rather than charging all the way to the end in one go. There's a trade-off hidden in here.

Take pathfinding, which people often complained about early on: the AI would sometimes break the player's house, even when it knew exactly where the house was. Not because it wanted to, but because it had no control over the "walking" workflow: the moment it issued goto xxx, Numen took over everything, and the program was rigid.

So I urgently needed a way for the model to have atomic-level actions while also being able to combine tools on its own, instead of relying on us to register yet another workflow for it.

The Command Line's Dead End

Around then, I happened to be improving AI building, and I quickly ran into a new problem.

At first, the model wrote an entire build script in a single tool call. The "script" here was just a DSL that Numen provided, such as "lay a layer of stone here" (width and length only) or "build a sphere here" (radius only). It was not Turing-complete. Writing out a whole build script in one go cost far too many tokens.

I wanted the model to work step by step: lay a layer of stone on the ground first, take a look, then carry on. That called for something like an "object": the model runs numen create build --name A to get a building object called A, then A.addlayer(xxx) to add to it step by step.

But a command line is stateless, while A is a stateful object. I tried forcing it onto the command line anyway: numen create build saved an A.numen text file to disk, A.addlayer appended a line to that file, and finally numen build A read A.numen and ran it line by line.

That was obviously very awkward. The real problem was the conflict between a stateless command line and a stateful object like A, and I had chosen to keep the state on disk. Soon I found all sorts of files piling up on disk, which clearly made no sense: it treated the symptom, not the cause. So I stopped: 0.1.4 was half rewritten, and I tore it down myself.

Turing Completeness Is the Answer

A command line still couldn't let the model control workflows, because a command line is not Turing-complete. I had to stop and rethink the whole architecture.

That's when I thought of Claude: how does Claude get so much done with only a dozen or so tools? What's the difference? When I lined up the tool lists, it was obvious: bash is the special one. bash can call all kinds of command-line programs and combine any operation with a Turing-complete language. Even if you deleted every tool except bash, Claude could still do most of its work: delete Read so the model can't read files, and it can use bash to write a Python script and read them through the standard library; delete Edit, and it can use bash to run Python for string replacement and still edit files.

Clearly, Numen needed a Turing-complete language to combine atomic tools into workflows, and to let the model control the details of a workflow itself, instead of, as before, walking along and digging straight through the player's house.

It was also at this moment that I realized Turing completeness is truly indispensable. The pioneers (Voyager, Mindcraft) chose to control their agents with scripts for good reason, and only then did I truly understand why the pioneers did it that way.

I kept researching and found the pioneer CC: Tweaked, plus Super Factory Manager, which I had played with in the ATM9 modpack. I found that most mods that need scripting chose a Turing-complete language. As for why it's Lua, there's a story behind that too, which we'll cover in detail in the Why Lua chapter.

The Current Architecture

Once we chose Lua, all we had to do was register atomic actions as a Lua API, and the model controls those atomic actions through Turing-complete Lua scripts. Both of the goals above were met:

• No more mindlessly piling up tools: we only provide a Lua API, and when the model needs something it calls a helper such as numen.api.help("numen.move") to get the details;
• The model controls its own workflows: for example, steering clear of certain areas while pathfinding.

Together with skills, this approach was without doubt a success, and it even greatly reduced the burden of supporting other mods.

Take Touhou Little Maid as an example. With the old tool-registration approach, I would probably have taught the model in a skill that "right-clicking a maid while holding cake tames her", and then bet on the model successfully calling two tools in a row: one to take out the cake, one to right-click and tame. The model would also need to think between the two calls, and I bet I'd soon find that right after the cake comes out, the maid wanders off, and the right-click just places the cake on the ground, and everything falls apart. Then I'd have to register yet another "tame maid" tool that takes out the cake and right-clicks in one go, which again means describing a tool and its definition to the model, and more tokens.

With a Lua script, the skill can spell it out, and a single Lua script can call the two APIs back to back. Both problems solved at once: fewer tokens, the same results, even better ones.

And so we arrived at the architecture for AI controlling a player:

All we have to do is register a Lua API, and the model can combine it freely. After a half-month delay (even though GitHub was getting updates every day), Numen 0.1.4 was finally released!