Inside a 1,300-Line Scripting Language

An earlier post in this series covered why my engine has its own scripting language and what wait states buy. This one goes through the implementation, because most of what makes it work is in how little of it there is.

The whole system is 1,320 lines across ten files.

It Doesn't Have a Parser

The engine vendors cparse, a shunting-yard implementation, and the language is built on top of it.

That changes the accounting substantially. Expression parsing is the part of a language most likely to be subtly wrong and most tedious to get right: operator precedence, associativity, unary versus binary minus, parenthesization. Handing that to an existing library means the bespoke part is only the statement layer.

What I wrote is a line-oriented statement interpreter that evaluates each line's expression through an existing parser. That's a much smaller and much more defensible undertaking than writing a full language from scratch, and it's most of the reason the whole thing fits in 1,300 lines.

The dependency is kept behind a pointer, so cparse doesn't appear in any engine header:

std::unique_ptr<cparse::TokenMap> m_Internal;

It Wasn't Meant to Be a Language

The commit history has this arriving sideways. On 2021-09-14 the engine gains "a cparse based expression system", and the commit says what it was for: "use it for things in guis". The goal was evaluating small expressions inside GUI definition files, so a label could compute its own text and a panel could size itself against a value.

Nine days later it was a language. Variables on 2021-09-15. Scoped functions on 2021-09-17, alongside a commit moving script programs out of the game and into the engine. Error handling, exception handling and argument binding for reference and rvalue parameters across 2021-09-18. The yield keyword on 2021-09-19, which is the entire wait state mechanism and the subject of the earlier post. Compiled expressions and local scopes on 2021-09-20, and program contexts that carry their own time deltas "so that they can be executed independently of frames".

cparse took the part of a language that has to be exactly right and has no shortcuts. Everything added afterwards was a statement layer, a scope chain and a binding layer, and each of those is a few days of work when the expression grammar underneath is somebody else's finished problem. Picking the dependency first is what made the nine days possible, and it is the same judgment as the reliable UDP layer two engines earlier.

One artifact of that week is a name. A commit on 2021-09-17 calls the language ranchscript, and it appears nowhere else in 585 commits and nowhere in the source. Dungeoneer was going to be a monster ranching and homesteading roguelite, and the language took its name from that.

The intent is still sitting in the data. The map the main menu loads is ranch.json, and it holds a house, four torches and a player spawn rather than a dungeon. There is a monster.schema.json that generates a Monster class with a size and a type. What there isn't anywhere in the game code is taming, feeding or breeding. The ranch is a place the player starts rather than a thing they run, and the language is still carrying the name of the game that was planned rather than the one that got built.

Execution State Is a Vector of Tuples

The entire state of a suspended program:

typedef std::tuple<std::string, int>              BlockStackFrame;   // keyword, start line
typedef std::vector<BlockStackFrame>              BlockStack;
typedef std::tuple<std::string, int, BlockStack>  FunctionStackFrame; // name, PC, blocks
typedef std::vector<FunctionStackFrame>           FunctionStack;

FunctionStack m_FunctionStack;
float m_waitTimer{ 0 };
float m_waitTime{ 0 };

A stack of function frames, each holding a name, a program counter, and a stack of open blocks. Two floats for timing.

There's no C++ stack involved, no fiber, no coroutine frame, no saved registers. Execution position is data, which is what makes suspension conceptually free. Nothing needs to be captured when a script pauses, because the position was never anywhere else.

This is the payoff of the line-based design. Control flow works by moving the program counter: a while pushes a block frame recording where it started and jumps back on block end, an if whose condition fails jumps past its block, a function call pushes a frame at counter zero. Each of those is an integer assignment.

One Program, Many Contexts

Parsed programs are cached and shared:

static cmVectorMap<std::string, Program*> m_programs;

A ProgramContext holds a pointer to the shared program plus its own tiny execution state. So a hundred bees running bee.script parse it once and carry a hundred small stacks.

That split matters more than it looks. The immutable, expensive thing is shared. The mutable, per-instance thing is a few dozen bytes. Getting that boundary wrong, by giving each entity its own parsed program, would make script memory scale with entity count times script size instead of with entity count.

Contexts also have a persistence mode:

enum class Persistence { Persistent, Volatile };

A volatile context owns its binding scope and deletes it on destruction. A persistent one doesn't. That's the difference between a one-shot script and an entity behavior that lives as long as the entity.

One Source of Truth for What's Scriptable

The type system is a variant:

using Value = std::variant<bool, int, intptr_t, float, std::string, Vector3, cmColor>;

and what's scriptable is derived from it with a concept:

template <typename T>
concept expression_arg = detail::get_value_index<T>() < std::variant_size_v<Value>;

get_value_index<T>() returns T's index in the variant. A type that isn't a member returns an index past the end, so the concept fails.

That's one source of truth. Add a type to Value and it immediately becomes usable as a script function argument and return type everywhere, with no separate registration table to update and no way for the two to disagree.

Registration then only accepts functions whose whole signature satisfies it:

template <expression_return_value Return, expression_arg... Args>
void RegisterFunction(const std::string_view& name, FunctionPtr<Return, Args...> function)

Binding a function taking an unsupported type is a compile error at the registration site rather than a runtime failure in a dungeon.

The marshalling is generated per signature. package_args extracts each argument from the variant by an index computed at compile time from the C++ parameter type:

std::forward_as_tuple(
    std::forward<Args>(std::get<get_value_index<Args>()>(values[i]))...);

No runtime type dispatch, no string comparison of type names, no manual unpacking per binding. Writing a new script function is writing a normal C++ function and one registration line.

Scopes are constructed through tag types that force the parent to be explicit:

struct Parent { explicit Parent(BindingScope* parent); BindingScope* p; };
struct Global : public Parent { Global() : Parent(&GetGlobalScope()) {} };

The default constructor is private, so a scope can't be created without saying what it inherits from. That's a small thing that prevents a whole category of "why can't this script see that function."

Suspension Is the Absence of an Action

The wait handling is where the line-based design pays off, and there is almost nothing to it.

else if (program.IsWaitStatement(keyword))
{
    if (m_waitTime == 0)
    {
        m_waitTime = line.compiledExpression.EvaluateFloat(*m_scope);
        m_waitTimer = 0;
        continueRunning = false;
    }
    else
    {
        m_waitTimer += gP->GetDeltaTimeSecs();
        if (m_waitTimer >= m_waitTime) { m_waitTime = 0; AdvanceProgramCounter(); }
        else { continueRunning = false; }
    }
}

There's no suspend call and no resume call. Suspending is declining to advance the program counter and then dropping out of the loop. Next frame the interpreter is re-entered, finds itself on the same line, accumulates another frame's worth of time, and eventually advances.

WaitUntil is the same shape with a condition instead of a timer: advance if it's true, give up the frame if it isn't. And yield is the degenerate case, advance and stop, which is a wait of exactly one frame.

Three behaviors, one mechanism. All of them are "leave the counter alone and return," which only works because the counter is a field rather than a position in the C++ call stack. A tree-walking interpreter would have to unwind and rebuild its recursion to do the same thing, and that's the reason this design is worth choosing for behavior scripting even though it's the less fashionable one.

The loop control needs care. Every path that suspends has to actually stop running, and every path that completes has to keep going so the next statement runs in the same frame rather than a frame later. Those two requirements pull in opposite directions and each branch has to answer both, which is a small amount of code carrying more weight than its size suggests.

What I'd Change

Cap the statements executed per frame. A script with a tight loop and no wait in it currently runs to completion inside one frame, or forever if it never terminates. A budget turns a runaway script into a slow one rather than a hang, which is the same degrade-rather-than-stall property the asset streamer has.

Make the wait absolute rather than accumulated. Storing a target time and comparing against the clock is one variable instead of two, can't drift, and doesn't depend on being called exactly once per frame. Summing deltas works and quietly assumes a caller that never double-steps.

And add tests. There are none, and the ones that matter here are small: a script with a wait should span more than one frame, a script that returns should pop its frame, and a while should re-enter its block. The execution state is a plain vector of tuples, so assertions about it are easy to write. Any behavior expressed as data is testable without a window, and this engine never took advantage of that.