Building an AI Development Orchestrator: Why the Runtime Comes Before the AI | Article

_{Steps 1 and 2 of the Series: Building DevRag: A Deterministic Runtime for AI Development}

Most AI coding setups fail for one reason: they have no state model.

The model gets a task. It writes something. You run it, it breaks, you paste the error back. The loop continues until either the code works or you give up. There is no ownership model, no audit trail, no way to reason about what the system actually did — and no recovery path when it goes wrong.

I’ve worked with large legacy enterprise systems — security tooling, data-heavy integrations, fintech — where debugging state is often harder than writing code. The production incidents that are hardest to fix are not the ones with clear error messages. They are the ones where something is wrong but nobody can reconstruct the sequence of state changes that led there.

This project is my attempt to apply that lesson to AI-assisted development. The project is called DevRag — a research prototype for automating the software development cycle using multiple AI agents. Not a chatbot that writes code. A system where development is a managed process and AI agents are role-based executors inside that process, constrained by a deterministic runtime that they cannot bypass.

In most AI coding setups, the model is the center of the system. In DevRag, the model is a worker. The runtime controls what it can do, when it can do it, and what counts as done.

The alternative — letting agents directly modify files, repository state, and task metadata — works in demos. It fails as soon as multiple actors are involved, because there is no longer a single source of truth about what happened and in what order.

This is the kind of constraint system that is required when AI is used in production, not in demos.

This article covers the first two milestones: what was built, the decisions that were made, and what the system can and cannot do at this stage. No AI agents are running yet. That’s not an accident.

The Architecture Before Any Agent Was Written

The central design decision in DevRag is the separation of two layers:

DevRag Runtime handles all state-changing operations. It manages the execution cycle, invokes utilities, and parses structured agent responses. It does not make semantic decisions. It cannot write code. It does not know what a “good implementation” looks like.

LLM Agents handle all semantic decisions. They receive structured task context, analyze it, and return structured JSON responses. They have no direct access to the filesystem, the repository, or the task queue. They cannot change the state of anything directly.

This separation is not optional. Without it, the system cannot guarantee correctness. It is the foundational correctness guarantee of the system. If the runtime is the only component that can change state, then state is always deterministic, auditable, and recoverable — regardless of what any agent decides to do.

The goal is not to automate single tasks. It is to build a system where complex, multi-step development workflows can be executed by multiple agents safely — where “safely” means the system can always tell you what state it is in, how it got there, and who authorized each change.

The agent roles, at a high level:

Manager — orchestrates the workflow, selects tasks from backlog, makes accept/rework/escalate decisions, handles merges
Worker — implements one atomic task at a time, operates in minimal context
QA — independently verifies acceptance criteria, runs tests, reports pass/fail per criterion
Architect — optional; decomposes complex tasks into subtasks
Tech Writer — documentation, changelogs, PR descriptions

None of these agents exist yet as running processes. Before any agent can be trusted with a task, the system that manages those tasks has to be correct.

Why This Architecture Matters in Practice

This design choice — runtime owns state, agents only propose decisions — directly addresses three failure modes that are expensive in production systems:

Uncontrolled state leads to incidents. In enterprise systems with multiple consumers touching shared resources, state that is not formally owned is state that will eventually become inconsistent. The runtime-as-single-writer pattern eliminates the class of bugs where two components disagree about the current state of a task.

Unclear ownership makes bugs unfixable. When an AI agent can directly modify files, the repository, and task metadata, you lose the ability to reason about cause and effect. If something is wrong, you cannot tell whether the agent did it, the orchestrator did it, or a race condition between the two did it. Centralizing mutations in the runtime makes the audit trail complete.

No audit trail creates compliance risk. In any system that matters — banking integrations, security tooling, enterprise workflows — the ability to reconstruct what happened and why is not optional. The changelog in DevRag is append-only and every state change is attributed to a role. This is not a feature that was added for completeness; it is a property of the architecture.

In practice, this translates to:

fewer production incidents caused by inconsistent state
faster debugging when something goes wrong
predictable execution of multi-step workflows

Milestone 1: The Deterministic Foundation

The task file is the single source of truth in DevRag. Every task is a markdown file with a strict structure: title, description, acceptance criteria, metadata block, and changelog. The metadata lives in HTML comments embedded directly in the file. The changelog is append-only.

The first milestone built exactly two things: the ability to reliably read these files and the ability to reliably write them — nothing more.

Domain Model

The domain model is three dataclasses: Task, Metadata, and ChangelogEntry. The TaskStatus and AgentRole values are str-subclassed enums so they serialize to their string representations at the file format boundary without a separate serialization layer.

Metadata includes every field that controls task lifecycle: branch, last_commit, retry, commits, status, assigned_to, assignee_role, parent, subtasks. Subtasks are stored as a CSV string in the file and parsed into a structured list at the IO boundary. The model itself never sees raw CSV.

Parser

TaskParser.parse(path) -> Task is the sole entry point for reading a task file. The parser is fail-fast: any structural deviation raises a ParseError with a line number and an actionable message. There is no silent fallback, no best-effort partial parse, no None returned on malformed input.

The validation contract is explicit: all nine metadata fields must be present, status must be a valid enum value (invalid strings fail at parse time, not later), changelog role values must belong to the allowed set, and entries with a header but no body raise an error with the offending line number.

One design decision worth examining: the error semantics for task ID validation went through three iterations. The initial implementation rejected malformed IDs inside the title regex, producing a generic “malformed title” error regardless of what was actually wrong. The fix separates the two failure cases — a malformed title structure and a malformed task ID — so the error message tells you specifically which constraint was violated. This matters at scale: when a pipeline is processing hundreds of task files, “malformed title” and “malformed task ID” are not the same debugging problem.

Writer and Atomic Operations

TaskWriter.write(task, path) serializes the domain model back to canonical markdown format. The output is deterministic: given the same Task object, it always produces the same bytes. This matters for roundtrip guarantees: parse(write(parse(file))) == parse(file).

All writes go through a temp file + atomic rename pattern. The file is written to a .tmp sibling, then renamed to the target path. A crash between write and rename leaves the original file intact. A crash after rename means the new file is already complete. There is no window where a partial write is visible as the current state of the file.

Changelog appends (append_log) work through a similar atomic pattern. The log is append-only by design — no existing entry is ever modified, and a failed append does not corrupt existing content.

What Milestone 1 Guarantees

After this milestone, the system has exactly these properties:

A task file can be parsed into a structured domain model without ambiguity
A domain model can be serialized back to a file in a canonical format
Parse errors are explicit, typed, and include enough context to fix the problem
File writes are atomic — partial writes cannot occur
The changelog cannot be corrupted by a failed append
There is no state machine, no repository abstraction, and no CLI

This is not the complete system. It is the layer that everything else depends on being correct.

Milestone 2: State Machine and Repository

A task file that can be reliably read and written is not yet a managed task. A managed task has a lifecycle: it moves between statuses, those movements are governed by rules, and the system enforces those rules unconditionally.

Milestone 2 built the state machine and the repository abstraction that operationalizes it.

The State Machine

The canonical task lifecycle in DevRag:

DevRag - Task Flow v1 — DevRag – Task Flow v1

Each transition has three constraints: the current status must match, the requesting role must be allowed to execute this transition, and the command name must be exact. The TransitionEngine enforces all three before touching any file.

There is no “skip” behavior. A task in running cannot transition directly to done by Manager decision without going through testing. The sequence is fixed.

Retry semantics are deterministic: the retry counter increments by one only on done → todo (rework path). It resets to zero on escalation. This means the retry count is a meaningful signal — it tells you how many rework cycles a task has gone through.

FileTaskRepository

TaskRepository is an abstract interface with six methods: create, get, list, move, update_metadata, append_log. The concrete FileTaskRepository implements this interface against the filesystem.

The key design decision: task status equals folder. A task in running state lives in tasks/running/. A task in done state lives in tasks/done/. If the file is in tasks/done/ but the metadata says running, that is an invariant violation. The folder is authoritative on recovery.

list() supports multi-status queries, filtering by role and parent task, and returns results in stable ID order. There are no hidden parse errors — if any file in any scanned folder fails to parse, the error surfaces immediately. Silent skip is not a behavior this system has.

Race conditions in create() are handled through O_EXCL — the file is created with exclusive open, and on conflict the system retries with a new ID. This ensures no two tasks ever share an ID, even under concurrent creation.

Invariants

The system formalizes four invariants and enforces them:

status in metadata must match the folder the file lives in
Tasks in todo, running, or testing must have a non-null assigned_to
Tasks in running must be assigned to a Worker role
Tasks in testing must be assigned to a QA role

These are not soft guidelines. They are checked before any state-mutating operation.

The integration of invariant enforcement into the repository went through three iterations — and this is worth examining in detail, because it illustrates a real problem in system design.

First attempt: InvariantChecker.check() was called inside FileTaskRepository.move(). This immediately broke 69 existing tests. The tests were moving tasks to running without setting assigned_to — which is an invalid state. The enforcement was correct, but the tests were testing invalid setups.

Manager AI’s response: Re-open the task. The regression suite fails. AC7 says “existing valid flows continue to pass without modification.” The Manager treated the 69 failures as genuine regressions.

Second attempt: Remove enforcement from move(), keep it only in TransitionEngine.apply(). Regression suite goes green. Manager re-opens again. The contract says “enforce in repository write paths.” Keeping enforcement only in the service layer violates the original scope.

Third iteration: Reframe what “valid flows” means. The 69 failing tests were testing invalid workflow states — moving tasks to running without assignment was never valid per the invariant specification. AC7’s “existing valid flows” does not protect tests that construct invalid states. The tests were updated to use valid states. Enforcement was restored to move(). Full suite: 638 passed, 0 failed.

This is the key observation: Manager AI found an inconsistency in the specification itself and enforced the original contract even when the easier path was to weaken it. The specification said one thing (enforce in repository write paths), a simpler interpretation would have accepted the weakened version, and the Manager rejected the weaker interpretation twice. That behavior is useful — and not something a single-agent system would give you.

The final invariant enforcement architecture:

get() and list(): call check_status_matches_folder() on every loaded task — folder/status consistency is enforced on every read
move(): calls _checker.check(task) after setting the new status but before any filesystem write — no invalid state can be persisted
update_metadata() and append_log(): intentionally not enforced here, because the pre-transition handoff pattern requires temporarily setting assignee_role = QA while the task is still in running before calling ToTest. This creates a transient intermediate state that would fail invariant checks. The enforcement at move() is the correct gate — the invalid intermediate state cannot become a status change without passing the check

What Milestone 2 Guarantees

After this milestone:

All task transitions are strictly governed by the state machine — no transition can be skipped
The task file remains the single source of truth; folder position is authoritative on recovery
No invalid task state can be persisted through the repository or transition engine
All operations are atomic — a crash does not leave the system in a partially-mutated state
All errors are typed and explicit — ParseError, InvariantError, TaskNotFoundError, RepositoryError
The full test suite covers repository operations, state transitions, retry semantics, invariants, and end-to-end filesystem flows

The system still has no CLI, no runtime orchestration, and no AI agents.

What the Manual Process Revealed

The current orchestration is manual. Tasks are created by hand, agents are invoked individually, branches are merged by hand, and results are fed back to the Manager AI with context. This is deliberate: running the system manually before automating it means the failure modes are known before the automation hides them.

Two observations from the first two milestones:

Manager AI found uncovered scenarios multiple times. Even with a detailed Application Design Document in context, the Manager consistently identified edge cases not covered by existing tasks and created follow-up tasks for them. Task 028 was reopened twice. Task 013 was closed as redundant after the Manager determined its cases were already covered by Tasks 004 and 010. The AI review cycle is not just a quality gate — it is an iterative specification refinement process.

The ADD is not a complete specification. The design document described what the system should do. The implementation process revealed ambiguities in the contract — specifically, where invariant enforcement belongs. The resolution required distinguishing between “repository write paths” as the spec described them and “the authoritative write path with semantic context” as the implementation required. The document did not make this distinction. The implementation process forced it.

This is what makes the Worker/Manager separation useful: the Worker optimizes for making the current task pass. The Manager holds the architectural contract and checks that the result satisfies the original intent — not just the current test suite.

Where Milestones 1 and 2 Leave the System

The hard part of building an AI-assisted development system is not the AI. It is building a system where the AI’s decisions are safe to act on.

After two milestones, DevRag has no CLI, no running agents, and no automation. What it has is a foundation where every state change is traceable, every error is explicit, and no component can produce an inconsistent state without the system rejecting it. Tasks can be reliably parsed and written. The state machine enforces transitions deterministically. The repository guarantees that no invalid state can be persisted. Invariants are checked before every write. The full test suite covers all of this — 638 tests, all green.

This is not a prototype that “mostly works.” It is a deterministic core that will not produce surprises when agents start running on top of it. Every failure mode that is not prevented at this layer becomes exponentially harder to reason about when AI agents are in the loop. That is a lesson that comes from debugging production systems, not from building prototypes.

The manual orchestration also revealed something that was not obvious from the design document: the Worker/Manager role split is not just an organizational pattern. It is a specification refinement mechanism. The Manager held the architectural contract through three iterations of TASK-028 and rejected two implementations that passed the tests but violated the original intent. That behavior — an agent that reviews outcomes against design intent, not just against a test suite — is what makes the pipeline worth building.

In the next article, Milestone 3: the CLI layer. This is where the runtime gets its public interface and the structural decisions made here will be tested under real usage — whether the contracts hold, where the abstractions leak, and how much friction appears at the boundary between a deterministic runtime and the code that invokes it. The expectation is that the foundation is solid. The question is what breaks when it meets reality.

This approach comes from working on systems where correctness matters more than speed — and where the cost of getting state wrong is measured in hours of debugging, not minutes of rework.

Steps 1 and 2 of the Series: Building DevRag: A Deterministic Runtime for AI Development: Building DevRag: A Deterministic Runtime for AI Development