LLM models are updating quickly and output can be extremely unpredictable day-to-day, month-to-month. The LLM is, in many ways, a giant dark spot in our process consciousness. We must wrap our LLM components with tight process to reduce risk and improve predictability and quality. As part of this move toward an LLM-first production process, we must also account for the eternally incompetent "human in the loop." Yes, that means you and me! We are the fallible human.
Humans make terrible effort calculations. Process methodologies like Agile, Scrum, Lean, and Kanban exist because we are not good at estimating time and effort. Our existing process methodologies all try to limit the cost of incorrect assumptions over time.
Our goal is simple: keep our process flow such that final delivery is optimal, repeatable, and predictable. Like water behind a dam, if flow overwhelms capacity, we will end up with a flood on the high plane or the low plane. Imbalances in our software development system will cause floods that lead to thrashing, burnout, missed opportunities, poor quality, lowered team morale, and many more cancers.
Lean places a large emphasis on balance within our process. System components should always match the cadence of their dependent peers. Upstream producers should respect the capacity of downstream receivers. Dependency gates should strategically handle deliveries without disrupting flow. Batch sizes and work-in-process (WIP) should match the normal flowing capacity of the system, and ultimately be optimized for rapid learning and assumption validation.
LLMs are radically changing the flow of our systems. With LLMs we benefit from being able to create and traverse incredible context and scope — both in planning and coding. This is equal parts glorious and dangerous. The cost to produce vast amounts of code is nearing zero. The LLMs like Claude and Codex can rip more code in an afternoon than one person can audit in a month. It's a revolution. Our new increase in coding velocity is creating huge challenges for our existing software development workflows.
Our traditional software development pipeline was very balanced. We thought it wasn't. We thought that a few days of misalignment here and there was the end of the world. Well...LLMs are here, and now we know what real imbalance looks like.
Our traditional dev pipeline was something like a horizontal block chart: | Strategy | Specs | Design | Dev | QA | Deploy |, then iterate back to an important block as needed.
The Dev process was usually about 50% longer than its upstream resources and downstream dependents. This worked. It allowed Strategy/Product/Design to catch up while Dev/QA finished their work. We could create a predictable cadence. Strategy and Design were expensive. Dev work was expensive. Things usually evened out in the end.
With LLMs, we've completely nuked our traditional process. The most dangerous pattern I see with LLM development is merging our spec definition and development processes.
Now we have something like a | Strategy / Direction | block next to a monolithic vertical bulge that contains Specs + Dev + QA + Deploy, followed by a state of limited visibility that causes confusion, risky assumptions, and thrashing. This limited visibility is caused by a flood of new work product so prolific that it cannot be reliably reviewed and audited.
Generally speaking, our slash commands like /plan or "Plan mode" or whatever such buzzterms are wholly insufficient. Too much is being loaded into the LLM for planning and coding, far too quickly — either due to product scope creep, unreasonably large batch sizes, or parallel changes across multiple features/scopes.
Simply, we need much more structure up front, with much smaller batch sizes.
LLM-first in a Lean process
We should start our LLM process with product strategy, specs, and planning as a dedicated and recursive step, totally isolated from coding. This recursive cycle should lead to mature data specs and structured planning artifacts, usually as .md files.
Our strategy/specs/plans iteration cycle should loop until we have reached our Team Quorum Threshold (TQT). For example, we agree as a team before the project that 4/6 members of the team must review and approve all of our planning artifacts before moving to coding. It's quite easy to build an LLM script that interviews team members about each product assumption and corresponding plan in order to score the output, even as a simple binary pass/fail.
Review effort is best spent on specs and planning artifacts with stakeholders, owners, and experts. This is where review is much cheaper and more manageable. A planning artifact is almost always small enough to be genuinely reviewed, whereas a flood of high-velocity LLM code will quickly exceed review capacity.
Another metric that we can introduce during the strategy/planning phase is Iterations-To-Plan (ITP). This should track the number of meaningful iterations or changes to the plan before it has reached our needed TQT. This is something our agent can tally inside of a .md file for us. This number can help us evaluate our prompting efficiency and our LLM production efficiency. In the case where we're using some planning library or skill, ITP is a great measure to track.
The important point here is that we want iterations and quorum in our strategy and planning cycle. This is where review and change are cheapest. Our iterative planning cycle will become our own unique flavor of Spec-Driven Development (SDD). There are countless harness skills, articles, and projects that can provide further direction: Kiro, SpecKit, OpenSpec, Superplanning, GSD Core, spec-compare. The concepts are simple enough that we can use a pre-existing tool or simply roll our own. I tend to lean toward developing our own, as it becomes part of our "alpha" or special sauce. Each team is different. Just like Lean or Scrum adapt to each organization, so should our iterative strategy/planning cycle.
Our structured planning artifacts from this type of SDD workflow should also include explicit acceptance criteria for every spec. Where applicable, our structured planning artifacts should also include LLM evals that cover both the product acceptance criteria and our own internal LLM process automations.
Acceptance criteria and evals
Acceptance criteria are a set of expectations or rules that can produce a predictable result when evaluated by human, code, or LLM (usually pass/fail or a score). These are our concise business and customer-facing rules of the system. The acceptance criteria are similar in principle to code unit tests, and will usually be incorporated into unit tests in some way.
Acceptance criteria come in all forms and flavors. Here are some examples:
- Behavioral: "Given a registered user with an expired session, when they submit the checkout form, they are redirected to login with their cart contents preserved." "After a successful charge authorization is captured for an order over $500, order status is immediately set to "pending" and a fraud-review task is created and placed in the Potential Fraud queue."
- Technical preferences: "All monetary values are stored as integer cents. A central server-side module will be in charge of converting to currency float based on provided currency. Default currency is always USD."
Evals are our LLM-only acceptance or performance criteria. Evals judge how our system is performing when models change or the system degrades. Here are some examples:
- "Given the set of 200 labeled support tickets in ./example-data/support-tickets-01.md, the classifier must agree with the example labels ≥92%, with no single category below 85%."
- "Given the set of 30 reference documents in ./example-data/foo-examples.md, generated summaries from ./example-data/foo-examples-prompt.md must be between 400-600 characters and contain zero fabricated named entities, judged by Claude Sonnet 5."
- "Given the set of 10 Typescript code problems in ./example-data/foo-examples.md, generate solutions and check them against the provided ideal examples in ./example-data/foo-examples-solutions.md and the conventions in ./coding-rules.md, flagging regressions."
Micro batch size until automations are golden
Let's re-state some assumptions. LLMs can be unpredictable, verbose, and overly eager. Humans make terrible effort calculations. We're inside of a process where we find variability and risk introduced by both the LLM and the human. The human is largely a known variable, where slower production velocity has much smaller variance. The LLM has the potential for a massive range of variance, in both production quantity and quality.
Wide variance in velocity and quality fundamentally increases risk from incorrect assumptions. We cannot afford this additional risk until our automations are rock solid. These automations will include our harness automations like hooks, plugins, and other event-based logic.
Automations should be a priority concern during each project sprint until we can repeatably produce a predictable output. Expect this to be roughly half of our total delivery work each cycle.
Assuming all of this, the best thing we can do to improve our new LLM system process is to reduce scope and batch size for each production task. We want to work in micro batch sizes. This is traditional Lean thinking.
How small is small enough? This has always been one of the central concerns in Agile and Lean, and the answer hasn't changed: if you think it's too small, it's not small enough.