As we start the second half of 2026, we're improving our understanding of the in-work utility of the latest LLMs.
So far the main constraint on our project optimism is consistency and trust with the LLM model outputs. Will the models perform the same each hour, each day? The answer so far is a resounding NO. Network stress, model and harness changes, and general lack of transparency are driving software engineers a bit mad.
Despite the variability with the models' quality and consistency, we have proven that the LLM models can be excellent at certain tasks. The LLMs are a genuine quality and efficiency multiplier if used for the correct task — like code review, rubric-based evaluation, chat, migrations, and integrations. LLMs are great at conforming to response shapes. If we can ask an LLM to perform an activity according to an exact output shape, like with an API wire-up, data or code migration, we can expect compounding results in our process.
However, tasks that are long-running, parallel (i.e. swarms), or that depend on open-ended or greenfield reasoning are still experimental at best. Some engineers on X tell a tale of autonomous looping agents easily hammering out production work. I have yet to meet any of these engineers in person or replicate any similar results — but I will surely keep trying.
Proven tasks
- Code review
- Code test coverage
- Log triage for bugs and errors
- Judgement rubric evaluation
- Human-in-the-loop task, research, or analysis
- User chat
- Tight specs integration
- Migrations
- Data extraction and classification
Experimental tasks
- Greenfield reasoning
- Autonomous coding
- Long-running tasks
- Single-point orchestrator workflows
- Agent swarms
- Looping self-improvement (research or performance)
- Living agent memory