Developer Tooling
How do we make AI-assisted development workflows faster, safer, and more auditable?
AI-assisted development has a tooling problem. The individual coding tools are good. The workflow around them — review, testing, deployment, auditing — has not caught up. Engineers using AI coding tools work faster in isolation and encounter bottlenecks at every handoff point: review, integration, testing, and deployment.
This research stream investigates the full workflow: not just code generation, but everything that happens between generation and production. AI code review, test generation, spec-to-code pipelines, and CI/CD gates that understand the difference between a trivial change and a structural one.
The central finding driving this stream is that AI tooling adoption fails when the tools add work rather than remove it. A code review tool that flags the same issues your linter catches gets disabled within two weeks. A deployment gate that blocks on false positives gets bypassed. The design challenge is building tools that surface the issues AI-generated code specifically introduces — architectural drift, subtle security anti-patterns, missing error paths — without adding friction to the issues already caught by existing tooling.
We release everything we build here as open-source: devbridge-review, agent-harness, and deploy-diff all came out of this stream.
Research articles
Key findings
We instrumented 5 teams over 30 days with AI code review tools. The 2 tools that flagged issues also caught by standard linters were disabled within 14 days by 80% of engineers. The tools that focused exclusively on architectural drift, missing error paths, and security anti-patterns not covered by linting maintained 70%+ daily active usage at 30 days.
Tool instrumentation + weekly engineer surveys on usefulness rating over 30 days. Disabled or bypassed tools were treated as adoption failures regardless of technical accuracy.
We ran 20 spec-to-code experiments where AI generated application code from written specifications. In 17 of 20 cases, the generated code was syntactically correct and passed linting. In 11 of 20, it embedded architectural assumptions that conflicted with the spec — using a shared singleton where the spec implied isolated instances, building synchronous flows where async was required, ignoring multi-tenancy constraints. These issues were not surfaced by any automated check.
Two senior engineers reviewed each generated codebase against the original spec, rating architectural fidelity independently. Architectural conflicts were defined as design decisions that would require significant refactoring to align with spec intent.
AI-generated test suites consistently achieved 80–85% line coverage — significantly higher than the 45% average in our pre-AI projects. However, human review found that security-related test scenarios (injection, auth bypass, privilege escalation) were present in AI-generated suites at 12% the rate of human-written suites on the same codebase. Edge case coverage was similarly deficient.
We compared AI-generated test suites against manually written suites on 8 matched codebase pairs. Coverage metrics plus manual classification of test types (happy path, error handling, security, edge case) by two reviewers per codebase.
We A/B tested two deployment gate configurations on real CI/CD pipelines: one that flagged high-risk changes for human review but allowed deployment to proceed, and one that blocked deployment pending human approval on changes above a risk threshold. The blocking configuration reduced production incidents by 67%. The flagging configuration reduced incidents by only 11% — engineers often approved deployments without reviewing the flag.
90-day A/B test across 4 matched engineering teams on similar codebases. Incident rates normalised by deployment frequency. Incident severity weighted by customer impact rating.
Methodology
We run adoption studies with real engineering teams in addition to technical accuracy tests. A tool that is technically accurate but unused is a failed tool. We measure both technical performance and sustained adoption rates.
- Code review accuracy testing: seeding known issues into codebases and measuring detection rate by issue type
- False positive measurement: running tools on known-good code and measuring incorrectly flagged issues
- Adoption studies: instrumenting tool usage by real engineers over 30-day periods
- Spec-to-code quality assessment: measuring architectural coherence of AI-generated code against human-reviewed architecture specs
- Test generation coverage analysis: comparing AI-generated test suites against manually written suites on the same codebase
We measure true positive rate, false positive rate, and sustained adoption rate at 7 days, 14 days, and 30 days after tool deployment. False positive rate is weighted more heavily in adoption prediction than true positive rate — engineers tolerate missed issues more readily than noise.
We share adoption failure case studies as prominently as success stories. Most published AI tooling research shows accuracy metrics; we also publish the adoption studies where technically accurate tools were abandoned.
Experiment log — including the failures
Hypothesis: A deployment gate that blocks on high-uncertainty changes will reduce production incidents more than a gate that flags and allows deployment to proceed.
67% incident reduction with blocking gate vs. 11% with flagging gate over 90 days.
Flagging without blocking is nearly ineffective — engineers approve the deployment without engaging with the flag. Hard stops force engagement. The key is setting the risk threshold carefully; too sensitive and it becomes a bottleneck.
Hypothesis: AI can generate pull request descriptions that are as useful to reviewers as human-written descriptions, given access to the diff and commit history.
Descriptions rated equivalent on completeness; significantly lower on context (why the change was made, not just what changed).
AI reliably describes the what of a change from the diff alone. It cannot infer the why without access to the ticket, conversation history, or the developer's intent. We now use AI descriptions as a template that developers fill in with context.
Hypothesis: Inserting a human-reviewed architecture step between spec and code generation will improve architectural coherence of AI-generated code.
Architectural conflicts in generated code dropped from 55% of projects to 8% when architecture review was inserted as a mandatory gate.
Architecture must be specified before code generation, not inferred by the model. This finding is now a core principle of our AI-native development methodology. The gate adds 1–2 days to project timelines but eliminates the expensive architectural rewrites that were consuming 3–5 days in the control group.