What AI Coding Agents Pick When You Let Them Choose Your Stack

Tell Claude Code, Codex, or Cursor to “add payments” or “wire up email” and it will not just write the integration, it will pick the vendor first, usually without asking you. A growth-marketing outfit called Armature spent the last few months measuring exactly what that choice looks like at scale: 16,893 coding-agent sessions, 5,292 of them kept for the published dataset, across 75 repositories, 1,163 prompt variations, and three agents actually writing and running the code, not just describing what they’d do. The number that should give you pause: given the identical task, all three agents picked the same vendor in only 42% of cases.

How the test was actually run

The setup is more rigorous than the usual “I asked ChatGPT and it said X” post. Armature built four personas representing how differently people brief a coding agent: vibe-coders who describe a symptom (“payments keep failing”), junior engineers who name the desired end state and category, senior engineers who specify concrete requirements and constraints, and enterprise engineers who add compliance and procurement language. Those personas generated 1,163 prompt variations, run against 51 codebases spanning 18 sectors and 10 programming languages, rotated across three different sandbox providers (E2B, Blaxel, and Daytona) “just to be safe” against provider-specific quirks.

A separate Gemini 3.7 Flash instance played the simulated human giving follow-up instructions mid-session, and a second Gemini instance validated the sessions and identified winners after the fact. Out of 16,893 total runs, 5,292 sessions were kept for the published analysis; the authors note the remainder “weren’t thrown in the bin” and may show up in a second wave later.

If you want more background on why coding agents behave so differently depending on how they’re scaffolded, our piece on AI harness engineering covers the layer that sits between a raw model and a tool like Claude Code or Codex.

The agents don’t agree with each other, or with you

The single most interesting number in the whole dataset is the 42% agreement rate. Give Claude Code, Codex, and Cursor the exact same prompt for the exact same feature, and less than half the time do they land on the same vendor. The voice agent category is the clearest example: asked to add a voice layer, Claude Code reaches for Twilio, Codex goes to the OpenAI Realtime API, and Cursor picks Vapi. Same task, same repo, three different production dependencies now sitting in your codebase depending on which tool happened to be open.

Web search behavior explains part of the split. Codex searches the web in 94% of sessions, and in 9 out of 10 of those searches it narrows results with a site: operator aimed at trusted domains. Cursor searches in about two-thirds of sessions. Claude Code is the outlier at roughly 30% baseline, since it leans on what it already knows rather than looking things up, though that rate climbs toward 80% in newer or less-familiar sectors where its training data is thinner. An agent that searches less isn’t necessarily wrong, but it is more likely to hand you a vendor that was popular when its training data was collected rather than the one that’s actually competitive today.

Who actually wins each category

Despite the disagreement between agents, individual categories still show strong winners once you aggregate across all three:

Repository language is doing a lot of the work here. It’s not that Resend is objectively better than SendGrid, it’s that agents are pattern-matching on what shows up in TypeScript codebases versus Python ones, which is exactly the kind of bias worth knowing about before you accept a recommendation uncritically.

Mentioned constantly, picked almost never

The gap between “the agent brings this vendor up” and “the agent actually writes it into your code” is one of the more useful things this dataset surfaces. PayPal was mentioned in 139 sessions and picked in zero of them. LangChain came up 194 times, more than almost anything else in the dataset, and was picked in 4 sessions, about 2%. Netlify appeared 152 times and converted at roughly 4%. Whatever training-data familiarity puts a name in an agent’s head is clearly not the same signal that gets it into the final diff.

That’s a useful sanity check if you’re on the other side of this: a tool showing up constantly in an agent’s reasoning traces doesn’t mean developers using that agent are actually adopting it. If you’re trying to gauge real usage, the pick rate matters far more than the mention rate.

Claude Code would rather build it than buy it

Across the dataset, Claude Code reached for a hand-rolled, in-house implementation instead of an external vendor in 19% of applicable sessions, against roughly 10% for Codex and Cursor combined. That’s consistent with how Claude Code tends to behave in practice: more willing to write a custom auth flow, a custom queue, or a custom rate limiter from scratch rather than pull in a managed service. It’s a real tradeoff, not obviously wrong. A hand-rolled solution has no vendor lock-in and no third-party billing surprise, but it also means your codebase now owns maintenance, edge cases, and security review for something a mature vendor already solved. Worth knowing which way your agent leans before it makes that call silently on a Friday afternoon.

Presentation beats capability more than you’d expect

Two specific findings stand out because they’re not about raw feature comparison at all. Agents rejected Supabase in several sessions specifically because of “too many unnecessary BaaS features… presented in a bundle pricing,” even when the functionality on offer was adequate for the task. And Mailgun “regularly lost against Postmark when agents read ‘1-day retention’ on its free plan,” a detail that reads like a footnote but was apparently enough to tip multiple decisions.

Neither of those is a capability gap. They’re packaging and copy decisions that happened to read badly to a model skimming a pricing page. If that pattern holds as more purchasing decisions get delegated to agents, how your free tier is worded starts to matter as much as what it actually does.

The part worth reading before you cite this study anywhere

Here’s the detail that changes how you should weigh all of the above: Armature, the company that ran this study, states plainly that it “sells growth services to dev tools,” and describes the research as “part of our broader work on how to influence coding agents’ choices and get products picked.” In other words, the same company measuring which vendors get chosen by AI coding agents also sells services to help vendors get chosen by AI coding agents. That doesn’t make the numbers fake, the methodology described is genuinely more rigorous than most vendor-commissioned research, but it does mean the study doubles as a sales pitch for a nascent field: optimizing your product for an agent’s judgment the way SEO optimized pages for a search engine’s ranking algorithm.

The Hacker News discussion around the study caught this immediately. One commenter put it bluntly: “if your business is basically getting [models] to use products that aren’t actually best for the job, you are scum.” Others were more measured, pointing out that having actual measurement, even from an interested party, still beats the alternative of everyone guessing based on vibes. Either way, treat the top-line numbers as a real signal about current agent behavior, and treat any future service that promises to get your product “picked by AI agents” with exactly the skepticism you’d apply to an SEO consultant promising a page-one ranking.

This isn’t a new problem so much as a new instance of an old one. We’ve written before about why LLM benchmark scores need more scrutiny than a leaderboard number implies, and the same instinct applies here: a striking statistic from a single study is a starting point for your own judgment, not a replacement for it.

What this actually means for you

If you’re building with an AI coding agent day to day, the practical takeaway is not “don’t trust the agent,” it’s “don’t let the agent’s first answer be the only answer you see.” Ask it to name what it considered and rejected. If a pick is tied to pricing or limits, make it search and cite something dated rather than relying on priors that might be stale. And if the vendor it picked is one you’ve never used, spend the five minutes to check it independently before it becomes a production dependency, the same way you’d want a junior engineer’s first suggestion checked before it ships. Our roundup of AI coding tools and CLI-based coding agents is a reasonable starting point if you’re still deciding which agent fits your workflow in the first place.

If you’re on the other side, building a developer tool that these agents might reference, the uncomfortable implication is that agent traffic is now part of your funnel whether you planned for it or not. A confusing pricing page or a scary-sounding retention policy in your free tier isn’t just a UX nit anymore, it’s something a model reads cold and can silently disqualify you over, at a scale no human sales team would ever notice happening.

Conclusion

The headline finding isn’t that Stripe wins payments or that Neon wins databases, both are plausible on their own merits. It’s that three of the most widely used coding agents, given the identical instruction, agree with each other only 42% of the time, and that popularity in an agent’s reasoning (139 mentions of PayPal) has almost no relationship to what actually ends up in your codebase (zero picks). That’s worth knowing before you treat any single agent’s first suggestion as a considered decision, and it’s worth remembering that the study surfacing all of this comes from a company whose business depends on influencing exactly the outcome it’s measuring.