Vendor or build: an honest decision tree for AI features

The wrong question

"Should we use OpenAI or build our own model?" is the question we get asked most often in the first meeting of an AI engagement. It's the wrong question. The right question is: "What is the smallest surface area of custom AI work that delivers meaningful differentiation for our users?"

Almost every product we've built in the last two years uses at least one vendor API and at least one custom layer. Pure vendor and pure build are both losing strategies for most products. The decisions that matter are where the boundary sits and why.

The decision tree

The framework we use has four variables: data sensitivity, latency requirements, customisation depth, and volume.

Data sensitivity is often decisive. If your use case involves PHI, PII, financial data, or confidential client information that cannot leave your infrastructure, your architecture is constrained regardless of what you'd prefer. You're looking at self-hosted models (Llama, Mistral, fine-tuned open-source) or providers with appropriate data processing agreements and isolated inference environments. Many teams discover this constraint too late, after they've built on a provider that can't sign a BAA or a DPA.

Latency requirements determine whether you can afford a provider round-trip. Sub-200ms voice applications almost always need models running in the same cloud region as your telephony stack, with persistent connections and pre-warmed context. If you're building a voice agent and your inference is routing through a general API endpoint, you've already lost 40–80ms you can't get back.

Customisation depth is where most teams over-engineer. The question is not "would a fine-tuned model perform better?" It almost certainly would. The question is whether the performance delta is measurable by your users and worth the operational cost. Fine-tuned models require training pipelines, eval harnesses, versioned datasets, and deployment infrastructure. That's a real engineering cost that most teams underestimate by 3–5×.

Volume changes the maths. At low volume, provider APIs are almost always cheaper and faster to build on than self-hosted alternatives. At high volume, the per-token cost of provider APIs can exceed the amortised cost of self-hosted inference. The crossover point depends heavily on the model size and your infrastructure efficiency, but for most products we've seen it occurs somewhere between 50M and 200M tokens per month.

When to buy

Buy — use vendor APIs as-is — when: the task is general-purpose, your data can leave your infrastructure, your volume is under the cost crossover, and the provider's off-the-shelf performance meets your quality bar.

Examples: document summarisation for internal tooling, general-purpose Q&A for a knowledge base, content generation for marketing workflows, code assistance for internal developer tools. In all of these, the provider's model is good enough, the integration is fast, and the operational burden is low.

The trap is over-customising in this bucket. We've seen teams spend three months fine-tuning a model for a task where GPT-4o mini with a well-crafted system prompt would have hit 95% of the performance at 2% of the cost.

When to wrap

Wrap — add a custom layer on top of a provider API — when: you need domain-specific context injection, output formatting constraints, retrieval augmentation, or routing logic that the provider's API doesn't support natively.

Wrapping is the most common pattern in production AI products. You use OpenAI or Anthropic for inference, but you own the prompt construction, the retrieval layer, the output parser, and the fallback logic. The provider is a commodity inference layer. Your wrapping code is the product.

The critical discipline when wrapping is provider abstraction. Define an interface that your product code calls, with the provider implementation behind it. This makes provider migration a one-file change rather than a refactor. We've migrated clients from GPT-4 to GPT-4o to Claude 3.5 Sonnet as relative price/performance shifted — teams with good abstraction did this in a day. Teams without it spent two weeks.

When to build

Build — train or fine-tune your own model — when: you have a genuinely specialised task that general-purpose models handle poorly, you have the data to support it, and the performance gap is large enough to justify the operational cost.

Real examples where this makes sense: a medical imaging analysis model where the task is highly specialised and you have proprietary training data. A code completion model fine-tuned on your internal codebase and APIs. A document classifier fine-tuned on your specific taxonomy with tens of thousands of labelled examples.

The signal that you're in "build" territory is that general-purpose models consistently fail on your eval set in ways that better prompting doesn't fix, and you have data that would let a fine-tuned model learn what the general model can't infer. Both conditions need to be true.

The hidden cost of each path

Every choice carries a hidden cost that rarely appears in the initial build estimate.

Vendor APIs carry dependency risk: the provider changes pricing, depreciates a model, changes output format, or goes down. Mitigation: provider abstraction, multi-provider fallback, and explicit SLA contracts for enterprise tiers.

Wrapping carries prompt debt: system prompts grow over time as edge cases get patched in. After six months, a 4,000-token system prompt is carrying fix after fix for cases that should have been handled structurally. Mitigation: versioned prompts, eval-driven iteration rather than patch-driven iteration, and periodic prompt audits.

Building carries model ops overhead: training pipelines, evaluation infrastructure, deployment infrastructure, and the ongoing cost of model maintenance as your data distribution shifts. Mitigation: only build when you have the team to maintain it. A fine-tuned model without an owner is technical debt that compounds.

The framework is not a formula. It's a set of questions that force you to be explicit about what you're optimising for before you commit to an architecture that's expensive to reverse.