In-House, Agency or Freelancer for AI Work

Which model to pick, and what actually decides it

The way this decision is usually run is a spreadsheet comparing a salary against a day rate. That spreadsheet answers a question nobody asked. Every AI system that has gone into production has a second year, and the second year is dominated by re-qualification work: a model version gets retired, the document corpus behind retrieval drifts, an upstream API changes its response shape and the agent quietly reasons over garbage. The cost of that year is set entirely by whether someone inside your building can make a change and prove it was safe.

So run the comparison over eighteen months, and score each option on capability transfer rather than on price. The rest of this guide is how to do that specifically.

The three models, and the failure mode each one carries

Each option is good at something real and fails in a specific, predictable way. None of them is the safe choice. Pick the failure mode you can actually manage.

In-house

What it is genuinely good at: the evaluation suite compounds. Every incident becomes a test case, and the accumulated test cases are what make your third system cheaper than your first. Domain knowledge is the other half. The reason a retrieval system works is usually a set of unwritten rules about which document supersedes which, and an external team receives a sanitised version of that at best.

The failure modes are hiring latency and the bus factor. An engineer new to LLM work spends roughly the first month rediscovering things the field already knows: that retrieval quality dominates model choice, that structured output needs a schema plus a repair loop rather than a firmer instruction, that any agent loop needs a hard step cap and a dead letter path. That month is paid partly in production incidents. Then there is the deeper problem. One person writes the prompts, nobody else can tell whether a change made the system better because there is no eval set, and when that person leaves you own a system that cannot be modified. A team of one is not a team, it is a dependency with a notice period.

Agency

What it is genuinely good at: pattern reuse. A team that has shipped this before arrives with a retry policy, idempotency keys, an eval harness and a logging schema already decided, instead of inventing each one badly on your time. They have also seen the failure modes and will argue with your scope, which is worth more than the code.

The failure mode is handover fiction. The deliverable is a repository plus a document, and the knowledge that matters is the twenty decisions in neither of them: why chunking is set the way it is, which three prompt variants were tried and rejected, which supplier API lies about its rate limit headers. Second failure: the demo optimises the comfortable majority of cases, and the operational cost of an AI system lives in the tail. The tail is discovered after go-live, which is after the contract closed. Third, and least discussed: the agency does not pay your token bill, so nothing in their incentives pushes them to make a run cheaper.

Freelancer

What it is genuinely good at: a bounded piece with a clean interface and a test set. A document extraction pipeline with a defined output schema. A migration script. An evaluation harness. Work that has an end, a definition of done, and no ongoing decisions attached to it.

The failure mode is continuity and blast radius. One person holds production credentials, there is no offboarding procedure because nobody wrote one, and there is no second pair of eyes on a tool that can write to a real system or on a prompt injection path. When they become unavailable the system does not announce it; a queue grows quietly for nine days. The honest counterweight is that variance in this market is enormous, and a strong freelancer will outbuild a weak agency comfortably. You are simply carrying that variance unhedged.

The Ownership Ladder

Ownership is not a single thing that changes hands on a Friday. It is five separate capabilities and they transfer independently, at different costs, and mostly not by accident. Write down who holds each rung today and who holds it in twelve months. The system dies at the highest rung with nobody's name against it.

Why AI work breaks the normal build-then-maintain split

A conventional web application that nobody touches keeps behaving the same way, so a build contract with a light support tail is a reasonable shape. An LLM system does not have that property, because three of its dependencies move on somebody else's schedule.

• Model retirement and silent version changes. Providers retire versions and ship new ones. Your prompts were tuned against the behaviour of one specific version. A newer version can be better on average and worse on the exact thing you depend on, which is often an output format constraint rather than reasoning quality.
• Retrieval corpus drift. The documents behind the system change. A policy gets rewritten, the chunk boundaries shift, and the retrieval quality on established queries drops without a single error being raised. Nothing fails. The answers just get worse, and the first person to notice is a customer.
• Input drift. The shape of what users send changes. A new customer segment writes differently, a supplier starts sending a different file layout, somebody begins using the system in a second language because it appeared to work once.
• Tool and API drift. An endpoint changes its response shape. The tool wrapper does not validate, so it returns something structurally plausible and semantically wrong, and the model dutifully reasons over it and produces a fluent, confident, entirely incorrect result. See how AI agents fail for the full set.

The consequence for staffing is direct. Whoever holds rung three needs to still exist when one of these lands, and at least one of them will land inside the first year. A fixed price build with no maintenance arrangement is not a cost saving. It is a decision to have this handled in a panic by whoever happens to be closest.

Run the arithmetic on the eighteen month window

Compare total cost of capability over eighteen months, not the price of the build. Two lines usually missing from the internal option are the months before the hire starts and the months after they start when output is reduced. Two lines usually missing from the external option are maintenance days and the port work in the drift section above.

Read the last two outputs together, because the gap on its own is misleading. Against a single system the internal column is far more expensive in cash, and it is also the only column that produces an asset: an evaluation suite, a failure log and a person who has read every incident. That asset is what makes the second and third systems cheaper, which is why the cost per system owned is the fairer line. One engineer covering one workflow is an expensive way to buy a workflow. One engineer covering four is a different proposition, and the arithmetic says so as soon as you fill the field in. The external column, meanwhile, is understated in almost every proposal, because maintenance days are quoted as a retainer floor rather than as what a month containing a model port actually consumes.

If you want the full cost picture across the life of a system rather than just the staffing slice, total cost of ownership for an AI system has the rest of the lines.

Which work suits which model

Match the staffing model to the shape of the work rather than to the size of the budget. The reliable signal is whether the work ends, and whether the decisions inside it are yours or generic.

The hybrid that works, and the one that does not

The arrangement that reliably works is an external team building alongside a named internal owner who ships the second change alone. The arrangement that reliably fails is the same sentence with the word 'afterwards' in it. The difference is not effort or goodwill. It is whether the transfer has a date, a test and money attached.

There is a third arrangement worth naming because it is common and it is the weakest of the three: staff augmentation dressed as delivery. A contractor sits inside your team with no delivery accountability, no defined outcome and no internal owner above them. You get external delivery risk at internal delivery speed, the knowledge still walks out at the end, and there is no contract instrument to point at because the contract bought hours. It is a legitimate model when you already have rung five occupied and simply need hands. It is the wrong one when the reason you are hiring is that nobody internally understands the problem yet.

Test the handover before you sign, not after

Handover is not a document, it is a set of drills that either pass or do not. Put them in the statement of work before price is discussed, because gates are a paragraph at that stage and a negotiation later. Attach a holdback to them so the drills happen while the supplier is still commercially present.

# handover-acceptance.yaml
# Run these gates BEFORE the final invoice, not after it.
# The person running them is the named internal owner, on their own machine.
# Every gate is pass or fail. There is no partial credit on a handover.

owner: "<name of the internal engineer who will hold rung three>"
run_before: "release of the final payment"

gates:

  - id: clean-clone
    do: >
      Clone the repository onto a machine that has never run this project.
      Follow the README only. Bring the system up in a local or staging
      environment.
    pass_if: "It runs without asking the supplier a single question."
    common_failure: >
      An undocumented environment variable, a database that was seeded by
      hand once, or a vector index that exists but has no build script.

  - id: eval-green
    do: "Run the evaluation suite from that clean clone."
    pass_if: >
      The score matches the number in the acceptance criteria, and the suite
      reports which cases failed and in which category, not one aggregate.
    common_failure: >
      The eval set lives in the supplier's account, or it is twenty cases
      that were chosen after the system already worked.

  - id: change-drill
    do: >
      The internal owner makes one real change alone: tighten a prompt, add a
      field to an output schema, or add a retrieval filter. No help.
    pass_if: >
      The change reaches staging, the eval score moves, and the owner can
      explain why it moved.
    common_failure: >
      The live prompt is assembled from three places and nobody can say which
      string the model actually received.

  - id: model-swap-drill
    do: "Point the system at a different model version in config. Re-run evals."
    pass_if: "One config change, no code edits, and a score you can compare."
    common_failure: >
      The model id is hardcoded in six files, or the prompt depends on one
      provider's formatting habits and produces unparseable output elsewhere.

  - id: incident-drill
    do: >
      The supplier breaks something in staging without telling the owner what.
      The owner diagnoses it from logs and dashboards only.
    pass_if: "Root cause found inside one hour, from the run trace."
    common_failure: >
      The logs record that a model call happened, not what went into it, so
      the trace proves nothing.

  - id: secret-rotation
    do: >
      Rotate every credential the supplier ever held, working from the
      documented inventory.
    pass_if: "The inventory is complete and the system still runs afterwards."
    common_failure: >
      A personal API key belonging to one contractor, billed to their card,
      still sitting in production.

  - id: cost-visibility
    do: "The internal owner produces last month's model spend, split by feature."
    pass_if: "The number comes from your billing account, not the supplier's."
    common_failure: "Everything ran on the supplier's provider account."

holdback:
  percent_of_fee: "<agree this at kickoff, not at handover>"
  released_when: "all gates pass"

The two gates that catch the most are the change drill and the incident drill, because neither one can be satisfied with documentation. Everything else on that list can be produced by a supplier who is tidy. Only those two prove that a person on your side can operate the system when it is behaving badly, which is the only condition under which operating it is difficult.

1. Write the maintenance sentenceDay 1

   Name, by person and role, who holds rung three in eighteen months. If the sentence cannot be written truthfully, you do not yet have a staffing decision to make. You have a resourcing decision wearing a procurement costume, and buying a supplier will not resolve it.

2. Build the acceptance evaluation set firstDay 1 to 2

   Thirty to sixty real cases from your own history, with the correct answer recorded for each, including the ugly ones you would rather not show a supplier. You need this artifact whichever way the decision goes, and it is the only thing that makes the three options comparable on quality rather than on confidence. Method in writing evals for LLM systems.

3. Price the eighteen month windowDay 3

   Run the calculator above with your own numbers, then add the port cost from the drift section for at least one model retirement. If the external option looks dramatically cheaper, check whether maintenance days are in it, because they usually are not.

4. Score every option on all five rungsDay 3

   Today and at twelve months, with a name against each rung. Any rung with an empty box at twelve months is the thing to fix, and it is usually cheaper to fix directly than to solve by choosing a different supplier.

5. Ask each external candidate to describe a failureDay 4

   Ask what their last difficult project failed at and what the failure mode was mechanically. A team that has only shipped successes has either shipped very little or is editing the story, and the specificity of the answer is the most reliable technical signal available in an hour. More questions in hiring an AI automation agency.

6. Put the gates into the statement of work before priceDay 5

   Handover drills, holdback percentage, eval set custody and the model version pin. All of these are inexpensive to agree while the commercial conversation is still open. The clauses that matter are set out in contract terms that matter for AI projects.

Terms worth being precise about

Definitions

Fully loaded cost

The total annual cost of employing one engineer, including employer taxes, benefits, equipment, software licences, amortised recruitment fees and their share of management time. It is materially higher than the salary line, and comparing a salary against a contractor day rate is the single most common arithmetic error in this decision.

Ramp cost

The output an engineer does not produce while learning a domain, a codebase and a toolchain. In LLM work part of it is paid in production incidents rather than in slow delivery, because several of the lessons only arrive as failures on real traffic.

Bus factor

The number of people who can become unavailable before a system can no longer be changed. For AI systems, count only the people who can change a prompt and then prove the change was safe. That number is usually smaller than the size of the team.

Handover debt

The gap between what a delivered system's documentation says and what somebody actually needs to know to change it safely. It is invisible on handover day and it is paid in full on the first incident afterwards, at whatever rate the supplier is charging by then.

Staff augmentation

A contract for a person's time rather than for a defined outcome. It is a legitimate arrangement when an internal owner already exists and simply needs hands, and it is the wrong one when the reason for hiring is that nobody internally understands the problem yet.

Re-qualification

Re-running the acceptance evaluation after something outside your control changes, such as a model version retirement or a rewrite of the document corpus behind retrieval. It is ordinary, recurring work in AI systems and it is absent from most build budgets.

Before you commit to a staffing model

Run this before signing anything, and treat any unticked box as a decision you have made by default rather than a detail you will sort out later.

One closing note on sequencing. If you cannot answer the rung three question today, the correct next move is usually not a supplier search. It is two weeks of internal work producing a measured baseline and a set of failing cases, after which the staffing question answers itself and every conversation with an external team gets shorter and more honest.