Part 2: How AI systems actually work

This is Part 2 of our The Field Guide to AI for Non-Tech CEOs.

Want to read Part 1 first? –> Part 1. Demystifying AI.

Demo vs AI wrapper vs real AI solution

Making an AI demo has never been easier.

Tools such as Claude Code, Lovable, Replit, Bolt, or v0 can generate functioning applications from a prompt. In less than an hour, someone with relatively little engineering experience can build something that uploads a document, sends it to an AI model and produces an impressive-looking answer, dashboard included.

On one hand, this is great. The time from idea to prototype has decreased significantly.

On the other hand, it creates a noise problem: everyone looks like they’re an expert.

So how do you sift through the noise?

To be clear, I am not saying this to dunk on people building demos. Demos are useful and help test ideas, make something abstract tangible and get internal buy-in.

The problem starts when the demo is presented as evidence that production implementation will be equally easy, or that AI will solve everything.

To understand just how easy it has become, try this yourself.

Copy the following into ChatGPT, Claude or whatever AI you use:

Then, take the prompt it gives you and paste it into Claude. You do not even need Claude Code. You can literally use claude.ai on the free plan and ask it to execute the prompt.

Claude will work for about 15 min. Et voilà! Here’s what you’ll get:

That is how easy prototyping has become. Let’s call it level 0.

Then comes Level 1: the “AI wrapper”.

Level 1 is about connecting an application to an OpenAI, Anthropic or other model API. Add a prompt. Let the user upload a document. Send the relevant content to the model and return its answer.

You now have an application that can talk, summarize, extract information, classify things, answer questions and perhaps call a few tools.

A thin application around an AI model can be valuable. In fact, many useful legit AI products will ultimately consist of existing models surrounded by very good software and data engineering.

So the term “AI wrapper” became a derogatory term because the thin application is just too thin: it just doesn’t survive contact with reality (eg your company environment).

That requires Level 2 execution: thinking about production-ready AI systems.

What a real production AI system requires

Imagine you want to build an AI assistant for your sales team so it can give instant answers about sales policies, sales playbooks, even customizing scripts per customer, what to do next for a particular account, etc.

That is trivial to demonstrate in Level 0 and Level 1.

But the version you want to give your salespeople has a much tougher problem: it has to live within your data, systems and existing policies.

Here is a non exhaustive list of what the AI solution needs to handle properly:

• Your data: What information does the AI receive? Where does it come from? How does it know which version is current? What happens when two systems contain conflicting information?
• Your permissions: Which employee is allowed to see which customer, document or transaction? As a result, which AI is allowed to read this information and disclose it to whom?
• Specific integrations: How does the AI actually interact with your CRM, ERP, email, databases, document repositories and other software?
• Reliability: What happens when the model gives a bad answer? What happens when your CRM is unavailable, a document is missing or an API call fails halfway through a process?
• Controls: Which actions can the AI take by itself? Which ones require a human to approve them first?
• Evaluation: How do you know whether the system is actually producing good answers? “It looks pretty good when we try it” is not an objective criteria.
• Monitoring: Can your engineers see what happened when something went wrong? Which information was retrieved? Which model was called? Which tool failed? What action did the AI attempt? At the end of the day, something needs fixing, so you need a trace of what went wrong
• Security: What company data leaves your systems? Where does it go? Is it stored anywhere? Which vendors can access it? Does this comply with your existing security policies?
• Cost and performance: Which model gets used for which task? How much does each workflow cost? What happens when 20 people using the prototype becomes 1,000 employees using the production system?
• Change: What happens when the model provider releases a new model, retires an old one, changes an API or your own business process changes?

It will take more work than prompting an AI to get this done.

This then gives you a useful rule:

What practical test could you use as a bullshit detector?

Before paying for a POC (Proof Of Concept) because the demo looked great, I suggest the following simple test to roughly evaluate real AI engineering understanding, not just write great prototyping prompts.

Do not ask this for the architecture of the demo (there will be nothing much to show), but of the real production engine they have in mind for your company.

By doing that, you are essentially asking to validate the blueprint of a house before building it.

A credible architecture should show most of the following, at least conceptually:

• Data, role & permission integration: what upstream integrations are required (eg how does it get all the data / policies in the first place);
• Data pipeline: what happens to the ingested data before it is even used by AI APIs (data cleansing? Normalizing? PII scrubbing? etc.)
• Auditability: how an audit trace can be produced (eg if an auditor asks you why a certain AI agent accessed a particular piece of data, and if it was authorized in the first place, can the architecture help you answer?)
• Tooling safety: how the systems restricts tools and access to AI
• Human gates: where human approval is required
• Data persistence: what data gets stored and where
• Action logging: what gets logged and where
• Data confidentiality from vendor: can devs or IT see the prompts and results in plain text in their logs
• AI evaluations: how the AI outputs are systematically evaluated against known criteria
• Integration to your systems: what downstream integrations are possible back into your system of records
• Failure modes: what happens when part of the workflow fails

The key is of course not to just read the doc, but to interview their tech team based on that doc. Any idiot can ask GPT / Claude to generate a good looking, comprehensive diagram. The point is to understand how their engineers think about it.

You obviously need someone technical enough to conduct the discussion, and it will take a few turns to get through the entire architecture. But most would agree that validating a blueprint is easier when the house is not built yet.

Can you rely on customer reference instead? That’s your call, but I would still conduct the architecture review anyway.

APIs, consumer AI and data privacy issues

Here are the main differences between using AI through a consumer product (e.g. ChatGPT, Claude.ai, Perplexity, etc.) and building with their API.

• Memory: consumer products may remember your preferences, projects and conversations. With an API, the model only sees the context you send. Long-term memory must be built into your application layer
• Pricing: consumer products usually charge a monthly subscription, like a normal SaaS product. APIs are usage-based: you pay for tokens, model calls and sometimes additional tools.
• Data privacy: “consumer version” and APIs come in different flavors and with different data privacy guarantees. But both types of guarantees are hard to enforce anyway, making data privacy a headache (also see below)
• Behavior: the same model may behave differently in a consumer product and through an API. Consumer products add their own instructions, memory, tools, search, safety controls and routing. With an API, you control more of that surrounding harness (on what’s a harness, see here.)

The implications for your company are as follows:

• To build a real AI solution requires serious AI engineering efforts. Do not conflate “I make my own MyGPT in ChatGPT and it just works” with building with the API. These are not comparable.
• If your staff are throwing company data into personal consumer AI accounts (GPT, Claude, etc), your data will be stored on their server. Whether it can be used for model improvement depends on the product and settings.
• API commercial contracts give stronger guarantees, but this is still ultimately a trust and risk decision. Your data may still be processed or temporarily retained by the provider, again depending on the service and configuration. For what you consider “proprietary” (e.g. literally no one has that data and you don’t want anyone to get their hands on it), then using frontier AI models will require much more work than simply calling their APIs.

On data privacy:

• On the consumer version of ChatGPT, Claude and Gemini, there are privacy settings and policies that provide some level of protection (like explicitly opting out of having your conversations used to improve the models). The problem of course is that you won’t have much of a recourse shall you find that they actually did train on your data 🙂
• On APIs, the guarantees are generally stronger. OpenAI says business and API data is not used to train its models by default, while Anthropic says the same for its commercial products, including its API. But “not used for training” does not necessarily mean “never retained”. Again, it is not clear that you will have much of a recourse…

Technically, once your data hits their servers, it is stored. A ZDR (Zero Detention Retention) guarantee is also as good as the provider ability to prove that the prompt and input weren’t stored after processing.

So a general rule of thumb on data confidentiality is this:

One solution to the conundrum (if you still really need frontier intelligence on proprietary data and workflows) is to essentially use frontier models via a different provider that is not the frontier labs themselves, as to never share your data with them directly.

For example, Azure (by MSFT) and AWS Bedrock (by Amazon) both host frontier model AIs on their own servers. Using OpenAI or Claude via their services should guarantee that your data is never stored on Open AI or Anthropic servers (they are, however, on MSFT or Amazon servers).

If you want total control over the server and own it yourself, then you can’t use frontier AI intelligence (they are just not available for self hosting and it would cost you a fortune anyway).

You could host open weights models, if you determine that what you need the AI to do doesn’t require the absolute frontier intelligence (and for most cases, you certainly don’t need frontier intelligence).

With that out of the way, here is a short explanation of what happens when you use AI APIs.

The anatomy of an AI API call

The stylized version of what happens is as follows:

Your application → API request → AI model → API response → your application

Suppose your application sends this: Read this customer complaint (that you provide in context) and tell me whether I should escalate it.

You may send additional data in context or not. Either way, all of the input will be tokenized, eg broken down into tokens (see What the hell is a token for more explanations). Parallel computations are then made on GPUs sitting in some distant data center.

As the model reasons through what you’re asking, it may decide that it needs additional information or that it needs to perform an action.

If you have given it access to tools that your company has vetted, it can use them to provide a richer answer. For example, it could decide that it needs use tools such as:

• get_customer_history(customer_id)
• search_company_policy(query)
• create_crm_note(customer_id, text)
• calculate_refund(order_id, amount)

If the only capability you give the model to retrieve customer information is a get_customer_history() tool, and that tool (programmed by whoever builds this for you) only returns information the user is authorized to see, then that is essentially the boundary of what it can do.

The caveat of course become as follows.

So the important point (which I make repeatedly throughout this Field Guide because it is important) is this:

What’s MCP, and what’s the difference with an API?

You will hear MCP a lot in AI land. One of my engineer friends even says “MCP is the new API,” which is even more confusing to non-tech people.

MCP (Model Context Protocol) is simply a standard way for an AI application to find and use external tools.

Remember our earlier section on tool access: a model by itself cannot check your CRM, search your company database or create a support ticket. You have to give it tools that can do those things.

Before MCP, developers typically had to build those capabilities into each AI application. MCP created a standard way to make such capabilities available for immediate use.

Think of it this way: if you have multiple applications that need a tool called “get_customer_history”, with different teams building each application, you certainly don’t want them to duplicate those tools in their own way. Instead, build the tool once and put it on an MCP server that other applications can use without rebuilding the tool.

So the distinction is roughly:

That is what my engineer friend means when he says MCP is becoming “the new API.” APIs aren’t disappearing. MCP often just sits one layer above them.

Then, do you need your own MCP server? It depends.

If you are building one simple AI application with a few custom integrations, maybe not. If you want the same company systems and tools to be available across multiple AI applications or agents, MCP becomes much more useful.

The good news is that lots of standard, generic tools to access popular SaaS software has already been built, so you can just use them.

For example, Google has published its MCP server for Gmail, Drive, Calendar, etc. You can use them to search emails, update meetings, browse Google Drive, etc.

Do you need to use open-weight models?

Open Weight Models (OWM) are AI models whose trained weights are made available for you to download.

What you download is effectively the model’s weights. To actually use them, you need computing infrastructure capable of running inference (called “inference infrastructure”). In a production environment, you would generally make that model available to your applications through an API.

This comes at a cost (think of open weights as a DIY option) but there are some very good reasons why you may want that control.

(Note that OWM are often loosely called “open-source models”, but technically this isn’t quite the same thing. Open-source AI implies broader access to things like the code and information about how the model was trained.)

Why use Open Weight Models (OWM) in the first place?

There are several legitimate reasons:

• Data confidentiality or sovereignty. You can run the model inside infrastructure you control, so that sensitive data never needs to reach the company that built the model.
• Customization. You have considerably more freedom to fine-tune, modify or optimize a model for a very specific workload.
• Offline deployment. A sufficiently small model can run on a laptop, phone, vehicle, factory equipment etc and give the answers you need without internet.
• Latency. If your application has a hard requirement for very fast response times, you can run the model physically close to the application calling it.
• Very high, predictable usage. At sufficient scale, operating your own inference infrastructure may cost less than paying somebody else per token (since you don’t pay per token anymore)
• Reducing model-provider dependency. You may want frontier AI APIs for some workloads while retaining the ability to use models you control elsewhere.

Data confidentiality deserves one big caveat though.

For example, AWS says that prompts and outputs sent through Bedrock are not used to train its models and aren’t shared with third-party model providers..

Similarly Google Cloud contractually states that it doesn’t use Customer Data to train or fine-tune AI models without permission.

Here, the data never hits AWS or Google Cloud servers, not the AI frontier labs’ ones. They companies don’t train Open AI / Anthropic models, so their commitment is that they basically won’t share your data to the frontier labs for AI training.

It’s a stronger guarantee than the frontier labs’ one indeed, althouth it is also not clear what your recourse would be if you found out your data made its way to the next AI model :)

So there are actually 3 different questions hiding behind the “I don’t want AI companies to have my data”:

1. Do I need to keep my data away from the company who built the model and can use my data to train its next model?
2. Do I need to keep it inside a particular cloud / country / region? (data residency constraint)
3. Or does it literally need to stay on physical infrastructure I own directly or control indirectly?

OWM gives you the option to solve #3, which a closed-weight API cannot. Depending on how you deploy it, it can also solve #1 and #2. More on this later.

Small Language Models (SLM) and composable architectures

You will find a lot of Small Language Models (SLMs) in the Open Weight ecosystem.

There is no standard definition of “small”, but we’re generally talking about models with a few billions to tens of billions parameters.

The smaller the model (parameter wise), the less resource it needs to work. Hence, if you want to put one on a phone, you really need a pretty small one.

For example, imagine a business workflow where you want AI to perform 10 different tasks:

• 1 is difficult and requires reasoning, for which you use a frontier AI model
• 9 others involve simple tasks like classification, extracting information, rewriting something into a particular format, etc

For these 9, you could use an SLM. You can therefore compose a system using different models for different parts of the workflow (referred to as a “composable architecture” or “hybrid architecture”).

The benefit isn’t simply reducing your frontier AI bill. It also means the application isn’t completely dependent on one model provider.

A useful design principle for your architect is therefore:

The catch: somebody now operates “inference infrastructure”

The OWM needs to run somewhere, called “inference infrastructure”.

You basically have four options:

For example, if you just rent an H100 GPU from AWS, AWS deals with the physical GPU, electricity, cooling and hardware failures.

But you are still operating the model.

Meaning that you still have to deploy the inference server, configure the model, manage capacity and scaling, monitor failures, patch vulnerabilities, secure the service, upgrade model versions, etc. That takes a specialized team to manage.

Alternatively, you can use a managed OWM endpoint where AWS, Azure, Google or another provider handles much more of that infrastructure for you.

Or simpler yet, you can call an API offered by the company behind the open-weight model. DeepSeek and Moonshot, for example, offer APIs for their models (which is then charged per token, just like a frontier model API)

How much does inference infrastructure actually cost?

It depends on the model. Let’s take a simple example: Mistral Small 3.1 (Vive la France!).

It’s an SLM with 24B-parameter. It handles text, images and tool calls, and has a 128k context window.

Mistral says it requires approximately 55 GB of GPU RAM to run a standard version (there are compressed versions), so for the sake of a simple illustration, let’s give it a single GPU of H100 with 80 GB of VRAM.

This is not necessarily the cheapest possible way to run the model, nor necessarily how you’d design a high-volume production deployment. It is just an example so we can compare costs with real numbers.

As of September 2026:

When you rent GPU, you pay by hour of usage. RunPod currently lists an H100 PCIe at $2.89/hour on its Secure Cloud. AWS currently lists a single H100 Capacity Block at $5.191/hour in US regions.

If you want to own that GPU infra, you’d buy it and set it up on a server. A single H100 GPU alone typically costs roughly $25,000–$35,000. So let’s round up the total capex at $40K to include all the rest (CPU, system RAM, storage, chassis, power supply, cooling, etc.).

Here, I chose a server rated at around 705W, so three years of continuous electricity at an illustrative $0.15/kWh adds roughly another $2,800 of electricity alone.

Then you’d add staffing OPEX since people will have to operate the thing for you.

But once you operate the model yourself, there is no token cost. Your cost is the infrastructure and the people required to operate it. Whether or not the investment makes sense depends on your use case.

This also gives a rough direction:

What are the trade-offs with OWM?

The useful way to think about OWM isn’t really as a list of pros and cons but more like a sequence of decisions.

First: why do you need an OWM?

Start with the hard constraint you are trying to solve.

For example:

• Data sovereignty: data cannot leave a particular infrastructure, cloud or jurisdiction.
• Proprietary data: you don’t want the model developer to ever receive your prompts or outputs.
• Physical control: the data cannot leave hardware you control for regulatory reasons.
• Latency: you have a hard requirement around response time.
• Offline: the AI needs to work without a network connection.
• Customization: you need to modify or fine-tune the model beyond what a managed provider allows.
• Economics: you have large, predictable volumes where operating inference is cheaper than using frontier models
• Strategic control: you need the ability to keep a specific model indefinitely or reduce dependence on a model provider.

Only then: decide who actually needs to operate it

If all you want is to keep data away from the company that created the model, renting a GPU in your existing AWS environment may be enough: you download the model and operate it there.

If you don’t even want to operate the inference software, a managed OWM endpoint may be enough.

If the data cannot leave your physical environment at all, then you may genuinely need on-prem infrastructure.

If you simply want cheaper tokens, you might not need to host anything. You could just call a cheaper model API.

These are the main choices you have when it comes to inference infrastructure.

Finally, spend time picking a model

If the above points you towards “I definitely need an OWM”, then the key is to pick the right one, since every OWM is different in capability.

There is no one-size-fits-all recommendation for this because it depends on the workload you need the AI to do.

And don’t assume that “bigger is better” in terms of parameter count.

In fact, a useful rule of thumb is:

Small models are cheaper and faster for simple tasks such as classification, data extraction, summarization, formatting, etc. Larger models are required as the task requires stronger reasoning, coding, broader knowledge, handling ambiguous inputs, better instruction following, etc.

Also check the license before you get too excited about a particular model. “Open weights” does not automatically mean unrestricted commercial use.

Here is the kicker that adds costs to your OWM choice: the only way you will know which OWM is actually good enough for your workload is to test them by building such workloads.

There is no real shortcut. Public benchmarks are useful for shortlisting models, but they are not your workload, your data or your quality threshold.

So the process is roughly:

Define the workload → define acceptable quality / latency / cost → shortlist models → test them on real examples → choose the smallest model that clears the threshold → size the infrastructure.

So, do you need open weights?

For most companies, probably not.

If a frontier API can do the job, meets your security and data requirements, and the economics are reasonable, it is usually the simplest and fastest way to get the job done.

You are effectively paying OpenAI, Anthropic, Google or somebody else to solve an enormous infrastructure problem for you.

My personal take is as follows (this is not investment advice):

If you are an enterprise operating in a regulated / sensitive environment (banking, finance, defence, government, etc.), use the composable architecture to keep optionality when it comes to model choice.

Some workloads may use frontier APIs through your enterprise cloud environment. Others may use managed OWM deployments on AWS/Azure/Google Cloud. The most sensitive workloads may justify OWM running on-prem.

If you operate in the mid-market or in a relatively unregulated sector, I would also use the composable architecture. But that doesn’t mean you need to use a bunch of OWM on day one.

Start with frontier / OWM APIs if they solve the problem. Only introduce managed OWM or self-hosting when there is a concrete reason to accept the additional complexity. If the architecture is truly composable, swapping models will not require the entire re-engineering of your workflow.

How to assess AI outputs objectively?

During a demo, you can eyeball the result of AI and decide if that particular output is correct. But if this process runs 1,000 times a day, how do you systematically evaluate AI outputs?

It is quite simple to evaluate an output where the possible values are known, closed-ended and where the right answer is also known. Think of assessing the answers from a multi-choice quiz where there is no “Other: type your own answer”.

However, evaluating outputs that are qualitative and don’t fall into neat categories is harder. Think of an AI that combines various other AI’s outputs to produce an investment memo.

To evaluate the latter, AI Evals borrow from a concept used in evaluating humans: “rubrics”. These are essentially tables where the relevant quality dimensions are broken down by the subject-matter-expert (in the investment memo example, it could be memo completeness, format, factual accuracy, sourcing quality, etc.).

Evaluating closed-ended outputs

For closed-ended outputs, you need a set of examples for which the correct answers are already known.

Say your AI has to perform part of a KYC screening process and classify customers as Approved , For review or Rejected. Give it historical customer cases whose outcomes have already been validated as correct and compare AI answers against those known outcomes.

You can then calculate something as simple as: The AI got 942 out of 1,000 cases right ~ 94% accuracy.

The wrinkle of course is that your initial data set would have to include cases that can reveal “false positives” and “false negatives”, so that you can also grade the AI on these. Eg:

• A false positive is a legitimate case incorrectly flagged.
• A false negative is a problematic case incorrectly approved.

There are other metrics like “precision” and “recall” but we won’t go too deep into the weeds here. All you need to know as a CxO for now is this:

Evaluating qualitative outputs

Things become harder when there is no single correct answer.

Take the investment memo example. You might have lots of examples, but you can’t cleanly derive a “this is a good memo and this is a bad memo”.

To solve this, we use rubrics: essentially a table that describes:

• The dimensions you care about and where the AI output will be graded
• A grading scale (say, from 1 to 5) with a clear description of what it means to give a score of 1, 2, 3 etc.

If this sounds like grading a human essay, that’s because it is.

Here’s a simplified example (a real rubric for investment memo would be much longer and detailed):

Then you could define the grading of Completeness like this:

For the sake of the example I am not breaking these further. In practice, you can already see how tricky this is. What’s “developed enough”? And “Adequately covered”?

The point is to make the grading criteria explicit enough that two different reviewers (be it human or AI) are more likely to arrive at similar scores.

You can check out Microsoft rubrics example here

It also raises the question: how do we know if the memo “is developed enough to convey the information expected from each” (score 5)?

This brings us to using LLM-as-a-judge techniques.

Using LLM-as-a-judge

The main way to automate evaluations for qualitative outputs is to use LLM-as-a-judge. Rubrics do the job of narrowing down the criteria enough, while AI will help grading at machine scale.

A good practice is to use a different model or model family to grade the output of the AI that produced the output. For example, if you gave GPT 5.6 the task of creating the investment memo, do not use GPT 5.6 as the judge.

To be clear: since API calls are stateless and “don’t remember you”, it’s not like GPT is magically remembering the work and giving itself a good score. The reason why it is best to use different models is AI bias: AI researchers found out that models may sometimes recognize and favor their own AI generations.

Also, if the rubric has been broken into narrow, very well-defined criteria, you might use an SLM rather than an expensive frontier model. The intuition is that if the scoring rubric has been simplified enough, a simpler model could be able to grade it.

So how do you decide which AI to pick as a judge, and what variations in evaluations you get?

The answer is simple: you have to test different versions, get the evaluation results and decide for yourself. When it comes to building AI systems, there is no magic shortcut: one has to build the damn thing in the first place and check.

So the rule of thumb for EVALS is this:

The anatomy of an effective EVAL system

Without EVALS, you may still manually inspect outputs, but that doesn’t scale and can be subjective.

The other big issue is you won’t know if your AI solution gets worse after changing a prompt, upgrading (or downgrading) model, or changing tool versions. It’s totally possible for AI output quality to regress when you update any of the above.

This makes EVAL design an important AI engineering skill, and something you should assess when picking an AI vendor.

At minimum, an effective eval system needs 4 things:

1. A representative test set
   This will come from you / your subject-matter experts. You don’t need an enormous dataset, for a vendor can augment one by using AI to generate additional examples (that’s called “synthetic data”). The point is that your seed data must be representative enough, as the synthetic data will inherit attributes of it
2. Clear grading criteria
   For closed-ended tasks, this means identifying the in/correct answers in the test data and the other key metrics you want to test for (accuracy, false positives, false negatives, etc.)
   For qualitative tasks, your subject-matter experts need to help define the rubrics that describe what “good” actually means.
3. An Eval harness
   This is the machinery that runs the evaluations at machine scale. At the core, a vendor should be able to explain for your specific workflow:
   • How AI outputs are graded (eg by deterministic code, LLM-as-a-judge, or something else?)
   • Why was that method chosen (you don’t always need an LLM to assess for example)
   • What the final scores / evaluations mean in the content of your business
4. A repeatable evaluation process
   Once the harness exists, you should be able to run the same Evals whenever you change an important part of the system and compare the results side by side. This is what makes evaluation scalable. As I said earlier, swapping for a better model doesn’t necessarily mean the AI output “grade” will automatically be higher.
   The only way to know this for sure is to run the EVAL with both models and see what’s coming out of it. This is called “EVAL regression testing”

So a good rule of thumb is this:

How to assess an AI vendor on their EVALS capabilities

A few practical questions will reveal quite quickly whether they know what they’re doing:

• Show me the Eval Set you would use for this workflow?
  • If they don’t know what an Eval Set is, this is not a good start. It should include your seed data as well as synthetic data
• What is being tracked as part of the Eval Harness?
  • Simply tracking the grades is not enough.
    • Remember that you want to be able to compare changes in output quality as various parts change in your system.
    • This means that you must track at the very least, for each EVAL test: the prompt, the context injected into the prompt, the model and its version, the tools it has access to, the output, the LLM being used (if LLM-as-a-judge has been used) and of course, the score
    • You must also track the rubric versions themselves, so that you know which eval set was used against which rubrics
• For anything graded by an LLM, how can you make sure the judge’s scores align with our subject-matter experts?
  • This is a trick question. The answer is simply that your SME will develop the rubrics in collaboration with their AI engineering team, and needs to validate the scores / outputs as the system is being built. The process should include SME validating the final thresholds of acceptance
• How will you detect AI drift?
  • Another trick question.
    • AI drift refers to the gradual difference between intended results from the initial system and actual results once in production even if nothing has changed in the system (which is slightly different than regression, which can happen when you change parts of the system).
    • This could happen because real-world processes may change (and hence, their prompts and relevant system config) but the system has not.
    • A vendor should sample approved production data periodically to run the EVAL test on it, compare against the baseline / thresholds agreed upon with your SME and if the system underperform.
    • Only then can they check if the reason is drift or regression (provided they have tracked all the versioning mentioned above)
• When the system makes a new type of mistake in production that was not tested against during development, and how will we know and how will it be added back into the Eval set?
  • A final trick question.
    • The answer should include the fact that the entire set (prompt, context, model etc.) that led to the mistake should be captured for a post-mortem with your SME.
    • Then it should be fixed accordingly, re-validated against an updated EVAL set and redeployed.

These are quite basic questions any experienced AI vendor should be able to answer. If they don’t, that’s a red flag.

On the risk of AI going rogue

If a good harness determines much of what an AI solution does, then why do people freak out about AI Agents going rogue? Can they actually “break out” of their harness?

Sometimes yes, but only under specific conditions.

Take the now famous “Hugging Face incident” of Jul 2026.

Hundreds of experimental AI agents participated in an unauthorized attack on Hugging Face infrastructure. An independent investigation by METR and Redwood Research provides a detailed reconstruction of what happened here: METR investigation OpenAI postmortem

The TLDR is that these were not ordinary API calls processing invoices or answering customer questions. They were highly autonomous agents running inside OpenAI’s frontier research infrastructure, asked to work on some extremely hard, sometimes impossible-to-achieve tasks.

In other words, they were operating in an unusually adversarial research environment, with unusually powerful capabilities, while being deliberately pushed to their limits. They found ways to bypass some of the controls, communicates with each other and eventually exploited systems outside the evaluation environment.

That is very different from saying that a mundane internal AI system will spontaneously decide to hack its way out of your CRM permissions.

Adversarial evaluation is just one of these things frontier labs do during post-training, where models are deliberately pushed hard to understand how they behave and where they fail before being released to the general public.

You are obviously not going to do that type of work just to process mundane business workflow. So the much more realistic concern is: