Want to read Part 1 first? –> Part 1. Demystifying AI.

Demo vs AI wrapper vs real AI solution

Making an AI demo has never been easier.

Tools such as Claude Code, Lovable, Replit, Bolt, or v0 can generate functioning applications from a prompt. In less than an hour, someone with relatively little engineering experience can build something that uploads a document, sends it to an AI model and produces an impressive-looking answer, dashboard included.

On one hand, this is great. The time from idea to prototype has decreased significantly.

On the other hand, it creates a noise problem: everyone looks like they’re an expert.

So how do you sift through the noise?

To be clear, I am not saying this to dunk on people building demos. Demos are useful and help test ideas, make something abstract tangible and get internal buy-in.

The problem starts when the demo is presented as evidence that production implementation will be equally easy, or that AI will solve everything.

To understand just how easy it has become, try this yourself.

Copy the following into ChatGPT, Claude or whatever AI you use:

Then, take the prompt it gives you and paste it into Claude. You do not even need Claude Code. You can literally use claude.ai on the free plan and ask it to execute the prompt.

Claude will work for about 15 min. Et voilà! Here’s what you’ll get:

That is how easy prototyping has become. Let’s call it level 0.

Then comes Level 1: the “AI wrapper”.

Level 1 is about connecting an application to an OpenAI, Anthropic or other model API. Add a prompt. Let the user upload a document. Send the relevant content to the model and return its answer.

You now have an application that can talk, summarize, extract information, classify things, answer questions and perhaps call a few tools.

A thin application around an AI model can be valuable. In fact, many useful legit AI products will ultimately consist of existing models surrounded by very good software and data engineering.

So the term “AI wrapper” became a derogatory term because the thin application is just too thin: it just doesn’t survive contact with reality (eg your company environment).

That requires Level 2 execution: thinking about production-ready AI systems.

What a real production AI system requires

Imagine you want to build an AI assistant for your sales team so it can give instant answers about sales policies, sales playbooks, even customizing scripts per customer, what to do next for a particular account, etc.

That is trivial to demonstrate in Level 0 and Level 1.

But the version you want to give your salespeople has a much tougher problem: it has to live within your data, systems and existing policies.

Here is a non exhaustive list of what the AI solution needs to handle properly:

• Your data: What information does the AI receive? Where does it come from? How does it know which version is current? What happens when two systems contain conflicting information?
• Your permissions: Which employee is allowed to see which customer, document or transaction? As a result, which AI is allowed to read this information and disclose it to whom?
• Specific integrations: How does the AI actually interact with your CRM, ERP, email, databases, document repositories and other software?
• Reliability: What happens when the model gives a bad answer? What happens when your CRM is unavailable, a document is missing or an API call fails halfway through a process?
• Controls: Which actions can the AI take by itself? Which ones require a human to approve them first?
• Evaluation: How do you know whether the system is actually producing good answers? “It looks pretty good when we try it” is not an objective criteria.
• Monitoring: Can your engineers see what happened when something went wrong? Which information was retrieved? Which model was called? Which tool failed? What action did the AI attempt? At the end of the day, something needs fixing, so you need a trace of what went wrong
• Security: What company data leaves your systems? Where does it go? Is it stored anywhere? Which vendors can access it? Does this comply with your existing security policies?
• Cost and performance: Which model gets used for which task? How much does each workflow cost? What happens when 20 people using the prototype becomes 1,000 employees using the production system?
• Change: What happens when the model provider releases a new model, retires an old one, changes an API or your own business process changes?

It will take more work than prompting an AI to get this done.

This then gives you a useful rule:

What practical test could you use as a bullshit detector?

Before paying for a POC (Proof Of Concept) because the demo looked great, I suggest the following simple test to roughly evaluate real AI engineering understanding, not just write great prototyping prompts.

Do not ask this for the architecture of the demo (there will be nothing much to show), but of the real production engine they have in mind for your company.

By doing that, you are essentially asking to validate the blueprint of a house before building it.

A credible architecture should show most of the following, at least conceptually:

• Data, role & permission integration: what upstream integrations are required (eg how does it get all the data / policies in the first place);
• Data pipeline: what happens to the ingested data before it is even used by AI APIs (data cleansing? Normalizing? PII scrubbing? etc.)
• Auditability: how an audit trace can be produced (eg if an auditor asks you why a certain AI agent accessed a particular piece of data, and if it was authorized in the first place, can the architecture help you answer?)
• Tooling safety: how the systems restricts tools and access to AI
• Human gates: where human approval is required
• Data persistence: what data gets stored and where
• Action logging: what gets logged and where
• Data confidentiality from vendor: can devs or IT see the prompts and results in plain text in their logs
• AI evaluations: how the AI outputs are systematically evaluated against known criteria
• Integration to your systems: what downstream integrations are possible back into your system of records
• Failure modes: what happens when part of the workflow fails

The key is of course not to just read the doc, but to interview their tech team based on that doc. Any idiot can ask GPT / Claude to generate a good looking, comprehensive diagram. The point is to understand how their engineers think about it.

You obviously need someone technical enough to conduct the discussion, and it will take a few turns to get through the entire architecture. But most would agree that validating a blueprint is easier when the house is not built yet.

Can you rely on customer reference instead? That’s your call, but I would still conduct the architecture review anyway.

APIs, consumer AI and data privacy issues

Here are the main differences between using AI through a consumer product (e.g. ChatGPT, Claude.ai, Perplexity, etc.) and building with their API.

• Memory: consumer products may remember your preferences, projects and conversations. With an API, the model only sees the context you send. Long-term memory must be built into your application layer
• Pricing: consumer products usually charge a monthly subscription, like a normal SaaS product. APIs are usage-based: you pay for tokens, model calls and sometimes additional tools.
• Data privacy: “consumer version” and APIs come in different flavors and with different data privacy guarantees. But both types of guarantees are hard to enforce anyway, making data privacy a headache (also see below)
• Behavior: the same model may behave differently in a consumer product and through an API. Consumer products add their own instructions, memory, tools, search, safety controls and routing. With an API, you control more of that surrounding harness (on what’s a harness, see here.)

The implications for your company are as follows:

• To build a real AI solution requires serious AI engineering efforts. Do not conflate “I make my own MyGPT in ChatGPT and it just works” with building with the API. These are not comparable.
• If your staff are throwing company data into personal consumer AI accounts (GPT, Claude, etc), your data will be stored on their server. Whether it can be used for model improvement depends on the product and settings.
• API commercial contracts give stronger guarantees, but this is still ultimately a trust and risk decision. Your data may still be processed or temporarily retained by the provider, again depending on the service and configuration. For what you consider “proprietary” (e.g. literally no one has that data and you don’t want anyone to get their hands on it), then using frontier AI models will require much more work than simply calling their APIs.

On data privacy:

• On the consumer version of ChatGPT, Claude and Gemini, there are privacy settings and policies that provide some level of protection (like explicitly opting out of having your conversations used to improve the models). The problem of course is that you won’t have much of a recourse shall you find that they actually did train on your data 🙂
• On APIs, the guarantees are generally stronger. OpenAI says business and API data is not used to train its models by default, while Anthropic says the same for its commercial products, including its API. But “not used for training” does not necessarily mean “never retained”. Again, it is not clear that you will have much of a recourse…

Technically, once your data hits their servers, it is stored. A ZDR (Zero Detention Retention) guarantee is also as good as the provider ability to prove that the prompt and input weren’t stored after processing.

So a general rule of thumb on data confidentiality is this:

One solution to the conundrum (if you still really need frontier intelligence on proprietary data and workflows) is to essentially use frontier models via a different provider that is not the frontier labs themselves, as to never share your data with them directly.

For example, Azure (by MSFT) and AWS Bedrock (by Amazon) both host frontier model AIs on their own servers. Using OpenAI or Claude via their services should guarantee that your data is never stored on Open AI or Anthropic servers (they are, however, on MSFT or Amazon servers).

If you want total control over the server and own it yourself, then you can’t use frontier AI intelligence (they are just not available for self hosting and it would cost you a fortune anyway).

You could host open weights models, if you determine that what you need the AI to do doesn’t require the absolute frontier intelligence (and for most cases, you certainly don’t need frontier intelligence).

With that out of the way, here is a short explanation of what happens when you use AI APIs.

The anatomy of an AI API call

The stylized version of what happens is as follows:

Your application → API request → AI model → API response → your application

Suppose your application sends this: Read this customer complaint (that you provide in context) and tell me whether I should escalate it.

You may send additional data in context or not. Either way, all of the input will be tokenized, eg broken down into tokens (see What the hell is a token for more explanations). Parallel computations are then made on GPUs sitting in some distant data center.

As the model reasons through what you’re asking, it may decide that it needs additional information or that it needs to perform an action.

If you have given it access to tools that your company has vetted, it can use them to provide a richer answer. For example, it could decide that it needs use tools such as:

• get_customer_history(customer_id)
• search_company_policy(query)
• create_crm_note(customer_id, text)
• calculate_refund(order_id, amount)

If the only capability you give the model to retrieve customer information is a get_customer_history() tool, and that tool (programmed by whoever builds this for you) only returns information the user is authorized to see, then that is essentially the boundary of what it can do.

The caveat of course become as follows.

So the important point (which I make repeatedly throughout this Field Guide because it is important) is this:

What’s MCP, and what’s the difference with an API?

You will hear MCP a lot in AI land. One of my engineer friends even says “MCP is the new API,” which is even more confusing to non-tech people.

MCP (Model Context Protocol) is simply a standard way for an AI application to find and use external tools.

Remember our earlier section on tool access: a model by itself cannot check your CRM, search your company database or create a support ticket. You have to give it tools that can do those things.

Before MCP, developers typically had to build those capabilities into each AI application. MCP created a standard way to make such capabilities available for immediate use.

Think of it this way: if you have multiple applications that need a tool called “get_customer_history”, with different teams building each application, you certainly don’t want them to duplicate those tools in their own way. Instead, build the tool once and put it on an MCP server that other applications can use without rebuilding the tool.

So the distinction is roughly:

That is what my engineer friend means when he says MCP is becoming “the new API.” APIs aren’t disappearing. MCP often just sits one layer above them.

Then, do you need your own MCP server? It depends.

If you are building one simple AI application with a few custom integrations, maybe not. If you want the same company systems and tools to be available across multiple AI applications or agents, MCP becomes much more useful.

The good news is that lots of standard, generic tools to access popular SaaS software has already been built, so you can just use them.

For example, Google has published its MCP server for Gmail, Drive, Calendar, etc. You can use them to search emails, update meetings, browse Google Drive, etc.

Do you need to use open-weight models?

Open Weight Models (OWM) are AI models whose trained weights are made available for you to download.

What you download is effectively the model’s weights. To actually use them, you need computing infrastructure capable of running inference (called “inference infrastructure”). In a production environment, you would generally make that model available to your applications through an API.

This comes at a cost (think of open weights as a DIY option) but there are some very good reasons why you may want that control.

(Note that OWM are often loosely called “open-source models”, but technically this isn’t quite the same thing. Open-source AI implies broader access to things like the code and information about how the model was trained.)

Why use Open Weight Models (OWM) in the first place?

There are several legitimate reasons:

• Data confidentiality or sovereignty. You can run the model inside infrastructure you control, so that sensitive data never needs to reach the company that built the model.
• Customization. You have considerably more freedom to fine-tune, modify or optimize a model for a very specific workload.
• Offline deployment. A sufficiently small model can run on a laptop, phone, vehicle, factory equipment etc and give the answers you need without internet.
• Latency. If your application has a hard requirement for very fast response times, you can run the model physically close to the application calling it.
• Very high, predictable usage. At sufficient scale, operating your own inference infrastructure may cost less than paying somebody else per token (since you don’t pay per token anymore)
• Reducing model-provider dependency. You may want frontier AI APIs for some workloads while retaining the ability to use models you control elsewhere.

Data confidentiality deserves one big caveat though.

For example, AWS says that prompts and outputs sent through Bedrock are not used to train its models and aren’t shared with third-party model providers..

Similarly Google Cloud contractually states that it doesn’t use Customer Data to train or fine-tune AI models without permission.

Here, the data never hits AWS or Google Cloud servers, not the AI frontier labs’ ones. They companies don’t train Open AI / Anthropic models, so their commitment is that they basically won’t share your data to the frontier labs for AI training.

It’s a stronger guarantee than the frontier labs’ one indeed, althouth it is also not clear what your recourse would be if you found out your data made its way to the next AI model :)

So there are actually 3 different questions hiding behind the “I don’t want AI companies to have my data”:

1. Do I need to keep my data away from the company who built the model and can use my data to train its next model?
2. Do I need to keep it inside a particular cloud / country / region? (data residency constraint)
3. Or does it literally need to stay on physical infrastructure I own directly or control indirectly?

OWM gives you the option to solve #3, which a closed-weight API cannot. Depending on how you deploy it, it can also solve #1 and #2. More on this later.

Small Language Models (SLM) and composable architectures

You will find a lot of Small Language Models (SLMs) in the Open Weight ecosystem.

There is no standard definition of “small”, but we’re generally talking about models with a few billions to tens of billions parameters.

The smaller the model (parameter wise), the less resource it needs to work. Hence, if you want to put one on a phone, you really need a pretty small one.

For example, imagine a business workflow where you want AI to perform 10 different tasks:

• 1 is difficult and requires reasoning, for which you use a frontier AI model
• 9 others involve simple tasks like classification, extracting information, rewriting something into a particular format, etc

For these 9, you could use an SLM. You can therefore compose a system using different models for different parts of the workflow (referred to as a “composable architecture” or “hybrid architecture”).

The benefit isn’t simply reducing your frontier AI bill. It also means the application isn’t completely dependent on one model provider.

A useful design principle for your architect is therefore:

The catch: somebody now operates “inference infrastructure”

The OWM needs to run somewhere, called “inference infrastructure”.

You basically have four options:

For example, if you just rent an H100 GPU from AWS, AWS deals with the physical GPU, electricity, cooling and hardware failures.

But you are still operating the model.

Meaning that you still have to deploy the inference server, configure the model, manage capacity and scaling, monitor failures, patch vulnerabilities, secure the service, upgrade model versions, etc. That takes a specialized team to manage.

Alternatively, you can use a managed OWM endpoint where AWS, Azure, Google or another provider handles much more of that infrastructure for you.

Or simpler yet, you can call an API offered by the company behind the open-weight model. DeepSeek and Moonshot, for example, offer APIs for their models (which is then charged per token, just like a frontier model API)

How much does inference infrastructure actually cost?

It depends on the model. Let’s take a simple example: Mistral Small 3.1 (Vive la France!).

It’s an SLM with 24B-parameter. It handles text, images and tool calls, and has a 128k context window.

Mistral says it requires approximately 55 GB of GPU RAM to run a standard version (there are compressed versions), so for the sake of a simple illustration, let’s give it a single GPU of H100 with 80 GB of VRAM.

This is not necessarily the cheapest possible way to run the model, nor necessarily how you’d design a high-volume production deployment. It is just an example so we can compare costs with real numbers.

As of September 2026:

When you rent GPU, you pay by hour of usage. RunPod currently lists an H100 PCIe at $2.89/hour on its Secure Cloud. AWS currently lists a single H100 Capacity Block at $5.191/hour in US regions.

If you want to own that GPU infra, you’d buy it and set it up on a server. A single H100 GPU alone typically costs roughly $25,000–$35,000. So let’s round up the total capex at $40K to include all the rest (CPU, system RAM, storage, chassis, power supply, cooling, etc.).

Here, I chose a server rated at around 705W, so three years of continuous electricity at an illustrative $0.15/kWh adds roughly another $2,800 of electricity alone.

Then you’d add staffing OPEX since people will have to operate the thing for you.

But once you operate the model yourself, there is no token cost. Your cost is the infrastructure and the people required to operate it. Whether or not the investment makes sense depends on your use case.

This also gives a rough direction:

What are the trade-offs with OWM?

The useful way to think about OWM isn’t really as a list of pros and cons but more like a sequence of decisions.

First: why do you need an OWM?

Start with the hard constraint you are trying to solve.

For example:

• Data sovereignty: data cannot leave a particular infrastructure, cloud or jurisdiction.
• Proprietary data: you don’t want the model developer to ever receive your prompts or outputs.
• Physical control: the data cannot leave hardware you control for regulatory reasons.
• Latency: you have a hard requirement around response time.
• Offline: the AI needs to work without a network connection.
• Customization: you need to modify or fine-tune the model beyond what a managed provider allows.
• Economics: you have large, predictable volumes where operating inference is cheaper than using frontier models
• Strategic control: you need the ability to keep a specific model indefinitely or reduce dependence on a model provider.

Only then: decide who actually needs to operate it

If all you want is to keep data away from the company that created the model, renting a GPU in your existing AWS environment may be enough: you download the model and operate it there.

If you don’t even want to operate the inference software, a managed OWM endpoint may be enough.

If the data cannot leave your physical environment at all, then you may genuinely need on-prem infrastructure.

If you simply want cheaper tokens, you might not need to host anything. You could just call a cheaper model API.

These are the main choices you have when it comes to inference infrastructure.

Finally, spend time picking a model

If the above points you towards “I definitely need an OWM”, then the key is to pick the right one, since every OWM is different in capability.

There is no one-size-fits-all recommendation for this because it depends on the workload you need the AI to do.

And don’t assume that “bigger is better” in terms of parameter count.

In fact, a useful rule of thumb is:

Small models are cheaper and faster for simple tasks such as classification, data extraction, summarization, formatting, etc. Larger models are required as the task requires stronger reasoning, coding, broader knowledge, handling ambiguous inputs, better instruction following, etc.

Also check the license before you get too excited about a particular model. “Open weights” does not automatically mean unrestricted commercial use.

Here is the kicker that adds costs to your OWM choice: the only way you will know which OWM is actually good enough for your workload is to test them by building such workloads.

There is no real shortcut. Public benchmarks are useful for shortlisting models, but they are not your workload, your data or your quality threshold.

So the process is roughly:

Define the workload → define acceptable quality / latency / cost → shortlist models → test them on real examples → choose the smallest model that clears the threshold → size the infrastructure.

So, do you need open weights?

For most companies, probably not.

If a frontier API can do the job, meets your security and data requirements, and the economics are reasonable, it is usually the simplest and fastest way to get the job done.

You are effectively paying OpenAI, Anthropic, Google or somebody else to solve an enormous infrastructure problem for you.

My personal take is as follows (this is not investment advice):

If you are an enterprise operating in a regulated / sensitive environment (banking, finance, defence, government, etc.), use the composable architecture to keep optionality when it comes to model choice.

Some workloads may use frontier APIs through your enterprise cloud environment. Others may use managed OWM deployments on AWS/Azure/Google Cloud. The most sensitive workloads may justify OWM running on-prem.

If you operate in the mid-market or in a relatively unregulated sector, I would also use the composable architecture. But that doesn’t mean you need to use a bunch of OWM on day one.

Start with frontier / OWM APIs if they solve the problem. Only introduce managed OWM or self-hosting when there is a concrete reason to accept the additional complexity. If the architecture is truly composable, swapping models will not require the entire re-engineering of your workflow.

How to assess AI outputs objectively?

During a demo, you can eyeball the result of AI and decide if that particular output is correct. But if this process runs 1,000 times a day, how do you systematically evaluate AI outputs?

It is quite simple to evaluate an output where the possible values are known, closed-ended and where the right answer is also known. Think of assessing the answers from a multi-choice quiz where there is no “Other: type your own answer”.

However, evaluating outputs that are qualitative and don’t fall into neat categories is harder. Think of an AI that combines various other AI’s outputs to produce an investment memo.

To evaluate the latter, AI Evals borrow from a concept used in evaluating humans: “rubrics”. These are essentially tables where the relevant quality dimensions are broken down by the subject-matter-expert (in the investment memo example, it could be memo completeness, format, factual accuracy, sourcing quality, etc.).

Evaluating closed-ended outputs

For closed-ended outputs, you need a set of examples for which the correct answers are already known.

Say your AI has to perform part of a KYC screening process and classify customers as Approved , For review or Rejected. Give it historical customer cases whose outcomes have already been validated as correct and compare AI answers against those known outcomes.

You can then calculate something as simple as: The AI got 942 out of 1,000 cases right ~ 94% accuracy.

The wrinkle of course is that your initial data set would have to include cases that can reveal “false positives” and “false negatives”, so that you can also grade the AI on these. Eg:

• A false positive is a legitimate case incorrectly flagged.
• A false negative is a problematic case incorrectly approved.

There are other metrics like “precision” and “recall” but we won’t go too deep into the weeds here. All you need to know as a CxO for now is this:

Evaluating qualitative outputs

Things become harder when there is no single correct answer.

Take the investment memo example. You might have lots of examples, but you can’t cleanly derive a “this is a good memo and this is a bad memo”.

To solve this, we use rubrics: essentially a table that describes:

• The dimensions you care about and where the AI output will be graded
• A grading scale (say, from 1 to 5) with a clear description of what it means to give a score of 1, 2, 3 etc.

If this sounds like grading a human essay, that’s because it is.

Here’s a simplified example (a real rubric for investment memo would be much longer and detailed):

Then you could define the grading of Completeness like this:

For the sake of the example I am not breaking these further. In practice, you can already see how tricky this is. What’s “developed enough”? And “Adequately covered”?

The point is to make the grading criteria explicit enough that two different reviewers (be it human or AI) are more likely to arrive at similar scores.

You can check out Microsoft rubrics example here

It also raises the question: how do we know if the memo “is developed enough to convey the information expected from each” (score 5)?

This brings us to using LLM-as-a-judge techniques.

Using LLM-as-a-judge

The main way to automate evaluations for qualitative outputs is to use LLM-as-a-judge. Rubrics do the job of narrowing down the criteria enough, while AI will help grading at machine scale.

A good practice is to use a different model or model family to grade the output of the AI that produced the output. For example, if you gave GPT 5.6 the task of creating the investment memo, do not use GPT 5.6 as the judge.

To be clear: since API calls are stateless and “don’t remember you”, it’s not like GPT is magically remembering the work and giving itself a good score. The reason why it is best to use different models is AI bias: AI researchers found out that models may sometimes recognize and favor their own AI generations.

Also, if the rubric has been broken into narrow, very well-defined criteria, you might use an SLM rather than an expensive frontier model. The intuition is that if the scoring rubric has been simplified enough, a simpler model could be able to grade it.

So how do you decide which AI to pick as a judge, and what variations in evaluations you get?

The answer is simple: you have to test different versions, get the evaluation results and decide for yourself. When it comes to building AI systems, there is no magic shortcut: one has to build the damn thing in the first place and check.

So the rule of thumb for EVALS is this:

The anatomy of an effective EVAL system

Without EVALS, you may still manually inspect outputs, but that doesn’t scale and can be subjective.

The other big issue is you won’t know if your AI solution gets worse after changing a prompt, upgrading (or downgrading) model, or changing tool versions. It’s totally possible for AI output quality to regress when you update any of the above.

This makes EVAL design an important AI engineering skill, and something you should assess when picking an AI vendor.

At minimum, an effective eval system needs 4 things:

1. A representative test set
   This will come from you / your subject-matter experts. You don’t need an enormous dataset, for a vendor can augment one by using AI to generate additional examples (that’s called “synthetic data”). The point is that your seed data must be representative enough, as the synthetic data will inherit attributes of it
2. Clear grading criteria
   For closed-ended tasks, this means identifying the in/correct answers in the test data and the other key metrics you want to test for (accuracy, false positives, false negatives, etc.)
   For qualitative tasks, your subject-matter experts need to help define the rubrics that describe what “good” actually means.
3. An Eval harness
   This is the machinery that runs the evaluations at machine scale. At the core, a vendor should be able to explain for your specific workflow:
   • How AI outputs are graded (eg by deterministic code, LLM-as-a-judge, or something else?)
   • Why was that method chosen (you don’t always need an LLM to assess for example)
   • What the final scores / evaluations mean in the content of your business
4. A repeatable evaluation process
   Once the harness exists, you should be able to run the same Evals whenever you change an important part of the system and compare the results side by side. This is what makes evaluation scalable. As I said earlier, swapping for a better model doesn’t necessarily mean the AI output “grade” will automatically be higher.
   The only way to know this for sure is to run the EVAL with both models and see what’s coming out of it. This is called “EVAL regression testing”

So a good rule of thumb is this:

How to assess an AI vendor on their EVALS capabilities

A few practical questions will reveal quite quickly whether they know what they’re doing:

• Show me the Eval Set you would use for this workflow?
  • If they don’t know what an Eval Set is, this is not a good start. It should include your seed data as well as synthetic data
• What is being tracked as part of the Eval Harness?
  • Simply tracking the grades is not enough.
    • Remember that you want to be able to compare changes in output quality as various parts change in your system.
    • This means that you must track at the very least, for each EVAL test: the prompt, the context injected into the prompt, the model and its version, the tools it has access to, the output, the LLM being used (if LLM-as-a-judge has been used) and of course, the score
    • You must also track the rubric versions themselves, so that you know which eval set was used against which rubrics
• For anything graded by an LLM, how can you make sure the judge’s scores align with our subject-matter experts?
  • This is a trick question. The answer is simply that your SME will develop the rubrics in collaboration with their AI engineering team, and needs to validate the scores / outputs as the system is being built. The process should include SME validating the final thresholds of acceptance
• How will you detect AI drift?
  • Another trick question.
    • AI drift refers to the gradual difference between intended results from the initial system and actual results once in production even if nothing has changed in the system (which is slightly different than regression, which can happen when you change parts of the system).
    • This could happen because real-world processes may change (and hence, their prompts and relevant system config) but the system has not.
    • A vendor should sample approved production data periodically to run the EVAL test on it, compare against the baseline / thresholds agreed upon with your SME and if the system underperform.
    • Only then can they check if the reason is drift or regression (provided they have tracked all the versioning mentioned above)
• When the system makes a new type of mistake in production that was not tested against during development, and how will we know and how will it be added back into the Eval set?
  • A final trick question.
    • The answer should include the fact that the entire set (prompt, context, model etc.) that led to the mistake should be captured for a post-mortem with your SME.
    • Then it should be fixed accordingly, re-validated against an updated EVAL set and redeployed.

These are quite basic questions any experienced AI vendor should be able to answer. If they don’t, that’s a red flag.

On the risk of AI going rogue

If a good harness determines much of what an AI solution does, then why do people freak out about AI Agents going rogue? Can they actually “break out” of their harness?

Sometimes yes, but only under specific conditions.

Take the now famous “Hugging Face incident” of Jul 2026.

Hundreds of experimental AI agents participated in an unauthorized attack on Hugging Face infrastructure. An independent investigation by METR and Redwood Research provides a detailed reconstruction of what happened here: METR investigation OpenAI postmortem

The TLDR is that these were not ordinary API calls processing invoices or answering customer questions. They were highly autonomous agents running inside OpenAI’s frontier research infrastructure, asked to work on some extremely hard, sometimes impossible-to-achieve tasks.

In other words, they were operating in an unusually adversarial research environment, with unusually powerful capabilities, while being deliberately pushed to their limits. They found ways to bypass some of the controls, communicates with each other and eventually exploited systems outside the evaluation environment.

That is very different from saying that a mundane internal AI system will spontaneously decide to hack its way out of your CRM permissions.

Adversarial evaluation is just one of these things frontier labs do during post-training, where models are deliberately pushed hard to understand how they behave and where they fail before being released to the general public.

You are obviously not going to do that type of work just to process mundane business workflow. So the much more realistic concern is:

Originally published on baolannguyen.com.

India is a follower and is highly impacted because it has the single largest concentration of software engineers catering to the US as offshore development centers.

Vietnam is always a few years behind India in terms of trends.

So if we look at what’s going on in India, can it inform us on what’s next for Vietnam?

India: when AI meets outsourcing

India built a large industry around selling outsourcing services to the rest of the world ( ~ 250 $B in software service export alone, or about 50% of Vietnam’s entire GDP!), primarily catering to the US.

AI is now directly threatening that outsourcing model.

By August 2026, Reuters reported that Indian IT giants including TCS, Infosys, Wipro, HCLTech and Cognizant were renegotiating the model as customers demanded substantially more productivity for the same money.

Contracts start to be priced around business outcome delivery rather than hours billed. And some work is simply disappearing because customers can now do it internally with AI.

A few quotes from leaders of some of the largest outsourcing companies are telling:

• “Clients were demanding the same work for 25% to 30% less while expecting faster delivery and higher productivity.” - Sandeep Kalra, CEO of Persistent Systems

• How fast and how much more we are able to go ahead of the (revenue) deflation will determine the growth going forward." - K. Krithivasan, CEO of Tata Consultancy Services (TCS)

• Some rivals are factoring in productivity gains of 70% to 80% over five to seven years and guaranteeing prices despite rising chip costs… Clearly, there is a ton of competition out there, and our competition at times is doing irrational things." - Mohit Joshi, CEO of Tech Mahindra

And investors have turned bearish. The Nifty IT index hit a three-year low in June. Wipro had fallen almost 30% for the year by August 2026. TCS announced more than 12,000 layoffs in 2025 as it restructured its workforce.

Is this the beginning of the outsourcing apocalypse?

I don’t think so. In fact, some pockets of engineering are benefiting from this.

Indian GCCs are growing on the back of AI

Global Capability Centers have been a thing in India for quite some time. In 2019, there were about 1,400 GCCs employing 1.4 Million people (probably around 70% engineers or similar, making an average of 700 engineers / GCC). In 2026, there are about 2,200 GCCs employing a whopping 2.4 million people (or ~760 engineers / GCC, on average).

What do GCCs do? They build products and in-house AI capabilities for their parent companies, but for a lower cost than doing it at home (eg, the US for most cases).

A few examples:

• Walmart’s teams in India work on agentic AI and core infrastructure.
• JPMorgan is hiring another roughly 1,000 technology people into its India GCC for cloud architecture, cybersecurity and AI data pipelines, while saying AI has reduced staffing needs considerably in some other parts of the bank.
• Costco announced its first India technology centre in 2025 , initially for around 1,000 people.

So there are really two things happening at once: outsourcing giants are getting smashed, while GCCs keep growing (note that some outsourcing companies also provide GCC service, but that’s another story).

And it is quite logical: enterprises want to keep the AI IP being produced, the bespoke AI harnesses being built, etc. Why outsource that strategic capability to some external vendor?

Will Vietnam GCCs also benefit from this tailwind?

Maybe a little. But definitely not in a way India is benefiting.

Vietnam has very few GCCs relative to the 2K+ in India. To name the main ones:

There are a few others with less than 1K engineers, but that’s nowhere near India. Even when you account for the relatively smaller pool of engineers in Vietnam (somewhere in the half million mark, plus ~ 50K every year).

For the sake of comparison, Indian GCCs basically employ about 50% of the total ~5 Mil SWE workforce in India.

Vietnam mostly has traditional outsourcing. What are the big players doing?

FPT: the canary in the coalmine is rebuilding the machine

FPT stock dropped in 2026Q1 by 25%. It is widely reported that foreign investors dumped about 400$M worth of stock over 2 months, mostly due to the same fears that cripples India. The 2026Q1 sell off was unprecedented and FTP now trades at 40% below its ATH of 2025.

However, here is what FPT reports and statements also tell us.

Its foreign-market IT-services business grew ~ 14% in 2025. Its average production workforce grew 12% to almost 30,000 people. Revenue per employee increased 7.6%. We don’t know how they internally account for this and they surely have to show good stuff to the market. But they also emphasize a point: FPT leadership calls AI a “structural reset.”

FPT wants to turn its engineering workforce into AI-augmented engineers. According to the company reports, the ambition is to move from “Time & Materials” (e.g. traditional outsourcing) toward managed services and end-to-end solutions. It is building its own fleet of AI products and wants AI-first work to eventually account for roughly one-third of revenue.

CMC is responding from another direction.

In May 2026, it reorganized its Technology and Solutions business, consolidated consulting capabilities and made AI transformation the central axis of its 2026–30 strategy. The stated model is end-to-end as well: identify the business problem, define the roadmap, implement the technology and measure performance.

Sounds like India to me, and for good reasons.

Putting these big players aside, what else does the local labor market already tells us?

What Vietnamese hiring trends tell us

I have been in the tech sector in Vietnam for over a decade now, and my continuous conversations in the industry currently tells me that:

• QA / Testers are being laid off or experience increasing pressure
• Fresh IT graduates struggle to find jobs, which now require AI engineering skills they have to learn by themselves
• Outsourcing companies are still in the process of reshuffling their org, with mounting anxiety among staff
• Customer demand for ai solutions beyond chatbot and POCs has yet to materialize
• Outsourcing company owners are questioning their own business model and trying to find the best way to pivot

These are insights from anecdotal conversations in the industry. For what it’s worth, here is what IT Viec says about SWE hiring trends.

• Overall IT postings fell 9.3% in Q1 2026)
• But AI/Data/ML positions were up 96.6% from a year earlier.
• By Q2, overall IT postings were growing again, up 4.4% from Q1.

Earlier data showed the same split. In Q3 2025, AI/Data/ML jobs were up 42.1% year-on-year while non-AI IT jobs were down 3.1%. QA/QC postings were down 9.2% quarter-on-quarter.

There is also evidence that companies are becoming less willing to add junior and middle-level headcount. An IT Viec survey found a 7% decline in junior and middle hiring demand in the second half of 2025, with 25% of surveyed companies saying AI-driven productivity was one reason for freezing hiring or reducing headcount.

Is it all due to AI? Probably not, as Vietnam is also bracing for economic challenges.

But it does look remarkably similar to what is appearing in much better US data. In fact, Stanford / ADP has done a pretty thorough, statistical research on the impact of AI on US jobs.

Here’s what it says.

What US payroll data tells us

ADP (a huge US payroll company) has the interesting proprietary around labor that is probably more relevant than government statistics: role, tenure, payroll, and overall labor trends. Stanford and ADP partnered to produce an quite rigorous study and here’s what they found.

As of June 2026, Stanford found no widespread economy-wide employment displacement associated with AI.

But among workers aged 22–25 in highly AI-exposed occupations (eg including software engineers), employment was around 19% below where it would have been had it followed employment in less-exposed jobs (eg say plumber or electrician).

Software developers are one of the clearest cases.

Stanford’s software-developer data shows large declines among early-career workers, modest declines in the next-youngest groups, and increase in older groups.

In a nutshell: less hiring of junior engineers, more hiring of senior ones. For the mid-level ones, there’s also a slight and steady increase in hiring.

Stanford explains it as follows. Employment is weaker for younger workers doing work dominated by knowledge that can be documented and taught (eg work now done by AI). Experienced workers whose value depends more on tacit knowledge and experience acquired through practice have a better chance (eg AI cannot entirely replicate their output yet).

That aligns with Sequoia’s piece on “the next trillion dollar company”. AI excels at reasoning, humans excel at judgment. Unfortunately for young graduates, it takes years of experience to acquire good judgment.

Then, what does the “engineer of the future” look like in a post-AI era? And is there even a future for young junior ones?

The post-AI SWE

Huyen Chip has popularized the term “AI Engineer.” Lots of things have changed since the publication of her work, and “AI engineering” is splitting into various forms as we speak.

Some AI Engineers build infrastructure. Some build agents. Some fine-tune models. Some are basically application engineers who use LLM APIs.

Here is what LinkedIn has to say about hiring trends in that broad category.

AI Engineer has overtaken Machine Learning Engineer as the most common AI role on its platform. More interestingly, Forward Deployed Engineer is already the third-most-common AI occupation in job postings (if we ignore the misc category “other AI occupations”).

And here’s what an FDE does. S/he collapses the SDLC into one neat loop and answers the following questions by herself.

• What problem are we solving?
• What should the model do?
• What should the model not do?
• What context and data does it need?
• Which parts should remain deterministic software?
• What tools should the agents have?
• How do we know whether the output is actually good?
• How does this fit into the client’s existing systems?
• How do we help the clients monitoring agentic work?
• Can we deploy it securely?
• How do we evaluate AI outputs against the clients’ goals?
• How do we make sure end users will use it?
• What’s the next iteration of the product?
• Repeat.

Under that view, the vast majority of the actual code-writing is delegated to competent AI harnesses. SWEs spend most of the time on domain-specific problem solving, talking to end users, AI architecture design, managing agents, etc.

In other words, coding ability used to be scarce but has now become a commodity. The scarce capability is the combination of the above.

I am not saying that every VN SWE must become an FDE. FDE is probably at the extreme end of the spectrum and one should not expect every VN SWE to sit next to the customer and speak their language. But it is a useful direction in terms of “what the destination looks like”.

In a sense, FDE is the new full-stack engineer.

There are other possible directions too. One that intrigues me the most is the 1-man-army direction.

Will the 1-man-army rise?

Forget about working at a company for a moment. In fact, probably one third of Vietnamese engineers that I know have tinkered (or think about tinkering) with the idea of starting something of their own.

Let’s flip the entrepreneurial approach on its head and ask: now that AI can give “1,000 geniuses in your pocket”, could you build a company without VC-backing and make 1 $M a year instead of the VC-threshold of 100 $M / year?

Or let’s say 100 $K, not even 1 $M.

Making 100 K$ / year doesn’t sound fancy. But I think it may be more practical than “starting up”. And if one can pull that off with no employees, that’s not a bad proposition.

Here are some extreme outliers for inspiration, who all started post-GPT. To be taken with a grain of salt.

• AudioPen by Louis Pereira (more here and here)
  • Started in 2023.
  • Background: solo builder/product tinkerer; had tried multiple projects before without a major commercial success. He did already have roughly 10,000 X followers, so he was not starting from nothing.
  • Sells: an AI voice-to-writing tool that turns rambling speech into structured text.
  • Numbers: around $15k/month revenue in 2024, later reported closer to $20k/month, with no employees.
  • Caveat: founder-reported revenue; prior audience helped launch distribution.
• AIDesigner.ai by Tyler Yin (more here)
  • Started in 2025.
  • Background: designer/developer and former product professional; no clear evidence of a previous major startup exit or successful company.
  • Sells: an AI product for generating and improving UI designs/code inside modern coding workflows.
  • Numbers: roughly $9k MRR and $60k+ cumulative revenue by August 2026, operated by one founder with no employees.
  • Caveat: still a young business, so durability and true net profit are unknown.
• 3AK Track by Christian Rac and Braylin Byrd
  • Started around early 2026.
  • Background: two college track athletes with no traditional coding background and previous unsuccessful small business experiments.
  • Sells: a subscription app for track-and-field athletes, built largely with AI/no-code tools.
  • Numbers: roughly $11k MRR, about 3,200 active subscriptions, run by the two founders and no other outside help at the time reported.
  • Caveat: they already had around 150,000 combined social-media followers, so distribution was a major pre-existing asset.
• Wrestle AI by George Lampropoulos (more here)
  • Started in 2025.
  • Background: teenager with no conventional software-development skills and no prior significant entrepreneurial success.
  • Sells: AI-powered mobile apps for wrestling/fight-related users, built largely through AI app-building tools.
  • Numbers: reported $131k+ revenue over six months, with Wrestle AI reaching roughly $23k MRR and combined apps reportedly peaking near $39k/month.
  • Caveat: revenue figures are largely reported by the AI development platform used to build the apps; he also had a creator/commercial partner, so this is not a pure one-person business..

Surely, this is anecdotal evidence and loosely audited. It only proves that it is possible, and suggests that the minimum organization required to generate some form of profit appears to be just you and AI.

But notice how they “made it”:

• They had distribution - a form of audience to monetize
• Or some domain expertise - eg Christian & Braylin, Tyler
• And understood how to sell

There’s probably more variations of success, and 1-man-army may become 3-(wo)men-army. But the idea is the same: own your independence and make enough for the lifestyle you want, by leveraging AI.

And no matter the choice - to work for someone else or for oneself - a similar mindset and skillset is required. It reads something like this:

I think that building such a skillset is a pretty safe bet for the foreseeable future.

A darker view of the future

As the Stanford study starts to show, companies hire less junior people, favoring the more experienced ones. Since every company has an economic incentive in doing so, everyone does it, until the hard question of “where is the succession bench?” comes up.

Which brings us to the following scenario: a generation of young computer science graduates gets partially hollowed out by AI.

But what’s the probability of that happening?

Of course, no one knows the exact number. The future cone of possibility gives us a useful heuristic to put some rough probabilities on events. They range from preposterous (say, less than 5-10% of happening) to possible, plausible and probable, with increasing probability (say, > 75% change for ‘probable’).

Where would the “AI leads to a hollowing out of junior engineers” scenario fall?

Which is akin to saying: “Unless companies have a new economic incentive to keep hiring juniors (or are forced to, by state intervention or similar), then hiring less juniors is an outcome to expect, which leads to a hollowing out of that group over time”.

A quite sobering thought.

Bringing it altogether

Based on all of the above, we can reasonably predict that the following things will happen in Vietnam, probably over the next 12-18 months:

• Lower employment opportunities for junior SWE.
• A period of lower demand from international clients for traditional outsourcing services, that may be offset by new demand for AI solutions, but creating profitability pressure in the meantime for small firms.
• Hiring demand for FDE-like engineers increases progressively, but not strong enough to absorb the number of SWE looking for jobs, depressing salaries overall.
• Mindset and skillset across the software engineering profession change to adapt to the new reality.
• Likewise, new economic and delivery models in the outsourcing industry appear to “follow the money”.

In other words, brace for impact. Looks like the road ahead is going to be very bumpy.

But as the saying goes, every challenge is an opportunity in disguise. We just don’t know what clothes they wear just quite yet.

Further reading

• Survey on Computer Software and Information Technology Enabled Services Exports: 2023-24
• Zinnov-Nasscom India GCC Landscape Report 2026
• IT services for foreign markets
• Vietnam IT Recruitment Market Overview in Q3-2025
• Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence
• New LinkedIn Research On The AI Divid In 2026

Originally published on baolannguyen.com.

The goal is to clear away enough of the terminology and hype so the rest of the AI conversation starts making sense.

Do you need AI or merely automation?

The 2 are often conflated and everything is suddenly packaged in an AI wrapper those days.

It’s important to tell the difference because automation is plain vanilla software (e.g. cheap) while a real AI workflow is way more complex and will just cost more (not because of AI itself but all the apparatus that goes around it, as you shall see as you read the entire post).

The distinction is subtle but can cause some confusion.

Here’s an example: guess if you need AI or only automation for this process.

You guessed it, the entire process can be automated with plain vanilla software.

There is very little ambiguity as to what should be done, and can be thought of as “if X, do A. Else, do B” where both inputs and outputs are predictable (eg the customer can only select from a few options in the drop down list and we know them all. We also know what to do for each option).

Some people may think you would use GenAI to write the email back to the customer. But to the extent that it’s the same for every customer, a template will do.

Now here is another example.

The reason is as follows: the inputs will need some interpretation, because not all business cards look alike (exotic names may sometimes be taken for company names, job titles may have special characters or even be in a different language, etc.). GenAI can handle all the variations in one go very effectively.

Here’s how you would proceed:

• Take a picture of all the cards and throw them on your favorite cloud (let’s say you are using Google Drive)
• Write an automated script to read such files one by one,
• Send that to an AI API for extraction with a pre-determined structure (eg you always want first name, last name, email, title, phone number etc),
• Get the data back,
• Programmatically add them one by one in your Google Contact lists (which hopefully you have sync’ed with your phone list). If you prefer, you can send that to a spreadsheet, a CRM, etc.

Everything except the reading is deterministic. You don’t need an AI to “guess” where “first name” goes into your CRM or contact list. That’s something that can be programmed in code.

These 2 simple examples illustrate a simple rule of thumb when it comes to AI vs Automation:

Which brings the question: are there cases where AI should handle the whole process and where you have no deterministic code at all?

Barely, at least in a business context. But I am getting ahead of myself, for this deserves a bit more explaining. More on this later.

For now, please don’t pay an AI price tag for a simple automation, and don’t look to AI to solve deterministic problems.

What the hell is a token?

Sooner or later, someone discussing AI costs with you will start talking about tokens, so you need to know what this is.

AI models do not read your prompt as words in the way you do. Before the model processes the text, it gets broken into small pieces called tokens. A token can be a whole word, part of a word, punctuation or even a small piece of formatting.

For English, a useful rough approximation is this: 750 words is roughly 1,000 tokens. It varies by language, model and type of content, so this is just a heuristic to keep in your back pocket.

What this means is that if your prompt is a full A4 word doc (~400 words), it will be about 500ish tokens. If you’re sending a whole contract, it’d be trivial to get the total word count and calculate the approximate equivalent in tokens.

We don’t need to get into the weeds of “But why use tokens and not words?”. What truly matters about these tokens is that you will pay based on the amount of token you use, and this is where it gets a bit confusing.

Here is why.

When an AI model receives your prompt, it will process them in parallel (not token by token). In terms of compute, that’s cheap and fast.

Generating an answer is different. It will produce tokens one after the other (just like we write one letter after the other), with each new token depending on what came before it. Producing 5,000 tokens therefore consumes materially more resources than simply reading another 5,000 tokens.

Hence, input tokens are always cheaper than output tokens. It’s just a function of the resources required to process those things.

Therefore, you can guess that every interaction with an AI model will have a different price tag (even for similar prompts, although the difference will be small). Not to mention that under the hood, the AI model might do different things (e.g. perform a web search to verify something live for example) that adds more tokens.

This is also why “$X per million tokens” tells you little about what an AI application will actually cost to operate. You have to know the type of workload you will need AI to process, build it, test it and check the cost. One can guesstimate, but the best estimate with AI is to try the thing out.

Open AI token pricing: Anthropic token pricing:

Putting on your CFO hat, you will then ask “How do we don’t blow up the budget with all this AI stuff?”

The answer is that there are plenty of ways to manage this. Good AI engineers can control how much information is sent to the model, limit output length, cache information that doesn’t need to be processed repeatedly, use cheaper models for simpler work and reserve expensive models for tasks where they actually make a difference.

So don’t despair at the lack of predictability in cost.

No one would use it if it was really impossible to limit unexpected costs. And if you suddenly see an alert on your AI bills, AI is probably not to blame. It’s probably because someone forgot to engineer the right boundaries around AI.

Model weights and the concept of machine intelligence

You will hear of AI models having 7 billion, 70 billion or trillions of “parameters”, often more loosely referred to as its “weights”.

You do not need to understand the mathematics behind (although it’s fascinating) but understanding how intelligence is produced from these weights is actually useful.

At its core, an AI model is just a huge (or many huge) file(s) with lots of numbers inside. Those numbers are the result of training, where the model was exposed to a large amount of data (essentially, a curated version of the Internet, plus some more) and the training process has encoded such knowledge using numbers.

Then, how does AI reason or behave intelligently based on just a bunch of numbers?

An (imperfect) analogy will help: our biological brain.

It has billions of neurons connected to each other. No individual connection contains our understanding of what a dog is, how to speak Vietnamese or how to calculate an EBITDA margin. What we know and what we can do somehow emerge from patterns of activity across various regions of the brain, which are themselves learned by experience.

Ai models share the same property: intelligence is not sitting inside one particular weight. It results from what the network has learned during training. No weight says “Paris is the capital of France”. However during training it has encoded the understanding of Paris, France, capitals, geography and the relationships between these concepts.

So when you send a prompt with some data, it is not “ingesting” this new knowledge into its weight in order to reuse it later. People have this fear that using AI will expose their proprietary data (which is definitely a risk). But it just doesn’t happen as “I give my data to AI, and AI incorporates it in its knowledge”.

We will talk about “training on your data” in a bit. For now, all your need to remember about AI model weights is that:

At the core, it’s just a file with numbers, which are encoded representation of everything a model has learned

Intelligence is not encoded or pre-programmed, but emerges from these representations.

AI models don’t incorporate your data and prompts into their own weights just because you sent it. Its weights are fixed

“AI is a prediction machine.” Yes, but there’s more.

You will often hear that AI is “just predicting the next token”, usually followed by the conclusion that it cannot really reason and therefore cannot be trusted.

The first part is technically true. The second does not follow from it.

At the very bottom of a language model, generating an answer still involves predicting one token, then another, then another. But describing AI that way tells you very little about what the resulting system is capable of doing.

There are a few reasons for that.

First, the model you use today is not simply the result of pre-training on a large pile of data. It has also been “post-trained”.

For example, post-training can teach the model to follow instructions, reason through particular classes of problems, respect certain constraints and learn how and when to use tools (like web search, browsing a website, writing some code, etc.).

As a result of pre and post training, you get a model who is intelligence and capable of doing stuff.

Which brings us to another concept you will increasingly hear about: the harness.

A harness is the software sitting around the model that lets it do more than simply receive a prompt and return an answer. As the name implies, if the AI is a horse, the harness allows you steer it towards where you want to go.

OpenAI describes the agent loop inside Codex (its harness for coding) as:

In other words, traditional software wraps around the AI model.

So when you ask Codex to fix a bug, the model is still generating tokens somewhere underneath all of this. But the overall system can now perform a loop that looks roughly like this:

read the problem → inspect the code → decide what to change → edit files → run tests → observe what happened → correct mistakes → repeat

Calling that whole thing “a next-token prediction machine” is therefore incomplete.

This also explains why the same underlying model can behave very differently depending on the harness you put around it.

Give a model a prompt and no tools, and it can produce basic answers.

Give it access to your documents, databases and APIs, a way to remember where it is in a task, rules about what it may or may not do, retries when something fails and a loop that lets it inspect the consequences of its actions, and you have built something much more powerful.

You can build these kinds of harnesses for your own business processes too.

For example, an insurance-claims harness might give the model access to claim documents, policy records and internal procedures, let it request specific checks, define when a human must approve something, limit what systems it may change and keep track of where each claim sits in the workflow.

For a non-tech CEO, it doesn’t matter much that there is a token prediction mechanism inside the engine. What matters is understanding what needs to be done using the engine, determine what harness should be used or built around it, and steer it towards the goal.

Demystifying “context.”

It is one of those simple words that became complicated once the AI industry discovered it could put the word “engineering” after it. Context engineering sounds fancy indeed.

But the basic idea is dead simple.

You now know the AI model weights are fixed. It has knowledge, reasoning abilities and access to tools. But it doesn’t mean it already knows all the things it needs to know to perform the task you need it to do.

Imagine asking a new employee: Please review this proposal and tell me whether we should accept it.

Unless they know who the customer is, what you are trying to achieve, your pricing rules, previous negotiations, what your CFO has already approved, what risks matter etc, they will probably give you the wrong answer.

AI is no different.

If you send it a contract and ask: Should we accept this contract as-is?,

the model may understand contracts very well. But it does not know your company’s risk appetite, your standard clauses, what was previously negotiated with this customer, etc.

All of that is context.

Context can therefore include almost anything useful for the task:

• your prompt;
• the conversation so far;
• documents;
• company policies;
• database records;
• previous decisions;
• examples of good answers;
• output from other AI;
• results from a web search;
• instructions about what the AI is allowed to do and forbidden to do;
• etc.

Can you give infinite context? Of course not. The limit is called the “context window”.

Many recent frontier models have a huge context window of about 1 million tokens. Using the rule of thumb of 500 tokens per A4 page, that’s a whopping 2,000 pages! That’s not exactly true (there are other system instructions, tools etc that also eat up tokens) but you get the idea.

There are wrinkles in the way AI models use their context. Shoving in 1000 pages of documents and asking AI very specific questions is probably not going to yield the best results (there is a phenomenon literally called “lost-in-the-middle” where AI models struggle with information in a large context, especially when buried in the middle).

This is where “context engineering” comes in.

Doing that once in a chat session with ChatGPT is easy. Doing it systematically within an automated process, securely and consistently across thousands of requests takes some real engineering muscle.

But there is no magic, it’s just a data/software engineering problem with a fancy name.

So when some consultant tries to dazzle you with “context engineering”, you’ll know they’re probably trying a little too hard.

Do you really need to “train AI on your data”?

Sooner or later, someone will tell you that to make AI useful for your company, you need to “train it on your data”.

Your first reaction should not be “great”. It should be: What exactly do you mean by training?

Because people use that phrase to describe several completely different things, with very different levels of complexity, cost and usefulness.

You already know from the section on model weights that actual training means changing or adapting those weights. A model is exposed to data, the training process adjusts its internal parameters, and the resulting model behaves differently because of it.

Now imagine you upload your employee handbook to ChatGPT and start asking questions about it.

No model weights have changed. The model has not “learned your company” in the training sense. You have simply given it information to work with.

That is context, not training.

A slightly more sophisticated version of the same thing is something called RAG, or Retrieval Augmented Generation. Instead of putting your entire employee handbook, pricing manual, SOP library and every policy you have into the prompt every time, you keep those documents somewhere outside the model.

When someone asks a question, the system searches for the pieces that appear relevant and puts only those pieces into the model’s context. Then it answers based on that.

Again, no weight has changed, and RAG is not AI training (it’s just called “retrieval”).

You will hear “vector database” mentioned a lot in these retrieval conversations. All you need to remember for now is that it’s a different kind of database.

Then the last version of “AI is trained on your data” which is actually not training is as follows.

A vendor may build an application around GPT, Claude or another frontier model, add some permanent instructions, connect it to your company documents, perhaps store some information about previous interactions, and suddenly the AI appears to “know your company”.

That’s usually how they pull off demos within a few hours. It’s still not training on your data, although from the user’s point of view, it can certainly feel as though the AI has been trained.

But configuring instructions is not training either.

Now, there are cases where they may actually mean “AI trained on your data” and it may be necessary for your business case.

The most common method is fine-tuning.

Fine-tuning changes (or adapts) a model’s weights using additional examples so that it becomes better suited to some particular task or behavior.

Suppose you have a highly specific classification problem and thousands of good historical examples showing how experienced employees handled it. You give a general model clear instructions and relevant context, but its behavior is still inconsistent in ways that matter to you.

Fine-tuning can solve that problem. By adapting/changing the model weights, you are essentially baking in a new behavior inside the AI model itself, which it will use consistently on every task that requires such behavior. If finding factual knowledge is more important than behavior, RAG / retrieval is generally a better approach.

The harder question is whether you should bother with fine-tuning, as this can come at a steep price tag. You’d also need:

• Enough high-quality data to train on,
• A clear idea of what behavior you are trying to improve,
• and some objective way to prove that the fine-tuned model is actually better than a plain-vanilla AI model

Otherwise you may simply spend money creating your own slightly different version of a model that was already good enough.

A useful rule of thumb is this. Consider fine-tuning when

• you have a recurring, well-defined task, enough high-quality examples of the desired behavior,
• and prompting and context alone are not getting you where you need to be,
• and there is enough economic value to justify the additional complexity.

So next time you hear “train AI on our data”, you will be able to mentally separate 4 possibilities:

1. Give the model your information as context. Not training, pretty standard engineering skills required.
2. Retrieve relevant information from your company systems when needed. Usually RAG or some other retrieval architecture. Still not training
3. Store useful information over time and retrieve it later. Using standard memory. Still not training.
4. Change the model’s weights using your examples. Now you are actually training on the company data usually with fine-tuning techniques. This takes more specialized skills than standard engineering

All 4 can be useful for different use cases. But just make sure you don’t end up paying the price of “AI training” for any of the 3 basic cases.

Here is a simple bullshit detector when a vendor is talking about this with you.

If someone says:

“We need to train the AI on your data.”

ask:

Are you actually changing or adapting the model’s weights?

If the answer is no, fine. Ask what they are really doing instead.

If the answer is yes, ask:

What problem requires changing the weights rather than using other techniques?

Then listen carefully to what they have to say and decide for yourself if the vendor is trying to bullshit you, or if they have a genuine reason to believe you really need AI to be “trained on your data”.

Originally published on baolannguyen.com.

No jargon and no hype, only the straight truth from someone who actually built and deployed real-world AI solutions.

Click on the links to go directly to the relevant topic.

Part 1. Demystifying AI

• Do you need AI or merely automation?
• What the hell is a token?
• Model weights and the concept of machine intelligence
• “AI is a prediction machine.” Yes, but there’s more.
• Demystifying “context.”
• Do you really need to “train AI on your data”?

Part 2. How AI systems actually work

• Demo vs AI wrapper vs real AI solution
• APIs, consumer AI and data privacy issues
• The anatomy of an AI API call
• What’s MCP, and what’s the difference with an API?
• Do you need to use open-weight models?
• How much does inference infrastructure actually cost?
• How to assess AI outputs objectively?
• On the risk of AI going rogue

The below index is for reference only and may change by the time of publication

Part 3. How to think of Org Design in a post-AI era?

• HITL, HOTL and HOOTL: how to think of the human role in AI workflows?
• If you can manage people, can you really manage AI agents?
• Should your AI org design just be a replica of your org chart?
• What does an AI engineer actually do?
• Do you need a CAIO?

Part 4. ROI, risks and rewards of AI

• Should you build it, buy it, or configure something you already have?
• How to think about AI ROI (financials AND non financials)
• Variables your CFO should know when calculating AI ROI
• AI improves fast. Should you just wait and see?
• Cybersecurity and AI

Originally published on baolannguyen.com.

Activity in M&A is always a good signal to track how “hot” the industry is. The data that I have shows that the Vietnam software industry is well past its golden age. 

Some disclaimer first: I spent time scrapping and analyzing this data based on publicly available information and using AI. Since not all deals were publicly announced, it is inevitably incomplete. 

Also, lots didn’t have data on how much the buyers paid. So I will stick to deal counts as a proxy for activity. 

Here’s what the analysis shows:

• I found 35 M&A publicly documented transactions in the software outsourcing space since 2015 in Vietnam.
• Japan bought a whopping 70% of all deals!
• Peak M&A time (eg with deal count above the average of 3.5 / year) was 2018 and 2019 with 11 deals (about ⅓ of the total deals of the last 10 years) as well as 2023-4 with 9 deals (25% of the total of the L10Y)
• 2026 saw a single transaction and I was generous to include it despite it being essentially a parent group buying its own subsidiary 

Anecdotally, casual conversations with friends in M&A also point in the same direction: there’s just not much happening in outsourcing this year, and multiples offered are way lower than what they used to be (some companies traded at 12-15x EBITDA c. 2023-4. In 2026 you’d be lucky to get even a 7-8x).

This can be seen in hiring and salary trends too.

Being an operator in the tech industry in Vietnam, I naturally track these, and they are definitely down. It was not unusual to pay 2,500-3,000 USD salaries for Full Stack Engineers or Mobile Engineers, depending on seniority. They were the most sought after engineers. 

Fast forward 8 years later: these labels have aged badly and one would be hard-pressed to get that type of salary. The AI Engineer is the new Full Stack Engineer, and even then salaries have adjusted down given the lack of actual AI experience (which is normal given how early we are in the AI wave).

It is a hard reckoning for Vietnamese engineers and outsourcing business owners. Similar trends are happening in India.

Bearish trends in the global epicenter of outsourcing: India

India’s IT-services industry generates roughly $315 billion a year, of which software outsourcing is ~ 250 $B. It is huge (about 50% of the entire Vietnam GDP). Lots of the Fortune 500 companies have outsourcing partners or captive GCCs in India (Global Capability Centers). The US takes up about half of all outsourcing services produced there.

But AI has been shaking up the industry quite a bit: as of Aug 2026, the Nifty IT index fell by 20%, erasing ~ $70 billion in market value. Why? Here’s what the biggest industry players are saying. 

• “Clients were demanding the same work for 25% to 30% less while expecting faster delivery and higher productivity.” - Sandeep Kalra, CEO of Persistent Systems

• Roughly 80% of its finance, HR and other business-services contracts are now outcome-based." - K. Krithivasan, CEO of Tata Consultancy Services (TCS)

• Some rivals are factoring in productivity gains of 70% to 80% over five to seven years and guaranteeing prices despite rising chip costs… Clearly, there is a ton of competition out there, and our competition at times is doing irrational things." - Mohit Joshi, CEO of Tech Mahindra

Of course, with such an exposure to the fastest moving market in the world when it comes to AI (US), one would expect that India gets hammered. 

Vietnam has a much smaller outsourcing industry (about ~7 $B / year only, or less than 3% of India) and a smaller exposure to the US (about 25%). However, the trend is quite the same.

Taking FPT numbers (one of the big boys in the industry), its stock fell by 25% in 2026 Q1 alone. Foreign investors have essentially exited their position as a way to price in the new reality and risks brought by AI. 

FPT leadership was quick to tell the press and investors that it is making efforts in adjusting its business model, with a target of getting as much as ⅓ of its revenue from AI-driven sources (!). 

Japan is still where ~ 40% of their clients are, which makes an uneasy concentration. Surely, they won’t in-shore and build AI by themselves like the US do (they just don’t have enough people, let alone qualified engineers to do the work), so FPT may still have a few good years milking that cow. 

But not every outsourcing firm is a juggernaut like FPT who can withstand adverse market conditions.  

Unfortunately, I think that most outsourcing firms face a very hard fork:

• hope that what they were doing before AI will still work and predictably go out of business, 
• drastically cut staff now to extend the runway while servicing existing legacy contracts and finding new AI demand (which doesn’t come easily), 
• or call it a day and decide to exit the industry in an orderly fashion. 

The sentiment is overall bearish, although there is still a base case and a bull case to be made. Let’s go through them now.  

The base case: the buyer owns demand / distribution

A buyer who has existing customers and just needs more engineers for a certain type of work (industry specificity, legacy migration specialist, etc.) may not be under immediate pressure to transform the business. 

There are at least two variations of this. 

1. The buyer needs to serve existing customers.

This could be a larger outsourcing company acquiring a smaller one. Think of the Alten-VMO deal (Alten is a listed company in France, and bought VMO in 2024). In these cases, the portfolio of clients you bring is not the major consideration in the deal. The ability to deliver work for existing clients is (generally in the APAC region). 

2. The buyer owns the companies for which it needs engineering capability

Think of an investment firm who has invested in a large portfolio of companies (commonly referred to as their PortCo) and want to bring more tech support to them. This is definitely trending up in the US among big and mid-sized PE firms, and increasingly lots of the support has to do with AI implementation.

Here, the demand is captive.

As the investor buys more companies, engineering works expand. Unless the fund is large enough (e.g. in the billions and not millions), the size of the engineering team would generally stay quite modest. 

Geographic proximity plays an important role here, so as the market matures, investment funds in SEA could potentially go down the route of their US counterparts and acquire technology capabilities. 

For now, I wouldn’t hold my breath for this. 

Then there is the bull case, for the most ambitious engineering firms who can partner with investors with high-risk appetite: the AI roll up.

The bull case: the AI Roll up

A traditional PE-led roll up is essentially: use debt to acquire a series of adjacent businesses (say, lots of independent dental clinics), professionalize and standardize them as a way to optimize EBITDA margins, and flip them for a nice profit. 

The AI roll up borrows that idea.

So my bull case for acquiring an outsourcing company is basically this: could a buyer buy a firm that already has good AI engineering capabilities, and use it to build the underlying AI platform(s) required to drive efficiencies in the more traditional business the same buyer would acquire? 

It is quite a complicated thing to pull off and orchestrate - hence being a bull case rather than a base case. In fact, building a native AI engineering team might be easier. A compelling acquisition would need an existing vertical specialization (say, e-commerce, insurance, etc.) that matches the main target companies. 

Bending Spoons is not exactly an AI roll up but it shows the direction of travel: it bought Evernote (among many other digital companies) and cut 60-80% of the company cost (mostly engineering), cranked up features using AI-augmented engineering teams and changed its pricing, turning what used to be a zombie company into a more profitable one.

Having honed its modus operandi, it recently acquired Airtable and is doing the same. There is no reason to believe that they will stop there. 

Other investors prefer to start from traditional businesses such as accounting, law firms, clinics etc, get VC backing for the tech build out and debt for the company acquisitions. Examples abound (General Catalyst, Crestline).

No matter the structure and format, the commonality is as follows:

There is also an obvious risk in this model. Anthropic and OpenAI are going public soon, and will have a large warchest to go after the most obvious industries (legal, accounting, insurance brokerage, to name a few) and run that exact playbook. 

They’ll certainly acquire existing AI roll ups to go fast (like SpaceX who bought Cursor after IPO). Could one build one of their next acquisitions? That’s a long stretch if one intends to build it from Vietnam. At the very least, the customer base and company should be in the US (that’s where the money is and where the big players go first).

The largest enterprise market around this area of the globe is Japan. If one has a strong appetite for that market and specific insights as to how this could work, this may be worth exploring further (not investment advice :)) 

Bringing it altogether

The old investment thesis of software outsourcing seems to progressively come to an end for Vietnam.

Big corporations (FPT, CMC, etc.) will probably stick around. Some smaller firms will make it through. For most, it’ll be tough. Some of my M&A friends are waiting for consolidation, but I’m not convinced.

On the other side of the table, some buyers may find good deals, I suspect mostly for strategic reasons than just “add more engineers quickly and cheaply”. However, Technical Due Diligence must change and include a thorough assessment of AI-readiness throughout the target organization. After all, whether the customer is captive or not, the same question will come up: what’s the playbook to use AI in ways to increase profits? 

As an outsourcing business owner, one ends up essentially estimating the odds of the bear, base and bull case. Everyone has a different insight and opinion on how this can turn out. My personal opinion is that besides the very large corporations, only outsourcing companies with an existing industry specialization will have a chance to survive (whether or not they do survive is a matter of execution). For everyone else, I am pretty bearish.

But no matter one’s opinion, the recurring question of the next 12-18 months in the industry will probably be: who can build real-world AI solutions? Who has a native AI-first SDLC? Who has a vertical industry specialization? And who has a demonstrated pipeline of revenue for AI consulting / implementation? 

Only time will tell.

Further reading

• Survey on Computer Software and Information Technology Enabled Services Exports: 2023-24
• Vietnamese software rises in global value chain
• 2024 Alten Annual Results
• FPT Targets One-Third of Revenue from AI-First Projects
• A New Phase in the History of Evernote
• The Future of Services
• Cursor is now a part of SpaceX
• OpenAI takes an ownership stake in Thrive Holdings to accelerate enterprise AI adoption

Originally published on baolannguyen.com.