The AI
Conversation

A token is simultaneously the billing unit, the performance unit and the risk unit. Scaling agentic applications requires control over the relationship between token consumption, model performance and cost, because an inefficient agentic loop or a spike in high-latency model calls produces unforecasted spend.

What to measure: input, output and total spend broken down by model rather than one aggregate; the same metrics filtered by agent and by version, which is what makes architectures comparable; and cost per resolved task rather than cost per call — cost per call rewards agents that give up early.

The regression test that matters: a new agent version consuming significantly more tokens without a measurable increase in task accuracy is a regression you should catch immediately rather than at invoice time. Tokens up and accuracy flat is the clearest bad signal in agentic operations.

Retries versus reasoning loops: a high token count is not automatically failure — it can be successful self-correction. The failure case is an agent stuck repeatedly calling the same failing tool. You separate the two by correlating evaluations with traces: watch a tool time out, then watch whether the agent recognised it and pivoted, or just called it again.

The security crossover: a runaway spend event and a compromised agent look identical from a distance. It cuts the other way too — MCP servers serving agent-like functions are susceptible to prompt storms, malformed inputs and recursive task requests that cause denial of service. Token exhaustion is an availability attack with an invoice attached.

An agent is not a chatbot. It is a model plus instructions plus tools plus memory plus an autonomy setting, and each is a separate control surface with a separate owner. Most seller credibility is lost because the two sides of the table are discussing different layers.

The three autonomy tiers, named: manual, where engineers investigate and remediate step by step; guided, where AI identifies issues and proposes hypotheses while a human validates and remediates; and autonomous, where AI attempts remediation and engineers supervise and intervene. Ask every customer which tier each agent is on. Most can answer per programme, not per agent — and that gap is the finding.

The structural flaw underneath all of it: instructions and data share one channel. The agent cannot reliably tell the difference between content it was told to read and content telling it what to do, because nothing marks which bytes are trustworthy. Every mitigation is a workaround for that single design property, which is why enforcement has to live outside the model.

The Model Context Protocol became the de facto standard for connecting agents to external tools, and with it the highest-value target in the stack. The core problem is that an agent treats everything in its context as trustworthy — tool descriptions, parameter schemas, and the data tools return — so anyone who can plant instructions in any of those can steer the agent.

Tool poisoning is the sharp edge: malicious instructions hidden in a tool’s description or schema, which the model reads and the user usually does not. It defeats human review by hiding in the metadata layer nobody inspects.

Aggregation is the risk multiplier. The number of MCP servers connected to a single agent is a policy decision most organisations have never made. Probe it specifically — the risk is not one server, it is five.

Controls that actually apply: treat tool descriptions and tool outputs as untrusted input and inspect the full schema before approving a server. Then add the boring supply-chain hygiene nobody has applied yet — an inventory, pinned versions, publisher reputation, least-privilege scopes per server, sandboxed execution, and a policy gate on how many servers a single agent may aggregate.

A skill is a portable bundle of instructions and executable scripts that teaches an agent a procedure. Functionally it is a package. Operationally it has none of the controls your organisation spent twenty years building for packages — no signing requirement, no registry policy, no version pinning, no review gate.

How to frame it to a customer: “You have a package policy for npm and PyPI. Do you have one for agent skills? Because the install path is a file copy, the review step is optional, and the thing you just installed can run code with your developer’s credentials.” That sentence sells the inventory better than any threat statistic.

“Skills go through the same review” is almost never true in practice, because a file copy never triggers the review that a package-manager transaction triggers. Test it by asking who approved the last one.

This is exactly the artifact layer that endpoint AI oversight is built to inventory and rank — packages, plugins, extensions, MCP servers, and agent-like activity that mimics legitimate user behaviour, which is why device-posture tooling never sees it.

Direct model calls and autonomous agents both introduce failure modes traditional monitoring misses entirely: hallucinations, context window overflows, infinite reasoning loops, and bad tool selection. The line worth memorising is that a fast response is still a failure if it is ungrounded, and a multi-step agent loop can succeed technically while failing logically. An agent can complete its workflow without a single server error and still return an inaccurate answer, so error rate is no longer the indicator of success.

Four things to instrument: token cost, to block runaway spend and catch efficiency regressions between agent versions; semantic evaluations sent in as metrics so accuracy becomes a monitored trend across the fleet rather than a pre-launch test; execution traces, so a dropping score takes you straight into the orchestration sequence and the arguments passed to each tool; and tool-call reliability across external dependencies.

The reframe to carry into every room: success for AI systems is no longer defined by uptime or latency, but by the efficiency of the reasoning process, the accuracy of autonomous decisions, and the quality of the output. Practically, the customer’s AI governance programme needs three numbers it almost certainly does not have today — cost per resolved task, a live accuracy trend, and tool-call failure rate per agent.

Build it on OpenTelemetry GenAI standards so the telemetry is portable rather than vendor-shaped. Instrument once; do not let the instrumentation become the lock-in.

The OWASP Top 10 for Agentic Applications, known as the ASI Top 10, focuses on what happens when AI systems act, unlike the LLM Top 10 which focuses on model behaviour. Use the IDs by name — it signals you have read the source rather than a vendor summary.

Two companions worth knowing: OWASP also maintains an updated GenAI LLM Top 10, and publishes periodic GenAI exploit round-up reports cataloguing real incidents — useful for keeping a customer briefing current without relying on vendor marketing.

NIST AI RMF 1.0 organises AI risk management into four functions — Govern, Map, Measure and Manage — with Govern as a cross-cutting function infused throughout the other three. It is explicitly voluntary and non-certifiable, so it is a structuring language, not a badge. Use it to organise your service: the assessment is Map, the telemetry is Measure, the retainer is Manage.

ISO/IEC 42001:2023 is the first international standard for an Artificial Intelligence Management System, specifying requirements for establishing, implementing, maintaining and continually improving an AIMS, for organisations both providing and using AI. Unlike the NIST framework it is certifiable through accredited bodies, which matters commercially — certification requests will arrive through procurement and customer questionnaires.

CycloneDX AI/ML-BOM is a machine-readable inventory of the components, configurations and processes of an AI/ML system. This is why the assessment deliverable is an AI-SBOM in a published format rather than a proprietary report — it is portable evidence the customer can reuse across audits and questionnaires.

How to use all three: never lead with the framework. Lead with the artifact the customer needs, then name the framework it maps to so the buyer can defend the spend internally.

This is where sellers lose credibility fastest, because timelines have moved and a lot of circulating collateral is stale. Two rules: never sell a deadline you cannot substantiate, and always check the date on the deck you are about to present.

What to sell instead of a deadline: the direction of travel. Logging, traceability, human oversight and third-party accountability are converging requirements across every framework and sector, and each one is an artifact your service produces anyway. That argument survives a timeline slipping; a specific date does not.

How to handle the “are our agents in scope?” question: “Not in scope” is rarely a considered position — it is usually the absence of one. If an agent authenticates, holds credentials, and touches regulated data, assuming it is out of scope is a decision worth making deliberately rather than by default.

See the current evidence section for the dates that are verified as in force versus deferred as of today.

Six moves, in this order. Anyone who starts at move five without doing move one is buying enforcement for a population they cannot describe. The full ladder with deliverables is in the section below, and the deep-dive discovery questions for the endpoint layer are on the AES Conversation Starters page.

The sequencing argument is the single most useful thing you can give a customer in a first meeting, because it converts an overwhelming topic into an ordered list with an owner per step. It also happens to describe the service ladder exactly: the assessment is moves one to three, the retainer is moves two to four run continuously, and the premium tier is moves five and six.