What makes AI Agents Work in Production
Picking a model feels like the big decision. It is not.
A capable model can write, reason, call tools, summarise documents, and inspect data. But a production agent needs much more than a good brain. It needs a way to act, remember, fetch facts, use tools safely, pass tasks to other systems, and prove that it did the job well.
That is where most agent builds fail. The demo works because the path is clean. The production version breaks because the world is messy: missing permissions, stale context, bad retrieval, unclear state, unsafe tool calls, and no way to measure quality before users complain.
The model matters. The nine layers around it decide whether the agent survives real use.
A production agent needs more than a prompt
A chatbot responds. An agent acts.
That difference changes the architecture. Once a system can open files, query databases, write tickets, send emails, update records, or trigger workflows, every layer around the model becomes part of the product.
A useful way to think about AI agents is to separate the model from the operating system around it. The model predicts and reasons. The surrounding layers decide what the model can see, what it can do, what it should remember, what it must not touch, and how success gets checked.
Here is the map.
Layer | What it answers | What breaks when it is missing |
Harness | How does the model act repeatedly? | The agent stalls after one response |
Memory and state | What should it remember and where is it in the task? | It repeats work or loses context |
RAG | Which facts should it use? | It guesses or cites the wrong source |
MCP | How does it connect to tools and data? | Each integration becomes custom glue |
Skills | How does it reuse task know-how? | Every request starts from scratch |
Guardrails | What is allowed and what needs approval? | Risky actions happen too easily |
Evals | How do we know it works? | Users become the test suite |
A2A | How do agents hand off work? | Systems stay isolated |
Multi-agent | How do specialists work together? | One agent becomes overloaded |
None of these layers has to be huge on day one. But each one needs a deliberate answer.
The harness gives the model an action loop
The harness is the loop that turns a model call into an agent.
A simple version looks like this:
Put the task and context into the model.
Let the model choose the next step.
If it needs a tool, call that tool.
Return the result to the model.
Repeat until the task is done or blocked.
That loop sounds basic, but it carries a lot of production weight. It handles tool selection, retries, timeouts, partial results, errors, logging, and stopping conditions.
Without a harness, the agent can only answer. With a harness, it can inspect a spreadsheet, search policy documents, draft a reply, ask for missing information, and then update a system when approved.
A good harness also prevents runaway behaviour. For example, an agent should not keep calling the same failing API for ten minutes. It should record the failure, try a safe fallback, or ask for help.
Practical questions to ask:
How many steps can the agent take before it must stop?
What happens when a tool fails?
Can it explain the path it took?
Does it have a clear success condition?
Can a human inspect the trace later?
The harness is where the agent becomes observable. If you cannot see the loop, you cannot debug the agent.
Memory and state keep the agent grounded
Memory and state sound similar, but they solve different problems.
State is where the agent is right now. It may include the current step, the active user request, pending tool calls, selected files, results already received, and the next planned action.
Memory is what the agent can carry across runs. It may include user preferences, project details, past decisions, known constraints, or facts that should not be rediscovered every time.
A travel assistant, for example, may store that a user prefers morning flights and vegetarian meals. That is memory. During one booking flow, it may track that the destination is Kochi, the budget is ₹20,000, and the hotel search is still pending. That is state.
Mixing the two creates problems. If every temporary detail becomes permanent memory, the agent becomes noisy. If nothing persists, it wastes time and frustrates users.
Production systems usually need memory rules:
What can be remembered?
What must expire?
What needs user consent?
What should be stored as structured data?
What should stay only in the current session?
The safer pattern is to keep memory narrow at first. Store what clearly improves future work. Leave the rest in session state.
RAG brings the right knowledge into the prompt
Retrieval-augmented generation, or RAG, helps the agent answer from the right material instead of from memory alone.
The pattern is simple:
Take the user’s question or task.
Search trusted documents, databases, tickets, policies, or code.
Pull the most relevant chunks into the prompt.
Ask the model to answer using that context.
Return sources where possible.
RAG quality depends on details that many teams skip:
Document chunking
Metadata
Access control
Freshness
Ranking
Source display
Handling contradictory documents
Bad retrieval can be worse than no retrieval. The model may sound confident while using the wrong policy or an outdated file.
For example, if an agent answers GST invoice questions for an Indian business, it should pull the relevant internal policy and current tax guidance from approved sources. It should not improvise from a vague memory of tax rules.
RAG works best when retrieval is treated as a product feature, not a hidden backend step.
MCP gives tools and data one standard plug
Tool use is where agents become useful, and where integration pain starts.
Every tool has its own API, permissions, schemas, errors, and quirks. One tool expects JSON. Another needs OAuth. A third returns nested data that needs cleaning before the model can use it.
The Model Context Protocol, or MCP, gives agents a common way to connect with tools and data sources. Instead of writing one-off glue for every model and every service, teams can expose capabilities through a shared protocol.
That helps in three ways.
Less custom wiring
A tool can be made available in a consistent format. The agent can discover what it can do, what inputs are needed, and what output to expect.
Cleaner separation
Tool owners can manage access, limits, and behaviour without hiding everything inside the agent code.
Less vendor lock-in
When tools speak a shared protocol, changing models or agent frameworks becomes less painful.
MCP does not remove the need for engineering judgement. You still need authentication, validation, logging, and careful tool design. But it gives teams a common interface instead of a pile of private adapters.
For production builds, that matters. The number of tools tends to grow quickly once the first agent works.
Skills turn repeated know-how into reusable modules
A skill is reusable task knowledge.
Instead of loading every instruction into every prompt, a skill lets the agent discover what it can do and load the detailed guidance only when needed. In many implementations, that knowledge can live in a `SKILL.md` file with a name, description, instructions, examples, and supporting files.
The key idea is simple: only the skill name and description need to be available upfront. The full skill loads when the task calls for it.
That keeps prompts smaller and cleaner.
Skills are useful for repeatable work such as:
Preparing a monthly finance summary
Reviewing a pull request against team rules
Converting support tickets into release notes
Checking a contract against standard clauses
Formatting a board update
Running a data quality checklist
A good skill is not just a prompt. It captures the steps, standards, edge cases, and output format for a known task.
For example, a “customer refund review” skill might tell the agent which policy documents to retrieve, what order to check them in, when to ask for approval, and what fields to include in the final response.
Skills also make agent behaviour easier to maintain. When the refund process changes, update the skill. Do not hunt through ten prompts spread across the codebase.
Guardrails decide what the agent is allowed to do
The moment an agent can act, guardrails become non-negotiable.
Guardrails are the boundaries around the agent’s behaviour. They include permissions, sandboxes, validation checks, rate limits, policy filters, and human approval steps.
A safe agent should know the difference between low-risk and high-risk actions.
Low-risk actions may run automatically:
Search a help article
Summarise a document
Draft a response
Classify a ticket
Read a non-sensitive record
High-risk actions need tighter control:
Send an email to a customer
Delete data
Change a payment setting
Approve a refund
Update production infrastructure
Access sensitive personal data
The best guardrails are not vague warnings in the prompt. They are enforced in the system.
For example, an agent can draft a vendor email, but the send action can require human sign-off. It can propose a database change, but execution can happen only in a sandbox unless an authorised person approves it.
Guardrails should also cover data access. If a user is not allowed to see a document, the agent should not retrieve it, summarise it, or expose it indirectly.
Prompts help shape behaviour. System controls make the boundary real.
Evals tell you whether the agent is actually good
A demo can hide failure. Evals expose it.
Evals score outputs against what you expected before users do that job for you and get disappointed. They can check accuracy, completeness, format, tool use, safety, latency, and source quality.
A useful eval set for a support agent might include:
A clear policy question
A question with two similar policies
A request that should be refused
A case that needs escalation
A stale document that should not be used
A question with missing information
A tool failure during the task
For each task, define:
What the agent should do
Which sources it should use
Which actions are forbidden
What a good answer contains
What counts as a serious failure
Evals also help when changing models. If a cheaper or faster model passes the same evals, you have evidence to switch. If a smarter model gets worse at following your tool rules, the evals catch it early.
Without evals, agent quality becomes opinion. With evals, quality becomes something you can improve.
A2A lets agents hand off work across boundaries
As agents spread, they need to work with each other.
Agent2Agent, or A2A, is about discovery and handoff between agents, including agents from different vendors. One agent may know that another agent exists, understand what it can do, send it a task, and receive a result.
This matters because not every agent should do everything.
A procurement agent should not also be a tax expert, a legal reviewer, a data analyst, and an infrastructure operator. It should know when to hand work to a specialist.
A2A gives teams a path beyond isolated agent islands. An agent can request help from another system without every connection becoming a custom project.
A handoff needs structure:
What task is being delegated?
What context is included?
What permissions apply?
What output is expected?
Who owns the final answer?
What happens if the specialist agent refuses or fails?
Open standards such as MCP, A2A, and agent skills reduce lock-in. They let teams build parts that can move across models, tools, and vendors more easily than closed, one-off integrations.
That does not make architecture automatic. It does make the architecture less brittle.
Multi-agent systems split complex work among specialists
A multi-agent system uses an orchestrator to split a job across specialist agents.
The orchestrator decides what needs to happen. Specialist agents handle focused parts of the work. Some tasks may run in parallel, then the orchestrator combines the results.
Each agent has a narrower job, clearer tools, and more focused evals.
This approach can help with complex work, but it adds coordination costs. More agents means more traces, more failures, more permissions, and more places where context can get lost.
Multi-agent is not the first layer to build. It works best after the earlier layers are stable.
A good rule is simple: split the work only when the split makes the system easier to control.
If one agent can do the task reliably, keep it simple. If the task has distinct expert steps, separate tools, or parallel checks, multi-agent may be worth it.
Build the layers in the right order
You do not need a perfect version of all above layers before shipping. You do need to know which layer you are relying on and which one you are ignoring.
A practical build order often looks like this:
Start with the harness so the agent can act and stop safely.
Add state before adding long-term memory.
Add RAG for any answer that depends on private or changing knowledge.
Connect tools through a clear interface, using MCP where it fits.
Package repeatable work as skills.
Put guardrails around every risky action.
Create evals before expanding the use case.
Add A2A when agents need to cross system boundaries.
Use multi-agent patterns when specialist work justifies the added complexity.
The common mistake is to jump from model selection to multi-agent architecture. That creates a system that looks advanced but is hard to trust.
A production agent earns trust through the boring layers: state, permissions, retrieval quality, traces, tests, and clear handoffs.
The next time an agent build gets stuck, do not start by changing the model. Look around it.
The missing layer is usually where the real problem lives.
Comments