The Infrastructure Behind Agentic Memory and Execution

Why memory is getting so much attention

Memory has become one of the central topics in AI over the past year. Most of the conversation has focused on rising demand for memory chips or product-level experiences, such as assistants remembering your preferences and agents learning from past work.

We want to add a more practical view. How does memory play out in real life, starting with a user request and ending with physical capacity inside a server?

This question is becoming more important as software moves from simple chat interactions to agents. A chat session might last a few turns; however, an agent might work for an hour, use 10 tools, read thousands of pages, pause for approval, and return later. The agent needs a record of the goal, the steps already taken, the files reviewed, and the errors found. All of this creates state. Someone has to store the state, choose the useful parts, and move those parts to the model.

At Cota, we have spent a lot of time looking at the broader AI infrastructure optimization category across data centers and edge devices. Agentic memory is part of the same theme.

What AI memory means

AI memory is a loose term. Four different concepts often get mixed together.

• First, model weights. These are the billions of numbers learned during training. Weights hold the model’s general knowledge and behavior. They usually stay fixed while the model serves users.
• Second, context. This is the information placed in front of the model for the current request. Your prompt, a contract, an email thread, tool instructions, and earlier messages all count as context.
• Third, KV cache. KV stands for key and value. As the model reads tokens, the system saves attention data from earlier tokens. The model reuses this data while generating the next token, instead of repeating much of the same work. This needs to stay very close to the model during inference.
• Fourth, long-term memory. This sits outside the model in databases, document stores, logs, or search indexes. The agent pulls selected records into the context when needed.

The user sees one agent, but under the hood, several memory systems work together.

For instance, you ask the agent to fix a bug and open a pull request. The agent reads the issue. Then the agent reviews repository instructions, searches the codebase, opens files, edits code, runs tests, reads the error output, changes the plan if necessary, and tries again.

A normal session might include 3,000 tokens of system instructions and tool descriptions. Another 4,000 tokens might come from the issue, logs, and test output. Relevant code files might add 15,000 tokens. Plans, edits, tool results, and retries might add another 8,000.

The working set is now around 30,000 tokens. This is a standard software task.

Some information sits in an external index, while other information resides inside the active prompt. The model weights stay loaded throughout the session. The KV cache grows as the model processes and generates more tokens.

Qwen3-32B is a useful open-weight model to demonstrate how agent actions turn into physical memory requirements. Qwen released the model in April 2025 with 32.8 billion parameters. Its native context length is 32,768 tokens.

The public model uses BF16, a 16-bit number format, so each parameter takes 2 bytes.

32.8 billion parameters × 2 bytes = about 65.6 GB

An NVIDIA B200 GPU has 180 GB of high-bandwidth memory.

One copy of Qwen3-32B parameters therefore takes about 66 GB of a B200’s 180 GB. This leaves roughly 114 GB for KV cache, runtime memory, temporary working space, and other serving needs.

Next is KV cache.

Qwen’s public configuration lists 64 layers, 8 KV heads, a head size of 128, and BF16 precision. We do not need to understand what each term means for this simple calculation, but they all increase compute and memory requirements linearly: the higher the number, the greater the memory need.

The rough KV-cache calculation for one token is:

2 for keys and values × 64 layers × 8 KV heads × 128 values × 2 bytes = 262,144 bytes. This works out to about 256 KB per token for one active sequence.

At 8,000 tokens, the KV cache takes ~2 GB. At 32,000 tokens, the KV cache takes ~8 GB.

Now let’s go back to the coding agent. A session with system instructions, tool definitions, an issue, code files, test results, etc. might reach 30,000 tokens.

One active session at this size needs around 7.5 GB of KV cache. 10 active agent sessions add around 75 GB. Together with the model weights, the total reaches about 141 GB.

A single B200 has enough physical memory for this rough setup, with around 39 GB left. Real serving capacity will be lower because the system also needs runtime memory, temporary workspace, and operating headroom.

Quantization, lower-precision cache formats, batching, and different model designs could improve performance, but longer jobs still use more memory, and additional simultaneous jobs fill the available memory quickly.

Most companies do not have B200 systems sitting in their own data centers. They rent access through hyperscalers or neoclouds. CoreWeave, for example, offers B200s as an 8-GPU HGX instance. Its current North American pricing lists $8.60 per GPU-hour for inference, or $68.80

per hour for the 8-GPU system. Running one GPU around the clock at that rate is roughly

$6,200 per month. The full 8-GPU node is close to $50k per month, before other services, data movement, and engineering costs.

This is why memory efficiency becomes so important. Cloud providers charge for the full GPU or server, not for memory alone. If the model weights and active KV caches no longer fit, a company has a few options: run fewer agents at the same time, shorten their context, move state to slower memory, compress it, or rent more GPUs. The last option adds more compute even when memory is the main limit. A company may end up paying for extra GPU hours mainly to get enough memory to keep its agents running.

What the memory stack needs next

The memory stack will require innovation across several layers.

The first layer is the physical memory architecture on which several companies are actively working. Google recently announced a custom AI chip code-named Frozen v2 designed specifically to run its Gemini models. By permanently etching or hardcoding elements of Gemini’s architecture directly into the silicon, Google aims to bypass traditional memory-shuttling and data back-and-forth, potentially delivering up to 6-10x more processing efficiency.

The second layer is model serving. KV cache needs to be allocated, shared, moved, and removed as agent sessions start, pause, and resume. Serving systems will also separate hot sessions from idle sessions. An agent actively generating tokens belongs in GPU memory; an agent waiting 2 hours for human approval does not. Its state should move to cheaper CPU memory or storage, then return when work resumes.

The third layer is application memory. This is where the agent records facts, actions, user preferences, tool results, and unfinished work. Today, many systems store these items as text in a database or vector index. That approach works for early applications, but enterprise use needs more structure. Each memory has a source, a timestamp, an owner, and an expiration rule. The system needs to know whether a record is a confirmed fact, a model-generated

summary, or an assumption from an earlier step.

The fourth layer is memory selection. Saving data is easy, but choosing what to retrieve is much harder. An agent does not benefit from every old message at every step. The system must search, rank, filter, and sometimes summarize the prior state before building the next prompt. This selection process will affect agent quality as much as the model itself. Poor selection gives the model stale or irrelevant information. Over-selection fills the context window and increases cost.

The fifth layer is context management. Long-running agents need a way to compact their active history. Some information should remain word for word, such as a legal instruction or a code change. Other information should become a short summary. Repeated tool output should disappear after the important result has been recorded. The hard part tends to be deciding what information is safe to compress.

The sixth layer is observability. Teams need to see where memory goes. How much high-bandwidth memory (HBM) does one task use? How much state gets copied across agents? Which prompts receive cache hits? How often does an agent reload old state? Which memories lead to bad and/or ineffective actions? Without these measurements, companies will add GPUs to solve problems caused by poor memory management.

Agentic memory will grow in importance because agents require continuity. They work across longer periods, more tools, and more data. Good agents need enough history to stay on track. They also need a way to forget. The key innovations will span the full stack, from deciding what to store and what to retrieve, to compressing context, managing KV cache, moving state across HBM, CPU memory, and storage, and measuring cost and task quality. The companies worth watching will make these layers work together and show clear gains in cost per completed task, reliability, and better overall economics.