- Memory drift is what happens when facts an AI agent has stored no longer match the source they came from. It is separate from model drift, data drift, and concept drift, and fixing the model does not fix it.
- The practical answer is to treat agent memory as a governed table (a claim ledger), where every record carries its source, source version or hash, owner, writer, extractor version, expiry, and an explicit trust state (verified, stale, quarantined, or retired).
- On AWS, Amazon Bedrock AgentCore Memory stores memories and provides retention controls (TTL). However, source-version validation, ownership metadata, trust-state transitions, and re-verification workflows remain part of the proposed governance design.
- On Databricks, a Delta table in Unity Catalog provides access grants and column-level lineage. Managed agent memory is currently in Beta and enforces access at the store level only.
- A midsized team with a small platform group can run this with a scheduled, deterministic verifier and four trust states, without requiring a dedicated SRE function for AI.
“An agent that remembers the wrong thing is more dangerous than one that remembers nothing, because it answers with confidence.”
The pattern is familiar to anyone who has put an agent in front of real users. The agent says a service belongs to one team, or that a discount above a certain amount needs approval, or that a customer prefers email. All of it was true when the agent stored it. Someone changed the source three weeks ago, and nothing told the agent.
The usual name for this is memory drift, and the usual fixes are reasonable: expiry times, tighter summarization, better retrieval, periodic refreshes. These approaches treat drift primarily as a defect inside the agent. That framing is incomplete once an agent’s memory begins influencing business decisions, because the stored information must also be managed as governed data. Once an agent’s memory influences a refund, a deployment, or a customer reply, that memory is business data. It has a source, an owner, a change history, a retention rule, and a set of people who should or should not see it. Data teams already have a discipline for managing exactly those things.
This article makes one argument: agent memory should be designed as a governed table, with the agent as one of its clients. It describes what that table holds, where AWS and Databricks help according to their current documentation, where they leave gaps you have to fill yourself, and what a midsized company with a small platform team can realistically run.
1. Four kinds of drift, and what the usual advice leaves out
Memory drift is what happens when the facts an AI agent has stored no longer match the source they came from. It is one of four drifts that get confused with each other, and only one of them is fixed by looking at the model.
| Drift Type | What changes | Where it shows up | Typical response |
|---|---|---|---|
| Model drift | The model’s behavior shifts, often because a provider updates a version | Same prompt, different answers; evaluation scores move | Pin versions, rerun evaluations |
| Data drift | The inputs the system sees change in distribution | Retrieval or classification quality falls | Monitor input statistics |
| Concept drift | The right answer for a given input changes | Accuracy decays although inputs look familiar | Retrain, relabel, redefine |
| Memory drift | What the agent stored no longer matches its source | Confident answers that were once true | Verify, expire, correct via ledger |
Model shift reaches memory through a side door. In AgentCore Memory, prompts guide a language model through extraction and consolidation. AWS describes built-in strategies that can be modified with custom prompts and models in your account, and self-managed strategies that leave extraction, models, prompts, and record schemas to you. When the model or prompt changes, the same conversation can produce different stored records. The extractor version should therefore be recorded for every claim so that changes in extraction behavior can be traced to the model and prompt versions used.
Two pieces of existing writing frame the problem well. Kore.ai describes memory drift as gradual precision loss, names four forms (temporal, semantic, behavioral, and coordination drift), and argues for a durable external source of truth. Tian Pan’s April 2026 piece goes further on mechanics: treat memory as a replica that has fallen behind its primary, store the source version with each memory, revalidate lazily at read time, and set expiry by how fast the underlying entity changes.
On the governance side, Atlan’s April 2026 post recommends provenance, lineage, retrieval-time permissions, validity metadata, and decision traces, and Microsoft’s Zero Trust guidance for agents asks for memory operations to be logged with identity, source, and model version. In the versions reviewed, neither specifies how to verify staleness deterministically or how to build the controls on AWS or Databricks.
Model drift is a change in how the model behaves: a provider updates a version and the same prompt returns different answers. Memory drift is a change in what the agent holds: the stored fact no longer matches its source, even though the model has not changed at all. You can pin every model version and still ship an agent that confidently repeats last quarter’s policy. The two also interact: in AgentCore, prompt or extractor shifts alter what is written to memory, which is why recording the extractor version is critical.
2. The claim ledger: what a governed memory table holds
A memory worth trusting stores each claim together with the evidence behind it, in a table that answers to three clocks instead of one:
The thing the claim came from has changed. A revision number, an ETag, or a content hash detects this deterministically without calling a model.
Some claims decay whatever the source does. An on-call rotation changes weekly; a company’s legal name changes rarely. Expiry must follow how fast the entity changes.
The rules around the claim changed. A retention period ended, a person lost access to the underlying system, a customer asked for deletion, or compliance rules updated.
There is early evidence for the age clock: ChurnBench, an arXiv preprint by independent researchers using synthetic enterprise data, found that freshness errors depended on how refresh intervals compared with each entity’s rate of change, not on how old the cache was.
CREATE TABLE agent_memory.claims (
claim_id STRING,
subject STRING, -- what the claim is about
statement STRING, -- the claim itself
source_uri STRING, -- where it came from
source_version STRING, -- commit, ETag or revision when read
source_hash STRING, -- hash of the content it was drawn from
observed_at TIMESTAMP,
stale_detected_at TIMESTAMP,
last_verified_at TIMESTAMP,
trust_state STRING, -- verified | stale | quarantined | retired
owner STRING, -- accountable team, never the agent
written_by STRING, -- person or agent identity
extractor_version STRING, -- model and prompt that produced the claim
expires_at TIMESTAMP, -- age clock
retention_class STRING -- policy clock
);The staleness check is then an ordinary query against a snapshot of current source state that your ingestion jobs keep up to date. No model is involved, which makes it cheap, repeatable, and explainable to an auditor.
MERGE INTO agent_memory.claims AS c
USING agent_memory.source_snapshot AS s
ON c.source_uri = s.source_uri
WHEN MATCHED AND c.trust_state = 'verified'
AND (c.source_hash <> s.content_hash OR c.expires_at < current_timestamp())
THEN UPDATE SET
c.trust_state = 'stale',
c.stale_detected_at = current_timestamp()
WHEN NOT MATCHED BY SOURCE AND c.trust_state = 'verified'
THEN UPDATE SET
c.trust_state = 'stale',
c.stale_detected_at = current_timestamp();The source snapshot must contain no more than one row per source_uri; otherwise, multiple source rows may match the same target row and cause the MERGE to fail. The WHEN NOT MATCHED BY SOURCE clause requires Databricks Runtime 12.2 LTS or above. Validate snapshot completeness before running the MERGE, and do not interpret a failed ingestion run as evidence that a source has disappeared.
3. Four trust states do most of the work
Four trust states handle the complete lifecycle. Verified and stale describe freshness; quarantined describes trust; and retired describes lifecycle. A claim may be stale and suspicious at the same time, so they represent distinct operational concerns.
The top row moves with the source. The bottom row moves with suspicion and policy; the two should never be merged into one flag.
Source version or hash matches. Passed deterministic verification. Served to agents with high confidence.
Source content hash changed or validity period expired. Withheld, or served with clear staleness label while re-verification is queued.
Suspected wrong, poisoned, or malicious. Completely withheld from retrieval gateway while security/ops investigates.
Past permitted retention period or deleted upon user GDPR request. Reduced to an append-only audit tombstone.
One design choice matters more than it looks: flagged claims persist. Deleting a stale claim silently removes the evidence that the agent once believed it. Keep every state change as an append-only event row and treat the claims table as the current view of that log. Delta table history is not a substitute: Databricks documents a default of 30 days of log retention and 7 days of deleted-file retention, and advises against using table history as a long-term archive.
4. Memory ledger architecture: 5 stages, 1 governance layer
Agents should read memory only through a ledger that continuously validates its own sources. Here is the reference architecture:
Sources of Truth
Code repositories, wikis, API schemas, databases, SaaS records with version tags.
Deterministic Verifier
No model calls. Compares source version/hash, age clock, and policy retention rules.
Claim Ledger
Governed table: claim statement, source receipt, version, state, owner, expiry.
Retrieval Gateway
Serves verified, labels stale with warnings, strictly blocks quarantined claims.
AI Agent Client
Reads through gateway only. Writes new claims back with source receipts via verifier.
5. Claim ledger on AWS
On AWS, storage, encryption, and basic retention are built in, while the verifier and trust states are built as cloud-native serverless components.
S3 documents, API endpoints, Aurora databases with ETag or content hash.
Scheduled EventBridge rule triggering an AWS Lambda function or Step Functions state machine.
Amazon Bedrock AgentCore Memory with a self-managed strategy (or Amazon Aurora / DynamoDB table).
Lambda function acting as the memory tool; filters by trust_state.
Reads only through gateway; writes new claims back through verifier with source metadata.
Governance Controls on AWS
- •IAM Policies: Granular restrictions on who and what can read or write memory stores.
- •KMS Encryption: Service-managed key is always used; customer-managed key (CMK) is optional (event metadata not CMK-encrypted).
- •Bedrock Guardrails: Input/output filtering to and from memory plus adversarial testing to defend against memory poisoning.
- •CloudTrail: Audit logging for management events; verify specific AgentCore Memory data operations captured before promising auditors.
6. Claim ledger on Databricks
On Databricks, the table, access grants, and column-level lineage come built into Unity Catalog. A dedicated event table handles long-term state auditability.
Delta tables, Lakehouse volumes, external operational databases via Lakehouse Federation.
Scheduled Databricks Workflow Job running the deterministic MERGE statement.
Delta table in Unity Catalog (or Lakebase managed Postgres when ultra-low-latency reads/writes matter).
Append-only record of every state transition. Preserves proof of what the agent believed.
Model Serving endpoint or LangChain tool enforcing trust-state filters.
Queries gateway; writes back claims with source snapshot receipts.
Governance Controls on Databricks
- •Unity Catalog Grants: Row/column privileges on the ledger table. (Managed agent memory Beta enforces access at store level only).
- •Lineage Tracking: Automated table and column-level lineage with a rolling 1-year window in system tables. Always reference tables by name.
- •Delta History Limits: Default 30-day log retention means table history is not an audit archive—use the separate append-only event table.
- •Lakebase in Unity Catalog: Appears as a read-only catalog requiring Serverless SQL; direct Postgres connections use Postgres roles.
7. Mapping the ledger to AWS and Databricks
Both platforms can host the ledger. Here is how their responsibilities align:
| Responsibility | AWS | Databricks |
|---|---|---|
| Claim storage | Amazon Bedrock AgentCore Memory with self-managed strategy, or an Aurora/DynamoDB table. | A Delta table in Unity Catalog; Lakebase (managed Postgres) for low latency. Managed agent memory is Beta. |
| Evidence & versions | Add source version, hash, and extractor version to your schema. | Columns in the Delta table; table history gives short-term versions within retention limits. |
| Access control | IAM policies; KMS encryption (service-managed always on; customer-managed optional). | Unity Catalog grants on the ledger table; managed agent memory enforces access at store level only. |
| Lineage & audit | CloudTrail logs management APIs. Verify specific AgentCore Memory data operations in your account. | Unity Catalog lineage at table/column level; rolling 1-year window in system tables. |
| Verifier | Scheduled EventBridge Scheduler triggering Lambda or Step Functions. | Scheduled Databricks Workflow Job running the deterministic MERGE statement. |
| Poisoning controls | Bedrock Guardrails on prompts to and from memory, plus adversarial testing (AWS guidance). | Table-level validation, schemas, and quarantining; no built-in agent memory poisoning checks yet. |
8. Five details worth checking before you commit to either path
01.Databricks managed agent memory is Beta
The documentation describes the store, rather than the individual actor, as the security boundary and states that per-entry and per-actor access control are not available. Strict isolation requires separate stores.
02.Self-managed Lakebase pattern is labeled legacy
The self-managed memory page says you own the schema, connections, and access controls, and recommends managed memory for new agents. A custom ledger is therefore your own pattern.
03.Unity Catalog lineage has sharp edges
Path-based references receive no column lineage, and renamed tables lose their lineage history. Reference the ledger strictly by table name and avoid renaming it.
04.Lakebase in Unity Catalog is read-only
A registered Lakebase database appears as a read-only catalog that requires a Serverless SQL warehouse, while direct Postgres connections still rely on Postgres roles. That is two access paths to govern.
05.AgentCore extraction/consolidation does not replace a ledger
Although AgentCore Memory provides retention controls, including TTL for long-term memory records, the proposed claim ledger adds source-version validation, explicit trust-state management, and a separate re-verification workflow.
How do you secure agent memory on AWS and Databricks?
Treat agent memory as business data and secure it accordingly. On AWS, use IAM to restrict which actors and services can read or write memory, and use KMS encryption while accounting for documented limitations on customer-managed keys regarding event metadata. On Databricks, govern the ledger through Unity Catalog grants and reference tables by name to preserve lineage. Across both platforms, record claim provenance, quarantine suspicious claims, and maintain an auditable state-change log.
9. What a midsized team can actually run
A company of 500 to 3,000 people can run a claim ledger with the team it already has, provided it limits the scope and gives every claim a named owner.
Start with one agent and its expensive claims
Pick the agent whose wrong answer costs the most. If a fact can be looked up live at answer time, do not store it in memory. Store only persistent decisions, user preferences, and derived conclusions.
Match the check to what the source offers
Use the source clock (hashes, ETags, commits) for systems that expose versions. For legacy systems that do not, use the age clock with a short expiry and accept the weaker guarantee openly.
Put claim IDs in the decision record
When the agent acts, log the claim IDs and source versions it relied on next to the action. Then the audit question (“what did the agent believe when it did this?”) becomes a simple SQL join.
Give three groups three jobs
| Owner | Accountable for | Suggested cadence |
|---|---|---|
| Data platform team | The ledger table, the verifier job, lineage, and the state-change log. | Job health weekly |
| Agent or app team | Which claim types exist, their expiry rules, and fallback behavior for stale claims. | When agent scope changes |
| Security & compliance | Access by actor, retention classes, GDPR deletion requests, quarantine decisions. | Quarterly review; deletions on arrival |
10. A maturity checklist for agent memory
Use this to evaluate where your own organization sits today:
| Level | What you can say | What it takes |
|---|---|---|
| 0. Unlabeled | “The agent remembers things.” | Text or embeddings in a store, no source or timestamp. |
| 1. Labeled | “We know where each memory came from and when.” | Source URI and timestamp attached to every record. |
| 2. Verified | “We know when a memory stopped matching its source.” | Version or hash per claim, deterministic check, stale flag, named owner. |
| 3. Governed | “We can show what the agent believed when it acted, and who could see it.” | Quarantine state, access by actor, retention/deletion rules, append-only log, claim IDs in decisions. |
Start here: if you can do only one thing this quarter, add a source version and a trust state to whatever you already store, and make your retrieval gateway respect them.
11. Objections a skeptical architect will raise
Q:Why not just retrieve from live sources and skip memory?
Often you should. If a fact can be fetched cheaply at answer time, fetching it removes the drift problem entirely. Memory earns its place for things no source system holds: decisions made in earlier sessions, user preferences, and conclusions the agent derived. Those are the claims that need a ledger.
Q:Won’t managed memory services handle this?
Managed services handle extraction and storage. AWS explicitly states customers are responsible for preventing memory poisoning. Databricks managed memory is in Beta and enforces access at the store level rather than entry-level. Neither service provides the complete combination of source-version validation, ownership, trust-state management, and auditable verification described here.
Q:A hash comparison only catches sources that changed. What about claims wrong from the start?
Correct—this is the real limit of the approach. Deterministic checks detect drift, not original error or deliberate poisoning. The ledger records who wrote each claim and the extractor version used. Quarantining provides a holding pen for suspicious claims, while periodic sampling of verified claims catches original extraction errors.
Q:Isn’t this too much for a company our size?
The smallest version is one table, one scheduled query, and one rule in the retrieval gateway that refuses to serve claims marked stale without a label. It can be built in the platform you already run in days.
12. Where this leaves agent memory
Memory is the part of an agent that outlives the conversation, and that is what makes it a data asset. Anything that persists needs what persistent data has always needed: a source, an owner, a change history, an access rule, and an end of life. The drift fixes in circulation cover the first two. Governance coordinates security, compliance, ownership, and operations across teams.
Start small. Pick one agent, list its claim types, add a source version and a trust state, and make the retrieval layer respect them. Everything else here can follow from that first table.
As a services company specializing in technology strategy and enterprise engineering, Abilytics helps organizations implement resilient, governed AI architectures on AWS and Databricks. If you are designing memory drift controls, claim ledgers, or Unity Catalog and Bedrock security for production agentic workloads, we are always happy to compare notes.
