Version drift: when a well-sourced AI answer uses the wrong policy

Quick brief

A correct quotation can still support the wrong decision when its policy version, scope or effective period does not apply to the case. Maintain the authoritative source lifecycle, constrain retrieval to applicable permitted versions, and recheck material conditions before an approved action is committed.

Full analysis

Revised October 18, 2026.

An assistant cites the policy correctly. The quoted paragraph exists, the citation opens, and the answer accurately reflects what the document says. The problem is that a different version now governs the decision.

I use version drift to describe this mismatch between the version a system relies on and the version that applies to the work. It is especially easy to miss when the answer looks well supported. Checking that a source exists does not establish that it is the right source for this case at this time.

The practical task is to keep source validity connected to the decision and, when the system can act, to the moment of execution. That requires content ownership, applicability rules, retrieval controls, and a response when the underlying facts change.

Download the printable framework map.

Work through a policy change while an approval is pending

Consider a synthetic purchasing example. Policy version 7 permits a named manager to approve a supplier exception up to $25,000. Version 8 requires an additional review for the same exception above $10,000 and takes effect at noon. An assistant prepares an $18,000 recommendation at 11:30, and the manager approves it at 11:45. The update is attempted at 12:05.

The earlier recommendation may have been correct when prepared. The approval may have been valid when given. Neither fact settles whether the action is permitted at 12:05. The workflow needs the organization’s rule for changes that cross an effective-date boundary: apply the new requirement, honor a defined transitional provision, or seek a fresh decision.

Those amounts and times are illustrative. Their purpose is to make the failure observable. A test should be able to show which policy applied, which approval was obtained, and why the execution continued or stopped. “The model had the right document” is an incomplete answer if the relevant document changed between preparation and action.

The same issue can arise without a new policy. A customer’s location, the requester’s role, the supplier’s status, or the transaction value can change. Version control for the document is necessary in many workflows, but the decision depends on the applicable state of the case as well.

Distinguish relevance, validity, and authority

Semantic retrieval helps find material that matches a question. A passage can be highly relevant and still come from a draft, a retired procedure, or a policy for another jurisdiction. Similar wording is not evidence that the passage governs the current task.

Validity requires a separate determination: has the material been approved, when does it apply, what does it supersede, and what scope does it cover? Authority adds another question: may this person or service use it for the proposed action? A current policy may authorize a different role or business unit.

This is also different from ordinary factual staleness. An old address might simply be wrong now. A previous policy can remain correct for a historical decision even though it no longer governs new work. An investigation into an action taken in May may need the version that applied in May, rather than today’s replacement.

The retrieval request therefore needs a purpose and a time basis. “What applies to this new request now?” and “What governed the decision made on May 4?” should not silently select the same source set. Preserve the distinction in the record so a later reviewer can understand it.

Give authoritative content an owner and a lifecycle

For each governed source, retain a stable identifier, version, owner, approval status, scope, effective interval, and known supersession relationships. Where corrections can be entered after their effective date, distinguish when a rule applies from when the system learned about it. That helps explain why two reviews made at different times may have had different information.

These fields are only as reliable as the publishing process. Decide who can approve a source, how a replacement becomes effective, how exceptions are recorded, and how downstream systems learn that something changed. An ingestion job that copies yesterday’s current flag into a vector store can preserve a mistake very efficiently.

Check consistency across the content, metadata, and search representation. If a new policy is published but the search index still contains only its predecessor, the application should know that its view is incomplete. Choose a defined response for that condition, especially before consequential action. Do not let the absence of a new search result become evidence that no new policy exists.

Retain earlier versions for legitimate history and investigation under the applicable retention rules. Removing a superseded source from current retrieval does not require erasing the record of what applied before. Historical access should remain subject to its own permissions and purpose.

Filter for applicability before asking the model to reason

My Version Drift framework separates the determination of valid material from the search for relevant passages. The application first identifies the approved source set for the case, user, purpose, and time. It then searches or retrieves within that set and carries the source identifiers into the result.

A separate governance index is one possible implementation, not a required product or a guarantee. An existing content service may already hold the necessary metadata and access rules. The important property is that the application can enforce those rules reliably and explain which source set it allowed.

This ordering reduces reliance on the model to notice that a plausible passage is obsolete. It does not eliminate all version errors. Incorrect metadata, ambiguous scope, delayed propagation, incomplete source coverage, and implementation defects can still produce a wrong result. Test those conditions rather than treating an architecture diagram as proof that drift has disappeared.

When applicability cannot be established, make the uncertainty visible. For a high-consequence action, the defined response may be to withhold the action and route the case to the policy owner. For exploratory reading, showing several versions with clear labels may be appropriate. The consequence of the task should determine the response.

Build the retrieval contract before choosing an index

A separate authority index and a semantic index do different jobs. The first resolves which source versions may govern the request; the second locates useful passages inside the permitted set. Separating them can make publishing and investigation easier, but it also creates synchronization work. An existing content service with adequate metadata, access enforcement and historical queries may provide both functions more reliably.

Design	Useful when	Failure to test
Source system resolves applicability and returns versioned content	The owner already exposes effective rules through a supported interface	Source outage or an interface returning only today’s version
Authority registry constrains a separate search index	Several repositories need common applicability decisions	Registry says version 8 applies while search can supply only version 7
Search service enforces metadata and permission filters	Its query path can enforce the required scope before disclosure	Filters omitted by an alternate query path, cache or aggregation
Small application reads approved immutable snapshots	A narrow workflow has few sources and controlled changes	Snapshot publication misses a withdrawal or effective boundary

The two-index design is an architectural option, not a claim that vector stores cannot filter metadata. What matters is the property the service can demonstrate: the model receives evidence from the applicable source set, and the application can identify that set later. A graph, vector database or catalog label does not establish the property.

A useful retrieval contract carries the policy family, case scope, business-effective time and the knowledge time needed for reconstruction. It also identifies the authenticated access context through a trusted channel. A user-supplied word such as “audit” cannot grant access to historical confidential policies.

The following is an illustrative application payload. Identifiers and hashes are placeholders; this is not a protocol standard or a credential:

{
  "policy_family": "supplier_exception",
  "entity": "example-entity-a",
  "effective_at": "2026-10-18T12:05:00Z",
  "known_at": "2026-10-18T12:05:00Z",
  "request_id": "example-request-018",
  "required_evidence": ["approved_rule", "transition_rule"],
  "retrieval_state": "incomplete",
  "reason": "applicable_version_not_searchable",
  "applicable_version": "example-policy-v8",
  "manifest_generation": "example-generation-23"
}

The service has established which version applies but cannot supply it through the available search generation. That is more informative than returning an empty result and letting the model assume there is no additional requirement. Whether the caller may see this diagnostic still follows the service’s access policy.

A small resolver exposes the boundary conditions

This Python example selects an approved version using half-open effective and recording intervals, then checks disclosure permission and the search manifest. A trusted internal resolver needs the complete authoritative record set for the requested family and entity, including records the caller cannot read. The caller’s scope and permitted identifiers are established outside this function. It does not authenticate users, implement a production registry or determine transaction authority.

from datetime import datetime

def instant(value):
    parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
    if parsed.tzinfo is None or parsed.utcoffset() is None:
        raise ValueError("an explicit UTC offset is required")
    return parsed

def contains(start, end, point):
    return instant(start) <= point and (end is None or point < instant(end))

def resolve_version(records, family, entity, effective_at, known_at,
                    permitted_ids, manifest):
    effective, known = instant(effective_at), instant(known_at)
    relevant = [r for r in records
        if r["family"] == family and r["entity"] == entity]
    for record in relevant:
        if type(record["approved"]) is not bool:
            raise ValueError("approved must be a boolean")
        if not isinstance(record["content_digest"], str) or not record["content_digest"].strip():
            raise ValueError("content_digest must be a nonblank string")
        for axis in ("valid", "recorded"):
            start = instant(record[f"{axis}_from"])
            end = record[f"{axis}_to"]
            if end is not None and instant(end) <= start:
                raise ValueError("interval end must follow its start")
    candidates = [r for r in relevant
        if r["approved"]
        and contains(r["valid_from"], r["valid_to"], effective)
        and contains(r["recorded_from"], r["recorded_to"], known)]
    if not candidates:
        return {"state": "incomplete"}
    if len(candidates) != 1:
        return {"state": "conflict"}
    selected = candidates[0]
    if selected["version_id"] not in permitted_ids:
        return {"state": "withheld"}
    if manifest.get(selected["version_id"]) != selected["content_digest"]:
        return {"state": "incomplete"}
    return {"state": "ready", "version_id": selected["version_id"]}

At noon, version 7’s interval ends and version 8’s begins. The end of version 7 is excluded, so the resolver should never choose both merely because the action falls exactly on the boundary. If the source records genuinely overlap, it returns a conflict instead of choosing the most permissive result or the highest version number.

The recording interval answers a separate question. A correction effective at noon may be recorded at 12:20. Reconstructing what the system knew at 12:05 requires the earlier recorded view; reconstructing the corrected business rule requires the revised history. A production system must preserve those views rather than overwrite the earlier record. The example assumes that the source owner has already established the intervals and correction semantics.

Temporal types help implement this model, but do not define its business meaning. PostgreSQL's range types, for example, support bounded intervals. They do not decide which policy family, jurisdiction or transition provision applies. Make that interpretation part of the source contract.

The resolver establishes applicability before disclosure. Hiding a conflicting policy from the caller must not turn another policy into the governing version; withholding the applicable version must not trigger a fallback to an older visible one. Even a conflict or withheld diagnostic can reveal information, so the service must decide which result the caller may receive.

The snippet expects validated, consistently shaped source records and explicit timezone offsets. It checks boolean approval values, nonblank content digests and interval order; it is not a complete schema validator. Malformed timestamps and missing fields can raise different exceptions, which the caller must route to an unresolved-data path rather than treat as permission to continue. It also selects only one governing record for this simplified policy family. Real cases involving several policies, precedence or transitional provisions need an explicit resolution model beyond this example.

Account for the extra retrieval work

Applicability resolution adds a dependency when it is a separate service. Its cost includes permission checks, metadata lookup, constrained search and any reconciliation needed to establish completeness. Keeping a registry next to search can simplify operations; maintaining one authority service across repositories can improve consistency but adds availability and propagation concerns. Neither choice supplies a universal latency figure.

Measure stages separately under a representative mix of routine, historical, conflicting and incomplete requests. Include the time spent returning a useful unresolved result, not only successful searches. A cache can reduce repeated lookup, but its key needs the relevant scope, permission context, time basis and source generation. Withdrawal or revocation must reach the reuse path. A cached answer cannot safely inherit authorization solely because the underlying text is unchanged.

For a narrow workflow, approved snapshots may be easier to validate and operate than a new distributed registry. For a broad collection with several publishers, a shared authority contract can avoid each application inventing its own interpretation. Begin with the source lifecycle and retrieval contract, then compare implementation choices against observed workload and failure behavior.

ContentOps is a publishing responsibility, not another mandatory department

The interval between policy approval and usable retrieval needs an owner. I use ContentOps here for the work of publishing authoritative content into an AI-supported service: preparing metadata, validating applicability, propagating changes, monitoring completeness and handling corrections. An existing records or policy-management team may already own much of it.

Start with the source publisher’s lifecycle. A document can be a draft, approved for future use, effective, withdrawn or superseded for a particular scope. These are different states. The system must not turn “approved” into “effective immediately,” or treat a replacement in one business unit as a global withdrawal.

Publishing step	Evidence produced	What can go wrong
Register source and owner	Stable policy family, version and scope	A duplicate is mistaken for a new authoritative policy
Approve applicability	Approved interval and transition rule	Future approval is treated as present applicability
Capture immutable content	Versioned bytes or controlled snapshot and digest	URL later serves different bytes without a new identity
Prepare search representation	Chunks/embeddings tied to version and generation	Chunk text and metadata come from different revisions
Validate coverage and access	Tests against the intended source set	Restricted titles or passages leak before final filtering
Publish query-ready generation	Manifest linking source version to indexed content	Partial update is reported as complete
Reconcile and correct	Discrepancy report, repair and affected-work list	Failed events leave a healthy-looking but stale index

Do not delay the business meaning of a policy merely because an embedding job is late. If version 8 becomes effective at noon but its index is unavailable, the registry should still know that version 8 applies. The workflow then follows its incomplete-evidence response. Keeping version 7 active for convenience would redefine policy through infrastructure failure.

An index generation can make publication coherent: build and validate the candidate representation, identify exactly which source revisions it contains, and make that generation available through a controlled switch. It does not remove the need to handle withdrawal, revocation or a source that changes before the switch. Track those changes independently.

Use events to reduce propagation delay and reconciliation to discover what events missed. Repeated or out-of-order notifications should not restore an older current pointer. Compare source revisions under the source’s own rules, preserve corrections and quarantine contradictory updates. “Newest timestamp wins” can select a later-arriving obsolete file.

Backfill authority before re-embedding every file

Migration from shared drives and email is usually harder than adding metadata to a clean repository. Several files may contain the same text under different names; one may be a working draft with a later modified time than the approved copy. Deleting duplicates or selecting the newest filename before an owner reviews them can erase the evidence needed to establish authority.

Begin with a bounded policy family. Inventory candidate files with their repository identities, versions, owners, access controls and content digests. Group likely duplicates for review, but preserve the original identities and any meaningful differences. A digest can establish equal bytes; it cannot tell which copy was approved.

Ask the owner to resolve the governing source, its scope, effective interval and supersession relationships. Record ambiguous cases as unresolved. Do not ask a language model to convert an uncertain document lineage into a definitive policy history without review. Extraction can propose dates and references, while the publisher remains responsible for their meaning.

Then ingest the approved set with stable version identifiers and a reproducible chunking/indexing process. Keep a crosswalk from old repository links to authoritative records so that existing applications and bookmarks can be reconciled. Evaluate current and historical queries before withdrawing the old search path.

For a large collection, prioritize by the decisions that depend on it. A policy family governing payments may need complete authority review before action, while a research archive can expose labeled uncertainty. Backfilling millions of documents need not mean applying the same expensive review to every informal note.

Existing document control is a useful starting point, but does not eliminate shadow copies in downloaded files, email attachments, personal indexes or caches. Identify the copies actually used by the application. A perfect source repository is insufficient if the agent still reads an old export.

Recheck the material conditions before execution

Return to the $18,000 supplier exception. The action service should compare the conditions supporting the approval with the conditions that apply when it is about to update the destination. The comparison concerns the policy, transaction values, relevant record state, identity, and authorization scope.

If a material condition changed, use a defined transition. The workflow might recompute the recommendation, ask for additional approval, or stop for an exception review. It should not ask the model to improvise whether an old approval “probably still counts.” The business owns that rule, and the system needs an enforceable representation of it.

There is a concurrency limit to a simple preflight check. A record can change between the check and the update. Where the destination supports conditional updates, version checks, or transactions, use those mechanisms to bind the operation to the expected state. Otherwise, define how to detect and recover from a conflicting update; a separate check followed by an unconditional write is not an atomic guarantee.

A scheduled policy boundary can also pass between the check and commitment while the supplier record remains unchanged. Define which action time and transitional rule govern. If the policy must apply at commit time, an authorization bounded by its effective interval helps only when the destination enforces that time condition with agreed clock semantics. The supplier record’s version check cannot enforce a separate policy registry’s state. Where the integration cannot support the required boundary, use a controlled stop or another supported process. Rechecking after commitment can identify affected work for recovery; it cannot retroactively authorize the action.

The same principle applies to cached recommendations. A useful cache entry needs an applicability and invalidation strategy, not just a time-to-live. A short expiry can reduce exposure, but it cannot establish that no material event occurred inside that interval. Treat high-consequence reuse accordingly.

Keep an evidence path another person can follow

The record should connect the request, relevant case state, source versions, selected applicability rule, recommendation, approval, execution attempt, and confirmed result. Preserve enough of the material inputs or their controlled references to reconstruct the decision under the organization’s retention and privacy requirements.

A URL and timestamp alone may be insufficient if the URL later serves a different document. A hash can help identify the bytes used, but it does not prove that the policy was authoritative or applicable. Keep the content identity and the authority record together. Avoid treating a technical integrity check as a substitute for business provenance.

The enterprise AI measurement article explains why changed policy and configuration versions belong with the outcome measures. Without that context, a deteriorating acceptance rate can look like a model problem when the workflow is actually applying a different rule.

For broader risk work, NIST’s Generative AI Profile provides suggested actions around information integrity, provenance, testing, and monitoring. A source-validity control can support that work. It does not establish compliance with every obligation that applies to the organization.

Test changes at the boundaries that matter

Use a synthetic policy pair with known effective dates and a transaction that falls between the two approval thresholds. Prepare and approve the case before the change, then attempt execution afterward. Record the expected transition before running the test. Confirm the destination state as well as the response shown to the reviewer.

Extend the test set with a future-dated replacement, a withdrawn policy, two conflicting approved records, a user without access, a delayed search-index update, a cached recommendation, and a request to reconstruct a historical decision. Test the period immediately before and after the effective boundary, with an explicit timezone.

Include a failure of the authority service itself. The system needs a defined behavior when it cannot establish the applicable source set. Quietly falling back to whatever text is available may defeat the control precisely when it is most needed.

The useful output is a small body of evidence: which cases passed, where the workflow deferred or stopped, which result remains uncertain, and who owns the correction. Repeat the affected cases after a material change. A passing test set establishes what was observed for that configuration; it cannot establish that every future policy transition is safe.

Measure applicability rather than rewarding the newest version

A metric that counts every retrieval of the latest document as correct will penalize valid historical answers. Build expected source sets around the query’s scope and time. Some questions legitimately need several versions to explain a transition; others require one governing version.

Measure	Definition for a chosen test set	Interpretation limit
Applicable-source precision	Returned source versions in the expected applicable set divided by returned versions	High precision can coexist with missing useful sources
Applicable-answer coverage	Answerable authorized cases completed with the required source evidence divided by those cases	An abstaining system should not appear perfect by never answering
Citation support	Assessed claims supported by their cited passages	A supporting passage can still come from an inapplicable version
Boundary-transition failures	Cases that violate the specified response at an effective boundary	The denominator must include transitions and failures, not only routine requests
Propagation lag	Observed delay between a source change and its correct availability/status in the service	Clock meaning, planned effective time and withdrawn states need definition
Unresolved outcomes	Attempts whose destination effect has not been established	Silence or timeout must not be counted as a confirmed rejection

Keep undefined ratios visible. With no returned sources, precision is not automatically 100%. Choose how to report it alongside coverage. Likewise, an authority-service outage is not an obsolete-document error, even though both may prevent a reliable answer. Separate causes so the right owner receives the repair.

Compare the same policy families, time bases, permissions and task mix across releases. A changed evaluation set can make apparent improvement meaningless. Bank or hospital improvement claims need measurements from the actual deployment, with the workload, evaluation conditions and denominator stated.

A batch of actions needs an affected-work list

Consider a synthetic service preparing 40 supplier exceptions under version 7. Twelve have already completed, eighteen are awaiting review and ten are queued for execution when version 8 takes effect. The counts are illustrative and sum to the same batch; they are not observed production results.

The control should identify which cases depend on the changed rule and classify them by state. Pending reviews can be refreshed. Queued actions may need additional approval or cancellation before dispatch. Completed transactions require the business owner’s correction process if the change or a backdated correction affects them; they cannot be undone by deleting a cache entry.

The affected-work query should use recorded dependencies: case identifier, source version, applicability rule, approval and destination operation. Searching generated prose for the words “version 7” is a weak substitute. Retain enough lineage to identify which decisions consumed an incorrect or withdrawn source.

Domain differences change the response. A synthetic credit workflow may need the product, borrower scope and application date to choose a rule. A hospital administrative workflow may need the facility, service date and scheduling authority. These examples identify context requirements; they do not establish lending or treatment eligibility, legal compliance or a measured institutional benefit.

Run an ambiguity stress test with the source owner

Prepare two plausible policies whose wording is similar but whose scope or effective interval differs. Ask the same question for each entity, just before and after the boundary, and for a historical date. Include an approved future version, a withdrawn source and an incomplete index generation.

Then make the system’s evidence disagree deliberately. Supply two overlapping approved records, delay a source notification, remove access to a policy after caching an answer, and enter a correction with a backdated effective period. Define which cases should answer, label alternatives, defer or block an action. A global rule to pick the newest version should fail several of these tests.

Rehearse recovery, not only detection. Identify affected recommendations, invalidate the relevant cache entries, obtain any renewed approval and reconcile queued or uncertain destination operations. The source owner should be able to explain why the repaired version applies, while the operator demonstrates what changed in the running service.

Make source maintenance part of operating the service

Someone has to own the interval between a source change and its effective use in the application. Measure that interval, track failed propagation, and provide a way to withdraw affected work. Include the people who publish the policy in the service’s change process rather than asking engineering to infer authority from document filenames.

There is a cost to this design. Many informal research tasks do not need transaction-grade version controls, and imposing them everywhere would create unnecessary friction. But when an AI-supported decision depends on a policy or can alter a consequential record, source validity is part of the service’s correctness.

I would begin with one policy-governed workflow and the noon-boundary test. It exposes the handoff between content ownership, retrieval, approval, and execution in a form each owner can inspect. Once that path works, extend the controls to the next decision class with the evidence from the first one still available.

Share or cite this article

LinkedInEmail

Ajith Vallath Prabhakar. “Version drift: when a well-sourced AI answer uses the wrong policy.” ajithp.com, 5 October 2025. https://ajithp.com/2025/10/05/enterprise-ai-version-drift/