The hidden cost of ROT in AI context engineering

Context engineering is emerging as the next major shift beyond prompt engineering in enterprise AI. Prompt engineering treated each query as a one-off task, refining wording for a useful answer. Context engineering is different: an architectural discipline that automates the ongoing supply of data into AI systems, so the right information reaches a model continuously rather than through a single query.

 

That automation raises the stakes. A flawed data set once caused a flawed answer – but in an automated pipeline, the same flaw is reproduced on every run, multiplying its effect across every token spent.

 

This is where ROT – redundant, obsolete, and trivial data – becomes a serious concern for CIOs and CTOs. Left unmanaged, it drives up the cost of running AI systems and degrades the accuracy of their output, undermining the productivity gains and better business outcomes that organisations expect from their AI investments.

 

Why ROT costs more in the AI era

Redundant data inflates the volume a model must process, and since most AI systems charge by token consumption, duplicated content adds cost with no added value. Meanwhile, trivial data introduces noise that obscures the details that matter, reducing precision. And obsolete data is the most dangerous of the three: a model cannot recognise that a fact is out of date unless told, so superseded information gets presented with the same confidence as anything current.

 

These risks intensify because of how large language models handle long context. Models are prone to a primacy-recency effect, weighting information at the start and end of a context window more heavily than material in the middle. The more unfiltered data that a pipeline pours in, the greater the chance something important gets lost.

 

Redundancy carries a subtler risk too: repetition can be mistaken for importance. A system may treat frequency as a signal of significance rather than duplication, underweighting rare but critical facts relative to noise that simply repeats more often.

 

Making data governance AI-ready

Addressing ROT starts with established data governance principles rather than any AI-specific tooling. Defining clear sources of truth for each data domain or attribute gives a model a reliable anchor for its reasoning, instead of leaving it to reconcile conflicting versions of a fact unaided. That clarity should be built into the instructions and metadata governing context.

 

A useful way to think about this is in three layers: raw data, the metadata describing what it means, and a contextualised layer explaining how each point relates to everything else, including its relative weighting. Maintaining that layer, akin to an ontology for the organisation’s information, sits at the heart of effective context engineering.

 

None of this removes the need for people. Data catalogues, schema definitions, and quality tools can support the work, and AI itself can help surface potential duplicates or ageing records, but someone still has to decide what counts as a source of truth, given how much data most organisations have accumulated across disparate systems. 

 

Sequencing matters too. Rather than letting a system decide which data to retrieve first, organisations benefit from prescribing an order of operations, much as a mathematical expression depends on a defined order of operations. Without that structure, a system may retrieve information in a different order each run, producing inconsistent answers to the same question.

 

Keeping drift – and ROT – in check

Because context engineering is typically deployed for narrowly scoped use cases, the case for managing ROT aggressively grows stronger, not weaker. A tightly defined tool built around well-specified instructions can still drift if its own responses are folded back into context, gradually skewing outputs away from the original intent.

 

Periodically resetting a system to a clean baseline is one way to counter drift, though doing so indiscriminately carries its own cost, since reprocessing the same foundational information repeatedly wastes tokens. The right cadence for a reset, and the point at which drift becomes material, is specific to each use case and calls for ongoing measurement.

 

Industry context shapes these decisions too. Legal services, with longer retention obligations, may weight historical context differently than sectors built on high-frequency transactional data – like finance or telecommunications – where recent information typically deserves more weight than a longer average. As with a moving average, the right window depends on how quickly the underlying facts change.

 

As organisations deploy multiple context-engineered tools drawing on overlapping data, there is a strong case for abstracting shared components, much as reusable functions are abstracted in software development, so common baselines are maintained once rather than duplicated per use case.

 

That same logic applies to ROT itself: curbing it is less a one-off clean-up than an ongoing design discipline, built into how context pipelines are assembled rather than patched after the fact. Organisations that build this rigour in early will not just curb spiralling AI token costs – they will be setting themselves up for ongoing and enduring AI success.

 

---

 

Joe Logan is CIO at iManage

 

Main image courtesy of iStockPhoto.com and MF3d