AI Context Management: RAG vs Long Context

RAG (Lewis et al., 2020) splits parametric memory from a swappable index: retrieve only when the window cannot hold the corpus. Current Claude windows are in Anthropic context windows—1M by default on Fable / Opus / Sonnet 5, 200k on Haiku 4.5. If count_tokens says the bundle fits and the task needs cross-section reasoning, stuff it. Index when the corpus is larger than the window, or when documents must be replaced without resending the rest.

Select / skip

Count tokens first. A larger window is not a quality guarantee. Anthropic’s own context-window doc names context rot: as the token count grows, accuracy and recall degrade. Filling the window is not the same as reading the middle of it.

Headroom is not a round number pulled from a blog. The window counts the whole turn: system, history, tool defs, tool results, thinking, output. Anthropic: input alone over the window → 400. On Claude 4.5 and newer, input + max_tokens over the window may be accepted, then stop with stop_reason: "model_context_window_exceeded".

Tokens, cost, freshness

Count with the target model’s token counting API. Do not estimate Claude tokens with another vendor’s tokenizer. Official note: the tokenizer from Claude 4.7 onward counts about 30% more tokens on the same text. Recount against the model that will run.

Rates below are from Anthropic pricing, checked 2026-09: Sonnet 5 input $2 / MTok, output $10 / MTok, cache read $0.20 / MTok (0.1×). Haiku 4.5 input $1 / MTok, 200k window. From Claude 4.6 on, the 1M window is billed at the standard per-token rate—no separate long-context surcharge. 1M is the default; no beta header. Prompt cache changes the bill only: cached prefixes still occupy the window. Haiku 4.5 is capped at 200k. The same 180k corpus still has room on Sonnet 5 and is already tight on Haiku once tools and thinking are added. Route on window first, then on unit price. Treating Haiku’s 200k cap as a quality cliff is unsourced. The vendor guarantees an error past the window, not that the middle of 199k is still readable.

A stable 200k-token corpus stuffed into Sonnet 5 on every query:

• Cache miss: 0.2 × $2 = $0.40 / query (that input only)
• Cache hit: 0.2 × $0.20 = $0.04 / query
• 5-minute cache write at 1.25×: about $0.50 the first time

The same question with about 8 × 512-token chunks: ~4k tokens → $0.008 / query of model input. Embedding is extra, paid at index time, amortized over dirty documents.

The boundary is arithmetic, not an accuracy table:

• Fits, repeated, stable: stuffing + cache. A couple of hits repay the 1.25× write.
• Fits, but the corpus changes every time: cache does nothing. 200k at full price versus 4k retrieved is roughly two orders of magnitude. RAG is cheaper.
• Does not fit: stuffing is not a quality knob. RAG or hybrid.
• Freshness: weights have a cutoff (Sonnet 5 reliable knowledge through 2026-01). Facts after that cutoff are invented unless they are in the prompt or the index. The RAG paper swapped 2016 and 2018 Wikipedia indexes without retraining the generator. The failure mode is the inverse: files on disk moved, vectors did not.

Spread over volume: 100 uncached 200k stuffing queries/day is about $40/day of input. The same 100 queries at 4k retrieved tokens is about $0.80/day. If the prefix is stable and cache hits, 200k stuffing falls to about $4/day—and that is the case that matches “stuff the tree for cross-file reasoning.” Low volume, corpus rewritten daily, cache almost never hits: do not pay 200k because the window is 1M.

Count, then stuff (long context)

Needs ANTHROPIC_API_KEY. Shape follows the official token-counting and Messages APIs. Runnable-looking; not executed in this repo.

from pathlib import Path

import anthropic

MODEL = "claude-sonnet-5"
WINDOW = 1_000_000  # Haiku 4.5: 200_000
HEADROOM = 32_000  # tools / thinking / output; raise for the real request

client = anthropic.Anthropic()
files = sorted(Path("./data").rglob("*.md"))
corpus = "\n\n".join(f"## {path}\n{path.read_text()}" for path in files)

counted = client.messages.count_tokens(
    model=MODEL,
    messages=[{"role": "user", "content": corpus}],
)
print("corpus_tokens", counted.input_tokens)

if counted.input_tokens > WINDOW - HEADROOM:
    raise SystemExit("does not fit; RAG or hybrid")

msg = client.messages.create(
    model=MODEL,
    max_tokens=2048,
    messages=[
        {
            "role": "user",
            "content": (
                f"{corpus}\n\n"
                "Question: What is RateLimiter's default burst?\n"
                "Answer only from the text. Say so if it is not there."
            ),
        }
    ],
)
print(msg.stop_reason, msg.usage)

stop_reason == "end_turn" is a normal stop. "model_context_window_exceeded" means the window filled and the answer was cut. "prompt is too long" means the count or the HEADROOM was wrong: generation never started.

Lost in the Middle still applies: relevant spans at the start or end of the context beat the middle. Put the question after the corpus. Do not bury the one file that matters in the middle of twenty hits. That is a positional result, not a made-up accuracy number.

RAG pipeline: chunk, embed, k, rerank

Self-hosted indexing via LlamaIndex. Documented defaults: chunk_size=1024, chunk_overlap=20, similarity_top_k default 2. Smaller chunks mean more nodes per file—raise k or the retrieved word count falls.

Change the embedding model and the index must be rebuilt. Query-time embeddings must match index-time embeddings. That is a hard rule in the same LlamaIndex page, not a tuning hint. OpenAI embeddings text-embedding-3-large is a different width from ada-002. Mixing them is noise.

Rerank with Cohere Rerank: retrieve a wide k, then keep top_n for the LLM. The LlamaIndex example is similarity_top_k=10 plus CohereRerank(top_n=2). k=2 and no rerank leaves ranking entirely to embedding ANN.

Schematic; not run in this repo. Needs llama-index, llama-index-embeddings-openai, llama-index-postprocessor-cohere-rerank, plus OPENAI_API_KEY / COHERE_API_KEY.

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.postprocessor.cohere_rerank import CohereRerank

Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-large")
Settings.text_splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

query = "What is RateLimiter's default burst?"

retriever = index.as_retriever(similarity_top_k=10)
hits = retriever.retrieve(query)
print("k", len(hits))
for node in hits:
    print(round(node.score, 4), node.node.metadata.get("file_name"), node.get_text()[:80])

engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[CohereRerank(top_n=3)],
)
response = engine.query(query)
print(response)
for node in response.source_nodes:
    print(node.node.metadata, node.get_text()[:120])

Inspect hits before the prose. Empty or off-topic hits will not be fixed by a better prompt.

Chunk size is a select/skip, not a default to copy. FAQ, policy, changelog: 512 with overlap 50—short question, short answer; a large chunk dilutes the embedding. 1024 is the documented default when the question needs neighboring sentences. Do not run a generic SentenceSplitter across a codebase. The Node Parser modules include CodeSplitter(language="python", chunk_lines=40, chunk_lines_overlap=15). Markdown with headings belongs in MarkdownNodeParser, so a heading does not detach from its table.

The same LlamaIndex strategies page also covers hybrid search: embeddings miss exact identifiers (function names, error codes, issue ids). When a keyword hits and the vector ranking does not, add BM25 or the vector store’s own hybrid. Raising k to 50 only feeds the reranker more candidates. If the identifier never appears, fifty neighbors are still wrong.

Hybrid: retrieve files, read them whole

Snippet RAG will not answer “what is the data flow through this module.” The unit is not a 512-token lookalike sentence. It is a handful of complete files.

Schematic:

from pathlib import Path

hits = retriever.retrieve(query)
paths = []
for node in hits:
    name = node.node.metadata.get("file_name") or node.node.metadata.get("file_path")
    if name and name not in paths:
        paths.append(name)

bundle_parts = []
for name in paths[:4]:
    path = Path("./data") / name
    if path.is_file():
        bundle_parts.append(f"## {path}\n{path.read_text()}")
bundle = "\n\n".join(bundle_parts)

counted = client.messages.count_tokens(
    model=MODEL,
    messages=[{"role": "user", "content": bundle}],
)
if counted.input_tokens > WINDOW - HEADROOM:
    raise SystemExit("retrieved files still overflow; drop files or split by path prefix")

msg = client.messages.create(
    model=MODEL,
    max_tokens=2048,
    messages=[{"role": "user", "content": f"{bundle}\n\nQuestion: {query}"}],
)

Hybrid bills the files that hit, not the whole tree and not unrelated sentences. A missed file shows up in paths, not as a fluent wrong paragraph.

How failures show up

Four faults, four observables. No unsourced accuracy table.

Bad chunk. Heading in chunk A, table in chunk B; signature split from the function body. Visible: get_text() starts mid-sentence or as a lone ##; the cited filename is right, the quoted span does not contain the answer. Fix: a structure-aware parser, then a handful of questions whose answer must live in one chunk.

Embedding mismatch. Indexed with model A, queried with B; or the model changed and the index did not. Visible: empty hits, or healthy-looking scores on unrelated text; a synonym query reshuffles the ranking. Vectors of different width in one table do not have a defined nearest neighbor. LlamaIndex: change the embedding model, rebuild; query with the same model.

Multi-hop. “Who authored this function, and when was it last changed” needs the code and git metadata. One embedding retrieve hugs the wording of the question. Visible: source_nodes are the function body, no author or date; the model invents the missing hop and still cites the body. Fix: two retrieves, or hybrid-load the file and add structured evidence (git log -L). Top-k sentences are not a historian.

Stale index. Files moved, vectors did not. Visible: the cited paragraph is gone from disk, or contradicts the current file; hits still use the old API name. Check: ask a fact that changed this week; source_nodes still quote the old sentence. Fix: dirty documents by mtime / git sha, re-embed those only. “Index behind the repo” belongs on the release checklist, next to process liveness.

Empty hits, wrong citation, context overflow are three checkpoints on one chain: len(hits)==0, citation that does not match the file, prompt is too long / model_context_window_exceeded. Fluent prose that never hits those three is not a test.

Check the file and the dirty set with commands, not impressions:

# Schematic. Point persist at the local index dir. Source newer than docstore = index behind the repo.
find ./data -type f \( -name '*.md' -o -name '*.py' \) -newer ./storage/docstore.json

# Schematic. The cited span must exist in the current file; otherwise the citation is wrong or the index is stale.
from pathlib import Path

node = response.source_nodes[0]
cited = node.get_text()
name = node.node.metadata.get("file_name")
text = Path("./data", name).read_text()
print("empty_hits", len(hits) == 0)
print("citation_in_file", cited in text)
print("score", None if not hits else hits[0].score)

citation_in_file is False usually means whitespace changed in the splitter, or the sentence was deleted. Treat the first as a looser “key sentence in file” check; treat the second as a stale index. For multi-hop, write a question whose answer cannot live in the retrieved class of files: index *.py only, ask for author and date. source_nodes must not grow a git log. If they do, the model filled the hop.