Working with long documents and code

A chat that started well and went sideways an hour later did not get worse. Its beginning fell out of the context-window.

What happens

On each call the system prompt, the conversation history, attached files and the answer being written share one budget, counted in tokens. Go over it and the oldest turns are dropped. Your careful setup goes first, and nothing tells you.

Two fixes

Move the rules somewhere that does not scroll away. That is the job of a system-prompt, and it is the lasting fix.

When you notice drift, restate the constraints. One line, as before: minimal edits, no reformatting, costs nothing and pulls the session back.

A big window is not perfect recall

Models retrieve well from the start and end of a long input and worse from the middle. A million-token window is no reason to paste a million tokens. Put the instruction and the part you care about at the edges.

Handing over code

Paste the file and skip the repository. Reviewing one function does not need your whole project, and the extra context costs money, time and accuracy.

If a file leans on something you have not shown, say so. End the prompt with: if part of this depends on code you cannot see, say which part. You get a question back instead of an invented implementation.

Use real paths. config/functions.php:41 is something you can act on. The helper function is not.

Chunking long documents

When the input really does not fit, cut on structure: chapters, sections, agenda items. Cutting on character count gives you a summary of half an argument.

Run the same prompt on each chunk and merge the outputs in a second pass. Summarising summaries loses less than a single call that saw a truncated document.

When it still drifts

Start a new conversation and paste a short handover: the goal, the decisions made, the current state. A fresh window with a good brief beats a long one carrying forty turns of dead talk.