IRIS: A Critical Look at Intelligent Document Histories
Transcript of the narrated video (21m 15s), made with Armature
Paper: IRIS: Navigating and Reflecting on Writing Traces Using Intelligent Document Histories (Zhou, Chen, Chung, Sterman), arXiv 2608.19614
Watch: https://armature.ai/demos/iris-hai-analysis

[0:00] The Missing Work Behind a Sentence

IRIS and the Negotiable Mirror of Writing. Picture finishing a sentence, then realizing the versions you tried and deleted might tell us something important about how you made it. A polished document hides that work, so we need a way to make those choices visible and examine what they mean. This video is about IRIS, a writing system that turns those traces into something a writer can reflect on, without handing the writing over to AI. Okay, let's follow the change from “The room was quiet” to “The room was silent.” The sentence shifts, but the first wording is still here for a moment, which lets us see that the finished line had an alternative. And now the sentence changes again to “Nobody answered.” These are three different ways of expressing the moment, but only the last one is becoming the finished line. Once those earlier wordings disappear, we're left with just “Nobody answered.” That gives us the finished sentence, but not the path that produced it.

Let's bring those earlier versions back. “The room was quiet” and “The room was silent” are alternatives to the surviving sentence, and seeing them together makes the missing work easier to notice. Your process can become something you inspect. That is the central possibility here: instead of seeing only the sentence that survived, a writer can look at the choices and changes behind it. But what exactly can those traces tell us, and where does an interpretation begin?

[1:26] The Route Through IRIS

First, we’ll follow how IRIS turns writing traces into interpretations. A trace is a record of actions like typing or deleting, and we’ll see how those records can help writers reflect on their work. Then we’ll read the interface and its process graph, which lets us see how a writer’s activity unfolds across the document and over time. After that, we’ll examine what the studies establish about how writers used IRIS and what those findings can tell us. Finally, we’ll use the idea of a negotiable mirror to connect the design to human agency and future research. And we’ll begin with how traces become interpretations.

[2:03] From Traces to Interpretations

IRIS is a research writing editor described by David Zhou and colleagues. An editor is where a writer works on a document, and IRIS keeps a history of how that writing changes. And just to be clear, we're walking through what the paper describes, not a hands-on session with the system. Okay, let's follow what the editor sends into the trace log. A trace is a record of an action as it happens, so here the entries are typing, deletion, and pause. That gives us a record of activity the finished document may no longer show.

So the trace log gives us actions, but not yet a writer's thoughts. The process parser is a set of rules that uses activity like these entries to sort writing into broad process categories; it isn't reading the writer's mind. Flower and Hayes describe three broad parts of writing, and those are the labels we're about to look at. Flower and Hayes call those processes planning, translating, and reviewing. Translating here means turning ideas into written words, while reviewing means looking back over what you've written. IRIS adapts earlier keystroke-analysis rules—rules that use patterns in typing activity—to infer related writing states.

Let's connect the parser to version history, where the writing is kept as a sequence of records. A long pause before a burst of typing is treated as a clue to planning, while rapid production is treated as drafting. These are clues drawn from activity, not direct observations of what the writer intended. Editing-heavy activity is treated as reviewing, with local revision for changes to a small passage and global revision for broader changes across the document. Those labels help organize the history, but the action recorded and the writer's intention remain different things.

Now the question is how a writer can ask about that history. A language model is an AI trained to generate text; here it receives history material along with the writer's question, then returns an interpretation. For Process Analysis, the paper says the full history and topic are sent to the model. So the explanation can draw on that broader record, rather than only one isolated change.

Here are three different layers to keep apart. The action is recorded, the process category is inferred from activity, and the model generates an explanation from the history and question. And the answer comes back as a cited interpretation. The citation points back to versions in the history, so the explanation is connected to the record without becoming the same thing as the record. The authors aren't claiming that logging activity, filtering it, or using a language model is new on its own. The idea is how these pieces are brought together for writers.

The two questions beneath the pipeline capture that design. One is where a change sits in the text; the other is what matters to this particular writer. That takes us from the record itself to the next question, which is how the interface lets writers explore that history.

[5:08] Reading the Interface

Okay, let’s connect a sentence in the document to the changes that led to it. The highlighted passage is repeated in the history below, where red text marks wording that was removed and green text shows what replaced it. Here, “grow” becomes “end up growing,” so you can inspect that wording choice without hunting through whole-document snapshots. That’s the useful shift here: a passage becomes an entrance to its own history. The text stays connected to the earlier versions that help explain how it took shape.

And there’s another reason this history belongs near the passage. In the first prototype, writers had to move to a sidebar to inspect changes, but their curiosity started with the text itself, so the redesign brought the history alongside the highlighted sentence. The darker shading marks more revision, not worse writing; deleted passages and pauses can also matter even when no words survive.

A heavily revised passage could still leave the writer unsure about a choice, so revision alone doesn’t tell us what deserves attention. Could it still be unresolved? And the reverse is worth asking too. An untouched passage might simply have been overlooked, so could it be worth revisiting? Neither pattern gives a verdict. The writer decides what needs attention by turning the history into a question about their own work.

Let’s turn from where a change happened to what kind of change it was. The cards show tags such as “wording changes,” “style,” “big deletion,” and “discussions of worldbuilding,” each attached to a revision. The examples make clear that a tag can describe a kind of change or its subject, without needing to read every excerpt. Here, the writer supplies the concept, and the system assigns the tags to relevant changes. That gives the writer a way to find revisions through ideas that matter to them; writers can also add or remove tags themselves.

There’s another way to narrow the history, and it uses process type rather than the concepts pictured on these cards. Process types are labels for activities such as drafting, revising, or pausing, which let writers filter the history by what kind of work was happening. Planning and pauses can leave no visible text change, so this filter can surface activity that a comparison of the words alone would miss.

Now let’s shift from finding edits to asking what pattern they make. Process Analysis is the view that turns revisions into a written explanation. Here, it groups wording changes into a narrative and gives examples tied to cited versions, including a summary across six versions, V1 through V1839. Those citations keep the changed wording close to the explanation, so it’s easier to check where a claim comes from. That’s the key connection: the explanation stays close to the evidence. You can compare the account with the cited writing rather than treating the summary as a free-floating answer.

The other view turns that history into a conversation. Ask Your Writing lets the writer ask, “Where did I have trouble finding the right words?” and receive a shorter answer linked to cited passages and revision ranges. So the answer gives you places to inspect, not proof by itself that the interpretation is correct or that the writing has improved. A cited answer is an invitation to inspect. The references let the writer check the account against the actual revision history, while the judgment about what it means remains theirs.

[8:32] What a Trace Cannot Tell You

Okay, let’s start with a made-up edit, so we can separate what the record shows from what we might read into it. Mara goes from refusing to help to hesitating before she helps. The change is visible, and we can keep both the removed wording and the added wording. But the record of the edit still doesn’t tell us what the writer meant by making it. Here are two possible readings. Maybe the writer softened the wording, or maybe the writer changed Mara’s motivation. Those are interpretations, not outputs from IRIS, and the edit by itself can’t tell us which one is right.

Let’s shift from those possible readings to two different things we might want to understand. Procedural evolution is how the writing activity unfolds, like drafting and revising, or pausing. Semantic evolution is how the ideas and motivations change. They’re connected, but a record of the activity doesn’t directly tell us what the writer meant. And a recorded pause tells us that the writing stopped for a while, not what the writer was thinking or intending. That’s the measurement gap here. We can see the action, but figuring out its purpose takes interpretation, and that’s where the author’s context matters.

Let’s take the same passage into a different kind of revision, where its place in the document changes. We’ll follow it from the opening toward the ending and see what happens to the history attached to it. As the passage is cut from the opening, a deletion marker takes its place, while the earlier history remains there. Now the passage is inserted into the ending, but its earlier history hasn’t moved with it. The insertion and deletion are recorded at different places, so the text’s path is no longer attached to the passage in its new location. That’s the boundary in IRIS. Cut and paste is treated as a deletion and an insertion, so moved text does not carry its history along with it.

[10:27] Reading the Process Graph

Okay, let’s read this process graph from the ground up. The vertical position tells us where an edit lands in the document, not how many words were written. So the upward drafting sequences track movement through the text. And the horizontal axis tracks time, but it isn’t a perfectly proportional clock. Gaps longer than an hour are shortened to ten minutes so writing sessions spread across multiple days fit on the chart. Planning and pause states are hidden because they don’t change the text.

Now let’s compare those earlier upward drafting sequences with the later editing points. The graph shows drafting moving forward through the document, while later structural edits reach across different text locations. That suggests writers are returning to existing material, but the pattern alone doesn’t tell us why each return happened.

There’s a useful difference between a graph showing a pattern and a writer deciding what that pattern means. W11 reflected on their own graph, not necessarily this example, and interpreted tinkering as testing words. They also saw it as a way of finding the next sentence. That’s metacognition, or noticing and interpreting how you think and work, rather than simply producing another sentence.

[11:39] What the Studies Establish

Okay, this is where the evaluation begins. First, fourteen creative writers tried an early version of the system. The researchers wanted to learn from writers' own work before shaping the redesign. They wrote for about thirty minutes before exploring the features, so the researchers could see how the prototype fit into their writing practice. That formative study—the early trial used to guide changes to the system—helped shape the redesign. Then eleven participants tried the revised system in a diary study, meaning they used it over time and reported on that experience rather than taking part in just one short session. Their study lasted from one to three weeks, giving the researchers a view of use across repeated writing sessions. Across those participants, the diary study included seventy-four writing sessions altogether.

Now the focus shifts to the diary study and what made it useful for understanding writers' experiences. Each writer took part in four to seven sessions, so the researchers could hear about the system as people returned to their own work. The participants wrote pieces typical of their practice, rather than writing an assignment invented just for the study. And interviews explored how writers used the system. That gives us accounts of how they made sense of it, but the evidence here is about their experiences, not a measured change in writing quality.

Let's turn from the study setup to the reported usage. On average, each participant had eight hundred seventy-nine version states—that's saved points in the writing history that let writers revisit how their text changed. They also averaged seven point nine Process Analysis queries. Process Analysis is the system's feature for asking about patterns in a writing history, so these queries show people using it to investigate their process. And they averaged three point five chat sessions, using the conversational feature to ask about their writing history. Fifty-two percent of chat sessions had more than one turn, meaning the writer and system went back and forth at least once. So the authors use these numbers as evidence of engagement with the system. They don't measure whether the writing got better.

But the study also asks what writers did with an interpretation. For W2, the system's account of character development offered confirmation about an area of the writing. That confirmation helped W2 move attention elsewhere, rather than keep working on character development. W5's case starts from a different concern. Organization was troubling, and the system's interpretation helped make that concern more specific. For W5, the concern became a specific improvement target, giving the writer something concrete to work on.

Let's compare two other responses, because the same kind of feedback didn't land the same way for everyone. W2 found an elaborate account encouraging. W1, by contrast, found familiar feedback frustrating. So a writer's response depended on their goals and on when that feedback arrived. That contrast points to goals and timing as part of how writers interpreted the system's output.

Let's move from those individual responses to a reported pattern. A writer has what they intended to say, while the system offers its own reading of the text. Putting the two readings side by side can prompt a useful question, namely, am I conveying what I mean? The authors describe this as a reported pattern, not an accuracy test. The contrast can help a writer reconsider how the writing comes across, but it doesn't establish that the system's reading is correct.

So the authors' conclusion is that writers used the traces as material for interpretation. That tells us what the evaluation supports, and the next question is what it leaves open. The study doesn't isolate which feature led to which benefit. The authors say feature-removal studies are needed to find out what each feature contributes. And because the study covered a limited stretch of writing, it doesn't show how the system would fit into the life of a whole long-form project. The authors leave that for future work.

[15:48] The Negotiable Mirror

Okay, the studies showed writers interpreting IRIS's output. Now let's use that to think about the relationship itself. This is an analytical lens, not another feature of IRIS. The writing record feeds into an AI account, so we can think of the system as a kind of mirror, reflecting a pattern in the writer's past activity. But this isn't a mirror that gets the final say. The writer brings remembered goals to the interpretation, and can decide whether that account fits.

And the writer's judgment also brings in context the system's account doesn't supply. That's why the account is something to consider, not a verdict. So how can a writer examine an AI account rather than just accept it? Here, the evidence view gives the account references back to the writing record. The cited revisions then reach writer judgment, and the evidence makes the interpretation inspectable. In human-AI interaction, that ability to examine and challenge a system's account is called contestability.

A citation lets you check what changed, but it doesn't tell you why you made that change. The record can show the revision; its purpose still needs the writer's own experience. So the writer brings remembered goals together with the evidence. That meeting is what makes the interpretation useful for deciding what the history means. Okay, let's shift from checking the account to deciding what happens next. The new-writing connection points back toward the record, but the decision belongs to the writer.

The writer might accept, challenge, or reframe an interpretation, and then choose a next action. The point isn't for AI to supply another paragraph; it's to support reflection on the writer's own practice. The authors call this a non-generative role for AI in creative work. Here that means the AI isn't being used to produce the writer's next passage. The model still generates explanations. What changes is their purpose: they support the writer's decision, rather than taking over the writing.

Now let's widen the lens beyond the paragraph itself. The proposed direction is a tool that responds to the decisions, attention, and effort behind the writing, as well as the finished words. Being good with language isn't the same as understanding someone's writing practice. The account needs to make room for the path behind the text. The same paragraph can come from different paths, and those differences may matter when deciding what kind of support would help.

But reflection can affect the writer too. When a system reflects your process back to you, it may change how you behave while writing. The reported experiences include W5 writing more slowly and W1 feeling an observer's presence. So reflection isn't always a neutral experience. W4 used highlighting during revision. That is a reported practice, not evidence of a new privacy or logging control.

The authors point toward giving writers a choice about visibility, so reflection is available without making the writing process feel like an unavoidable performance. One proposed future direction is author annotation, letting writers record why a revision happened. This is a proposal, not a feature IRIS already provides. That annotation could connect an observable substitution to the meaning the writer intended, adding context the trace itself doesn't contain.

There's another reason that context matters. Categories for writing processes can shape what looks normal or desirable, rather than fitting every writer equally well. So the proposed direction is to make room for the writer's own perspective, instead of treating one account of how writing should happen as universal. Now there's a boundary to this richer history. Writing records can include sensitive material, so where that history goes matters.

The AI account uses a commercial language model, meaning a model provided as a commercial service rather than kept entirely within the writing tool. The authors raise privacy and intellectual-property concerns about using it with creative records. The authors also report that their local-model prototypes did not meet the needed quality, speed, and context capacity. A local model would run on the user's own device or environment, but in their prototypes that alternative came with these limits. So a useful research question is how to preserve reflection while giving writers more control over their creative records, and reducing disruption and exposure.

Let's bring the whole relationship together. IRIS's contribution isn't an AI verdict about your writing; it's a way to make past writing decisions available for the writer to consider. The writing record can feed a new conversation, and the writer decides what to do with it. For that mirror to be useful, it needs to stay grounded in the writing, open to disagreement, and under the writer's control.

And the useful answer may be a better question about your own work. The authors' conclusion reaches beyond editing words. Reflection is part of creative expression, and here the writer's judgment opens two possibilities, reconsidering goals or revising words. As reconsidering goals connects back to the writing record, we can see why reflection may change what the writer is trying to accomplish, not just the wording.

That leaves a broader point from the authors: writing tools shape the text and process, and also how writers understand their own practice.