A context analysis of a failed code-review revision: Kimi K3 and LLVM PR #220979

Abstract

Kimi K3 submitted PR #220979 to LLVM over one long session. After the review arrived, K3 proposed three revisions in the original session, none of which adopted it; GPT-5.6 Sol, however, handling the same task in a new session, produced on its first attempt a revision logically equivalent to the final merged version. This contrast may stem either from a gap in model capability or from the context of the original session. To separate the two, I truncated the original session at the decision point where K3 received the review, applied single-variable edits to the transcript (removing the auto-compaction, adding the function's code, replacing one sentence of the compaction summary, replacing the task description, and appending the review intent to the end of the resume message), and resampled repeatedly from the same decision point: 10 conditions and 36 valid runs in total, with each condition's prediction written down before it ran. Without history, K3 folded the check into the existing loop 5/5 times, whereas resumed unedited it did so only 1/5; removing the auto-compaction gave 3/5; adding the function's code (0/3), replacing the summary's problem-framing sentence (1/3), and replacing the task description (0/3) brought no improvement; appending one complete sentence of review intent gave 3/3, as did its first or second half given alone; by contrast, only reminding K3 to reread the review gave 0/3. The results indicate that the failure in the original session cannot be put down simply to a lack of capability or to missing information, and that one unambiguous statement of intent is enough to change K3's choice; the reasoning transcripts suggest that the failures arise mainly in how K3, carrying history, interprets and weighs the review.

1. Introduction

An assertion failure in LLVM's register coalescer (RegisterCoalescer) was reproduced, localized, and fixed by Kimi K3 over one long session, and the patch was submitted as PR #220979. The reviewer asked that, instead of adding a separate scan, the existing check be extended (quoted in Section 2.2). However, the three revisions K3 proposed in the original session all remained separate scans, and none was adopted; when I gave the same task to GPT-5.6 Sol in a new session, Sol produced, on its first attempt, a revision logically equivalent to the final merged version.

The most direct explanation is a capability gap between the two models. This post tests an alternative explanation: K3 is capable of applying the review, but the context of the original session prevented it from doing so; if this holds, the further question is which part of the context is responsible. The method follows Anthropic's alignment assessment of cybersecurity incidents (An alignment assessment of recent cybersecurity incidents): locate the key decision point in the incident transcript, implement each hypothesis as one concrete transcript edit, resample from the decision point and compare behavior rates, and do not rely on the model's explanations of its own behavior.

The division of work is as follows. K3 wrote the patch, and the merged revision came from Sol. I set the research question, the experimental conditions, and each condition's prediction, and I also introduced the three turns the experiment took. First, H-a rested on the premise that "K3 cannot see the pre-check loop when deciding"; when I went back through the original session, I found that before compaction K3 had not only read this loop but had also used it as the precedent for writing a separate scan, so the information-gap hypothesis does not hold (Section 4.3). Second, after the weak cues failed, I did not continue dissecting the context but switched directly to an explicit statement of intent, which brought K3 with history from 1/5 back to 3/3 (Section 4.5). Third, I questioned the keyword excerpts on which the analysis rested and found that they covered only about 6% of the text; after the full text was read, the conclusion that "10 of the 14 failures clearly found the correct approach" was corrected into three groups (Section 5.1). Claude helped write the run scripts, run the experiments, and organize the analysis; its lapses during the analysis are listed in Appendix B.

The main contributions of this post are:

• Separating the effects of capability and context in a real coding agent failure. Following the reference study, the original session is truncated at the decision point, single-variable transcript edits are applied, and the session is resampled (Section 3): the same K3 went 5/5 without history and 1/5 when resumed unedited with history; one explicit sentence of review intent was enough to restore folding (9/9 pooled), whereas only a reminder to rethink gave 0/3 (Section 4).
• A run-by-run reading of the failed runs' reasoning transcripts: the failures did not stem from misreading the code but fall into three groups, namely decision failures, no workable way found, and misread review intent; runs without history met the same obstacles but treated the first sentence of the review as a hard constraint (Section 5).
• Three testable harness design hypotheses, and a reusable eval case built from this decision point (Section 7).

2. Background

In brief, the first patch added a new loop to fix the bug, and the reviewer considered that extending a loop already in the function would cover the same case. The rest of this section gives technical details for compiler readers.

2.1 The bug

Since f44079d, x86 instruction selection can emit sub-register writes of the form %27.sub_8bit = MOV8rm .... The write carries no undef flag, so it implicitly reads the old value of the whole register in order to preserve the remaining bits.

To delete a COPY, RegisterCoalescer::removeCopyByCommutingDef commutes the preceding OR, rewrites the explicit operands, and then removes the old definition from the live interval. The partial def, however, is still implicitly reading that old value, so when coalescing is retried, JoinVals::analyzeValue asserts: Instruction is reading nonexistent value.

The function already contains a pre-check loop that runs before commuting (around line 922). It iterates use_nodbg_operands and gives up on commuting if it finds a use that reads the current value number AValNo and is tied to a def. A partial def is a def operand, so it lies outside the scope of that loop.

2.2 The first patch, the review, and the merged version

The first patch added a new loop dedicated to scanning defs:

for (const MachineOperand &DefMO : MRI->def_operands(IntA.reg())) {
  if (!DefMO.getSubReg() || DefMO.isUndef())
    continue;
  SlotIndex ReadIdx =
      LIS->getInstructionIndex(*DefMO.getParent()).getRegSlot();
  if (IntA.Query(ReadIdx).valueIn() == AValNo)
    return {false, false};
}

The reviewer, arsenm, left the following comment on this new loop:

I don't think you should need a new scan over defs. Most code should be checking readsReg, not getSubReg / isUndef directly.

MachineOperand::readsReg() is defined as !isUndef() && !isInternalRead() && (isUse() || getSubReg()), so a sub-register def without undef itself counts as a read. The merged version therefore no longer scans defs separately and instead widens the existing pre-check loop to all operands:

  for (MachineOperand &MO : MRI->use_nodbg_operands(IntA.reg())) {
  // Make sure all reads of AValNo can be rewritten to the new register.
  for (MachineOperand &MO : MRI->reg_nodbg_operands(IntA.reg())) {
    if (!MO.readsReg())
      continue;
    ...
    if (UseMI->isRegTiedToDefOperand(OpNo))
    // Partial defs and tied uses can't be rewritten independently.
    if (MO.isDef() || UseMI->isRegTiedToDefOperand(OpNo))
      return {false, false};

The two sentences of the review correspond to these two changes: the first asks that no scan be added, and the second asks that readsReg() replace the hand-written getSubReg() / isUndef() checks.

2.3 Configuration of the original session

The original session ran on Kimi Work's k3-agent, with a 256k context window. At about 210k tokens it triggered auto-compaction, which condensed several hundred messages into a handoff summary; the first patch was written after compaction. After K3 fetched the review with curl, it gave all three revisions in prose only, without changing the code.

3. Methods

Figure 1 summarizes the experimental design.

3.1 Decision point and resampling

The decision point is placed in the original session after K3 fetched the review with curl and before it began thinking, that is, where K3 first encountered the review and had not yet formed a plan. The original session is truncated at this point, edited according to each condition, and resumed with Kimi Code CLI 0.39.1, configured as follows:

• model: kimi-code/k3, thinking effort max, all thinking retained;
• default resume message: "Continue. (Note: use only the local repository, do not go online.)" (translated from Chinese);
• environment: an independent clone per run with the first patch already committed, no build directory, and a 30-minute limit.

The first request of a resume with history is about 150k tokens, and about 335k tokens for the no-compaction condition. As for sample sizes, fresh ran 5 times; baseline and no-compaction first ran 3 times each and were then increased to 5 under a sampling rule set in advance (if either group differed from baseline by 2 or more, both groups were topped up to 5); because of quota limits, every other condition ran 3 times.

The no-history control condition (fresh) receives only a frozen task description containing the issue, the root cause, the first patch, and the review text.

3.2 Conditions and pre-registered predictions

Each condition changes only one variable (Table 1). The conditions were designed in sequence: later conditions were decided after earlier results were in, rather than all being fixed before the experiment began. In this post, "pre-registered" means that each condition's prediction was written down before that condition ran and was not changed afterward.

Table 1. Experimental conditions, edits, and pre-registered predictions

Condition	History	Edit	N	Prediction
fresh	None	Task description only	5	—
baseline	Compacted	No edit	5	Reference
no-compaction	Uncompacted	Compaction event removed, full pre-compaction history kept	5	Folds more than baseline
task-prompt	Compacted	Resume message replaced with fresh's task description	3	None
H-a	Compacted	One tool call reading the whole function, with its result, inserted at the decision point	3	≥2/3 drop the separate scan
reframe	Compacted	The summary sentence that frames the bug replaced with a purely factual one	3	≥2/3 fold
clarify	Compacted	One complete sentence of review intent appended to the resume message	3	3/3 fold
negate	Compacted	Only the first half of the intent sentence appended	3	3/3 fold
rethink	Compacted	Only a reminder to reread the first sentence, with no answer	3	3/3 fold
direction	Compacted	Only the second half of the intent sentence appended	3	3/3 fold

The "History" column gives the original session record carried into the resume: "Compacted" is the handoff summary produced by auto-compaction together with the messages after it, and "Uncompacted" is all messages kept after the compaction event is removed. Throughout this post, "with history" means carrying the original session record, compacted or uncompacted, as opposed to fresh, which runs "without history"; it does not refer specifically to the uncompacted transcript.

The four appended sentences read as follows (translated from Chinese):

• clarify: "Additional note: what arsenm means is don't add a new separate scan; extend the existing check in the function instead."
• negate: "Additional note: what arsenm means is don't add a new separate scan."
• rethink: "Additional note: first think through what arsenm's first sentence actually wants us to change."
• direction: "Additional note: what arsenm means is extend the existing check in the function."

The direction group was added after the other conditions were complete; together with the negate group, it tests the two halves of the clarify sentence separately.

H-a inserts the full text of removeCopyByCommutingDef with the first patch applied, without singling out the pre-check loop, so that the edit does not act as a directional hint. reframe changes only one sentence of the compaction summary; see Section 4.4.

3.3 Measures

In this post, "fold" means dropping the separate scan and extending the existing pre-check loop instead. The default counting measure is folded total: the number of runs that changed the code to fold plus the number of runs that left the code unchanged but proposed folding in prose. The alternative measure is code only, which counts only the former. The text reports folded total by default and gives both only where the two measures lead to different conclusions; both columns appear in Table 3. The per-run results table is the sole source for every run's outcome; see Table 5 in Appendix A. Runs that hit the 30-minute limit are scored on the code at termination; 8 runs timed out in total, all marked in Appendix A.

Code quality is measured by blind grading. The grader is Claude Sonnet, which reads only the anonymized final code and scores it on a 10-point scale: structure 3, applying the review 3, correctness 3, comments 1. Each run is graded 3 times and the median is taken. Runs that did not change the code are graded on the first patch.

The hard anchor is to compile the final code into llc and run two MIR tests; a crash or FileCheck failure sets correctness to 0. The first patch also passes both tests, so they can only rule out crashes and cannot distinguish implementations.

The first patch already fixes the bug. This study measures adoption of the review, not functional correctness in general. Six of the 10 grading points come from structure and applying the review, so the blind grade is highly correlated with folding and cannot serve as independent evidence of quality.

Blind grades correspond almost one-to-one to implementations:

• folded: 9–10;
• kept a separate scan but switched to readsReg(), or replaced it with a different scan: 7–8;
• code unchanged: 3–4.

The only exception is r027 (H-a); see Section 4.3. Figure 3 shows the distribution of grades across runs.

To assess differences at small sample sizes, this post reports two-sided Fisher exact p-values for some comparisons. These comparisons were selected after the fact and are not corrected for multiple comparisons; they are descriptive only. The proportions in Figure 2 are also given with Wilson 95% confidence intervals.

3.4 Reading the reasoning transcripts

The reasoning transcripts were extracted and read by Claude. They cover each run's design phase, comprising thinking, replies, and tool calls, from the start of the resume to the first code edit; for runs that never edited code, reading continued to the last reply.

Of the 17 runs that did not fold, the design phase of 14 was read in full. The 3 rethink runs were added later, and only a keyword search over all of their thinking was performed; the results are in Section 4.5. Of the 5 fresh runs, r005, r008, r009, and r010 were read in full; for r011, only the decision passages, located by keyword, were read.

Thinking transcripts do not necessarily reflect the real causes of behavior, so this post does not ask the model about its reasoning, and the conclusions rest on behavior and code alone.

3.5 Validity of the edits and samples

Before the main experiment, 4 very short requests verified that transcript edits are feasible. When resuming, Kimi Code CLI rebuilds the context from the session transcript without integrity checks, so editing the transcript takes effect; a tool call and result inserted in the existing structure are treated by the model as a real tool result. A resume always appends a user message, so every condition uses the same resume message and differs only in the sentence appended at its end.

Network access was restricted only by the prompt and audited after every run; every run had zero network candidates. Three leaks occurred at the file-system level: r023 (no-compaction) read the final fix in my real local repository; r046 (negate) copied build tools from r038's run directory; and r052 (direction), while searching the whole disk for build tools, found the completed run directories that had been moved out of the run area and copied the cmake and ninja left by r046, without reading any source. All three leaks occurred after the code edits and do not affect the results.

To save quota, the negate, rethink, and direction conditions were changed to 3 serial runs sharing the same clone path, so that the history prefix hit the cache. Caching combined with near-deterministic decoding could make samples duplicate one another, so a rule was fixed in advance: if the first thinking block of the 2nd or 3rd run was identical word for word to that of the 1st run, the group would be stopped. The rule was not triggered.

3.6 Practices borrowed from the reference study

Table 2 lists the practices of the reference study and their counterparts in this post.

Table 2. Practices of the reference study and their counterparts in this post

Practice in the reference study	Counterpart in this post
When a weak cue fails, escalate to an unambiguous statement	Start with the strongest statement of intent (clarify), then weaken it step by step in the same direction
Where an edit is inserted is itself a variable; the closer, the more effective	Every appended sentence goes in the most recent turn, at a fixed decision point
One good behavior in the original trajectory may be luck; resample to obtain rates	Sol's single success is treated as motivation, not evidence; every condition is resampled
Evidence judged in isolation and in its original context may be judged very differently	Compare the fresh control against resumes with history
Do not rely on the model's self-explanation	Observed measures are limited to behavior and code; the model is not questioned

4. Results

Table 3 lists the counts for each condition and the outcome of each prediction.

Table 3. Folds per condition and prediction outcomes

Condition	N	Folded total	Of which in code	Prediction
fresh	5	5/5	4	—
baseline	5	1/5	0	Reference
no-compaction	5	3/5	3	Held
task-prompt	3	0/3	0	—
H-a	3	0/3	0	Depends on measure
reframe	3	1/3	1	Failed
clarify	3	3/3	2	Held
negate	3	3/3	2	Held
rethink	3	0/3	0	Failed
direction	3	3/3	3	Held

Sol ran only once, under the fresh condition (r007): it changed the code to fold and scored 10.

Figure 3 shows the blind grade of every run.

4.1 Without history, K3 can apply the review

All 5 fresh runs folded the check into the existing loop, 4 of them in code. r011 (fresh) read files without specifying an offset parameter, saw only the first 1000 lines, and in the end proposed the fold in prose. r008 (fresh) is logically equivalent to the merged version and scored 10; the other three runs that changed code scored 9 and applied readsReg() only to defs.

Against baseline, 5/5 versus 1/5 gives p ≈ 0.048. This comparison is confounded by a difference in model configuration: the original session's k3-agent has a 256k window, while the kimi-code/k3 used in the experiment has a 1M window.

4.2 Removing auto-compaction increased folding, but not to the no-history level

The main hypothesis was that phrases in the compaction summary such as "patch written, verified, and approved" made K3 overconfident in the first design.

The direction of the result matched the prediction: the no-compaction group's folded total was 3/5 against baseline's 1/5, and the difference is not significant (p ≈ 0.52). I had further expected no-compaction to approach fresh, or even to produce the merged version; this expectation was not met, as all three folds scored 9.

The task-prompt group changed code in all 3 runs, but none folded: two kept the scan and one replaced it with a different scan.

r023 of the no-compaction group entered my real local repository while looking for a build environment and read the final merged fix. Both of its code edits came before that, and it did not edit code afterward. The result is kept and flagged; without it, no-compaction is 2/4.

4.3 Adding the function's code at decision time brought no improvement

H-a tests the information-gap hypothesis: the pre-check loop appeared only before compaction, so K3 could not see it when deciding.

Two of the three H-a runs deleted the first patch's scan, but both replaced it with another separate scan: r026 (H-a) walked the segments of the live interval and scored 8; r027 (H-a) walked the value numbers, crashed in the behavioral test, had its correctness set to 0, and scored 3. r025 (H-a) left the code unchanged and kept the scan in its prose proposal.

Whether the prediction held depends on the measure: by the label used when the group ran, "deleted the first scan", it is 2/3 and the prediction held; by the folding measure it is 0/3.

However, searching the original session afterward showed that the premise of the hypothesis does not hold. Before compaction, K3 had already read the whole function, treated this loop in its thinking as the closest precedent for its own fix, and deliberately made the new def scan symmetric with it. It also considered readsReg(), but reached the wrong conclusion:

defs don't 'read' unless... a subreg def without undef IMPLIES reading the other lanes — there's no readsReg flag on it

The same holds when resuming: all 5 baseline runs read the pre-check loop on their own at the start of the resume and looked up the definition of readsReg(). Under resume conditions, H-a therefore injected information K3 would have read anyway. The two sentences of the review correspond exactly to K3's two judgments at the time: that a separate scan had to be written, and that readsReg() does not cover defs.

4.4 Replacing the summary's problem-framing sentence brought no improvement

All 4 failed baseline runs based their decision on the same reason: the use list cannot reflect this implicit read, which is the root cause of the bug, so defs must be scanned separately. For example, the reply drafted by r018 (baseline):

The existing pre-check scans use operands only; the hazard comes from a def operand (subreg non-undef implicit read), which is invisible in use lists — that's the root cause of the bug. So the def scan is needed, but I switched it to readsReg().

The compaction summary describes the bug with exactly the same framing (translated from Chinese):

but its use-rewrite loop only walks explicit operands (around lines 951-1000), and can't see this implicit read;

By contrast, no run quoted the summary's finality phrases such as "verified" or "approved". I therefore hypothesized that what mattered was the problem framing given by the summary, not the certainty it conveyed.

The edit replaced that sentence with "while that partial def is still implicitly reading the old value of %27 from before the commute (value 4);" (translated). The replacement keeps the bug mechanism, does not mention the pre-check loop, gives no hint of direction, and is of similar length.

The result was 1/3; the prediction (≥2/3) failed, and the result is close to baseline's 1/5. r036 (reframe), which did not change code, used the phrase "invisible to the use list" 47 times in its thinking, having derived it on its own from the existence of the bug.

4.5 Stating the review intent explicitly is enough to restore folding; a reminder to rethink is not

The clarify group appended only one sentence of review intent to the resume message, with no implementation details. The result was 3/3, and the prediction held. r038 (clarify) has the same shape as the merged version and scored 10.

The strength of the statement was then reduced step by step in the same direction:

• negate went 3/3, and the prediction held.
• rethink went 0/3, and the prediction (3/3) failed.
• direction went 3/3, and the prediction held. All three changed the code to fold and scored 9, applying readsReg() only to defs.

Given separately, each half of the clarify sentence produced 3/3. This result shows only that either half suffices on its own; it does not explain the mechanism. Pooling the clarify, negate, and direction groups, 9/9 against baseline's 1/5 gives p ≈ 0.005, and against rethink's 0/3, p ≈ 0.005.

None of the three rethink runs changed code. Two of them switched to other approaches in prose, one walking the value numbers and one checking the instruction at the end of the interval segment; the third kept the scan. All three reinterpreted the first sentence of the review as "the information is already in the interval or an existing data structure, so no scan is needed".

r048 (rethink) at one point wrote out the correct direction:

So "no new scan over defs" could mean: scan reg_operands (or extend the existing use-loops to reg loops) with MO.readsReg() && MO.getParent() != ....

Its subsequent reasoning, however, turned in another direction:

Hmm wait — actually! Maybe his point is architectural in a different direction

It did not evaluate that option again, and in the end recommended keeping the def scan, adding only an early hasSubRanges() return in front of it.

5. Analysis of reasoning transcripts

This section is an observational analysis of the same data, used to explain the results of Section 4; it does not constitute independent evidence.

5.1 Failures fall into three groups

The criterion is whether a run thought of changing the pre-check loop to walk all operands and distinguishing defs from uses inside the loop. By this criterion, the 14 failed runs read in full fall into three groups (Table 4):

Table 4. Classification of the 14 failed runs read in full

Group	Runs	Typical behavior
Decision failure: saw the correct structure but did not use it	5	Dropped it for reasons such as clarity first, too large a change, "I don't think" being mere hedging, leaving the choice to the reviewer, or deferring to the next round; 2 of them wrote out the complete code
No workable way found	6	Assumed folding meant applying one condition to every operand, saw that this would block ordinary uses, and fell back to scanning defs only
Misread the review's intent	3	Read the first sentence as "don't scan defs; use the information in the live interval instead"

A sentence-by-sentence check shows that the failed runs described the pre-check loop and readsReg() itself correctly; the failures did not stem from misreading the code. The three groups differ in nature. The first is a decision failure: the correct structure was written out but abandoned because of review strategy or an earlier conclusion. Most runs in the second group had understood that the review required folding into the existing loop; the obstacle lay in the implementation, and what was missing was the step of branching on def versus use inside the loop (see Section 5.2). Only an individual run in this group misunderstood the intent, for example r025 (H-a), which held that walking all operands covers a larger set and so could not be what the reviewer meant: "That's a superset — but my def_operands scan is smaller. So "no new scan" can't mean that." The third group misread the review's intent, and the 3 rethink runs also belong to it (see Section 4.5). Decision failures do exist, but they account for only 5/14 of the runs read in full.

5.2 Runs without history met the same obstacles but treated the first sentence as a hard constraint

All 5 fresh runs encountered the obstacle that "the loop at line 922 only walks uses". r009 (fresh) also ran into the problem that "applying readsReg() to every operand would block ordinary uses". r005 (fresh) likewise considered review strategy:

The "I don't think you should need a new scan over defs" is a "I don't think" — a suggestion/question.

The difference lies in what followed: all 5 treated the first sentence as a constraint that had to be met and explicitly rejected the option of keeping the scan and only switching to readsReg(). For example, r008 (fresh) wrote:

But that still has a "new scan over defs", which the reviewer questions.

They resolved it by redefining what the existing check intercepts. r008 (fresh):

So the existing check is a specific case of "a read of AValNo that cannot be rewritten to NewReg". A partial def reading AValNo also cannot be rewritten.

What the 6 runs that found no workable way were missing is exactly this step of branching on def versus use inside the loop.

5.3 Failed runs with history kept their earlier conclusion or cited review strategy

The first kind of reason is adherence to an earlier conclusion. "It is invisible, so a scan is required" is a conclusion K3 reached on its own before compaction, which the summary carried into the resume; the raw history of the no-compaction group contains the same reasoning. The input of fresh contained only the patch and the review, so "invisible" was merely a fact noticed while reading the code. r036 (reframe) went further and inferred that folding is equivalent to scanning only defs:

Scanning reg_operands with (MO.isDef() && MO.readsReg()) == scanning def_operands with readsReg (same set, since def_operands = reg_operands filtered by isDef). So either formulation is equivalent; def_operands is cleaner.

The second kind of reason is review strategy. r024 (no-compaction) wrote out the complete folded code and then wrote:

This is arguably the "no new scan" ideal. But it rewrites arsenm's existing loop — bigger diff, higher review friction. My recommendation: minimal variant (readsReg filter, keep separate loop), mention in reply that the check complements the tied-use guard right below and can be merged into one reg_operands walk if he prefers. Give him the choice in the reply — reviewers love that.

r034 (reframe) likewise wrote correct code and ultimately chose to defer the change:

If he then says "fold it into the existing loop" or "use interval valnos", we do that next round. That's a normal review dialogue — you don't have to preempt everything.

No reason of this kind appears in the 4 fresh design phases read in full or in r011's decision passages.

6. Discussion

6.1 What the edit experiments support

By strength of evidence, the results of Section 4 support the following judgments:

• Stronger: the failure cannot be put down simply to a lack of capability. K3 without history went 5/5, against 1/5 for baseline (p ≈ 0.048).
• Stronger: one explicit statement of intent is enough to make K3 with history fold. The complete intent sentence and each of its halves given alone all went 3/3, 9/9 in total; rethink went 0/3.
• Weaker: auto-compaction may have played a part. Removing compaction gave 3/5, not significantly different from baseline (p ≈ 0.52).
• Negative: neither an information gap nor the summary's framing sentence is the main cause. Adding the function's code (0/3) and replacing the framing sentence (1/3) brought no improvement, and K3 reads that code on its own when resumed.

The second point is consistent in direction with the reference study's finding: when a weak cue fails, an unambiguous statement is needed to change the behavior.

The two failed predictions are also informative. That reframe had no effect shows that the sentence is not the only source of the framing: K3 derives the same framing on its own. That rethink had no effect shows that the problem is not whether the review is re-examined, but the interpretation reached upon re-examination.

6.2 An untested explanation

Taking Section 4 and Section 5 together: faced with an ambiguous review, K3 with history tends to resolve the ambiguity toward "keep the current design and reply with an explanation", whereas K3 without history treats the same review as a hard constraint.

I initially took this to be a capability problem, or a sign that K3 handles uncertain tasks poorly. The fresh group, however, faced the same ambiguous review and went 5/5, which shows that K3 can produce the correct revision under at least some conditions, so the failure cannot be put down simply to a lack of capability. Fresh runs and resumes with history nonetheless differ in context length, information organization, and task description, and these differences have not been fully separated. In light of the fresh runs' reasoning, my current explanation is that the behavior more likely stems from preferences, pattern matching, and strategies learned in training, which in a long history manifest as adherence to the model's own earlier conclusion or as the application of a "minimal change" review strategy.

6.3 About Sol

My agreement with Sol's version stems from a prior that I had not shared with any model at the time. Earlier, in #202599, I had attempted to fix a related RegisterCoalescer problem; that attempt was ultimately superseded by bababuck's #204039, which changed only a few lines of the existing logic, and #202599 was eventually merged as regression tests only. I studied the approach of #204039 closely at the time, and it left me with a prior: keep the existing structure and extend an existing check.

Sol succeeded only once, in a new session. That success may stem from LLVM fix patterns learned in training, or it may be due to chance. This post has no Sol-with-history condition and therefore cannot compare the two models in the same context.

7. Implications for coding agent design

The following are design hypotheses derived from the results above; none has been tested. Each can be tested with this post's method: apply the corresponding change at the same decision point, resample, and compare fold rates.

7.1 Restate the review as constraints when it arrives

One explicit statement of intent raised K3 with history from 1/5 to 3/3 without any implementation details. This suggests that what was missing may not have been a plan but confirmation of which requirements had to be met. A harness could insert a step after the agent reads a review, requiring it to list what each comment asks for and which of those requirements are hard constraints before it begins designing.

The effect of this step depends on whether the agent's list is correct. The rethink group is a counterexample: asked to reread the first sentence on its own, K3 interpreted it as "the information is already in the interval", and the result was 0/3. A test should therefore record two things separately: whether the self-restatement is correct, and whether the fold rate approaches 3/3 when it is.

7.2 Hand revisions to a sub-agent without history

K3 without history went 5/5. When revising for a review, the patch, the review, and the repository could be handed to a sub-agent without session history to propose the revision, with the main agent responsible for integration.

A foreseeable point of failure is the main agent rejecting the sub-agent's proposal on the basis of its own earlier conclusion, just as r024 (no-compaction) and r034 (reframe) rejected correct code they had written themselves. A test should record both the sub-agent's proposal and the version the main agent finally adopts.

7.3 Improving the compaction summary may not be the main lever

No run quoted the summary's phrases such as "verified" or "approved", and the summary preamble built into the harness already reminds the model to "Treat it as notes, not proof". Replacing the summary's problem-framing sentence was likewise ineffective, because K3 derives the same framing on its own.

One direct improvement is to keep rejected alternatives and design reasoning in the summary. The original session's summary kept only the patch code and mentioned neither readsReg() nor the pre-check loop. The no-compaction context, however, kept all of the reasoning, including the misjudgment about readsReg(), and still folded only 3/5; keeping reasoning can also carry errors forward. On current data, the point at which new feedback arrives (7.1 and 7.2) is a more promising place for improvement than the summary itself.

7.4 Turn the decision point into an eval case

The decision point in this post is itself an eval case derived from a real trace: fix the history prefix and the resume message, then measure the fold rate. It can serve as a regression test for changes to the compaction strategy, the system prompt, or the review-handling flow, and 7.1 through 7.3 can all be tested with it. The three failure groups also correspond to signals that can be checked automatically: whether the correct structure was written out but not adopted, whether the loop branches on def versus use, and whether the first sentence was reread as "use the interval information instead".

A single case has limited coverage. Extending it into an eval set requires identifying similar decision points in more sessions, namely the moments at which external feedback conflicts with a conclusion the agent reached earlier.

8. Limitations

The strength of evidence in this study is limited by sample size and by the order of the design. Each condition ran only 3–5 times; the p-values in the text are descriptive only and are not corrected for multiple comparisons. The conditions were designed in sequence, with later conditions informed by earlier results, although each condition's prediction was written down before it ran. This limitation mainly affects the weaker judgment in Section 6.1: at the current sample size, the difference between no-compaction's 3/5 and baseline's 1/5 cannot be distinguished from sampling variation, whereas the gaps between fresh's 5/5 and baseline's 1/5, and between the pooled 9/9 of the intent-statement groups and rethink's 0/3, are large and are relatively less affected by sample size.

Three points about the measures require comment. First, both the primary metric and the scoring rule were fixed after part of the results had been seen: the primary metric was changed from "byte-identical to the merged version" to blind grading only after r005 (fresh) and r007 (Sol) had been seen, and the scoring rule was fixed after the first round, r012–r014, had been seen. Second, this study measures adoption of the review, and the first patch already fixes the bug, so the blind grade is highly correlated with folding and mainly reflects whether the review was adopted rather than code quality in an independent sense. Third, the tests on which grading relies have limited discriminating power: the two MIR tests cannot distinguish implementations, and the grader was lenient about old comments that were not updated.

The experimental conditions differ from the original session in several respects. The original session's k3-agent has a 256k window, while the experiment's kimi-code/k3 has a 1M window; the no-compaction condition is about 335k tokens, beyond the original session's window, and exists only in the CLI. Resumes reuse the system prompt from the original transcript, but the tool set differs slightly from that of the original session. Every resume is a single response without the multi-turn feedback from later in the original session, and the prefix contains all of K3's earlier steps. The step inserted by H-a has no thinking before it; this form occurs in the original session but is not typical. Apart from the insertion form of H-a, these differences are the same across conditions, so they mainly limit how far the conclusions generalize to the original session rather than the comparisons between conditions.

Isolation was likewise incomplete. Network access was restricted only by the prompt. r023 (no-compaction) entered my real local repository; its handling is described in Section 4.2. Completed run directories had been moved out of the run area but remained on the same disk, and r052 (direction) found them with a whole-disk search. In addition, the isolated clones carry artificial traces: the dates are a month apart, and the branch name and commit author differ. These traces are the same across conditions but may affect behavior.

Finally, Sol in the original conversation (the instance described in the introduction, which motivated this study) is not fully equivalent to the fresh condition: its input included my remark that "K3's changes still aren't right", which the fresh input did not contain. The Sol run in the experiment, r007, by contrast used the same frozen task description as K3's fresh runs, so its input was identical.

9. Future work

Future work first addresses those limitations that affect the interpretation of the conclusions. Section 6.3 could not compare the two models in the same context; adding a Sol-with-history condition would test whether Sol's original success depended on fix patterns learned in training. On isolation, K3 with history was able to infer the location of my real local repository, so later experiments need a separate system account or a virtual machine. On the measures, the existing tests cannot distinguish the incomplete adoption seen in fresh runs, in which readsReg() is applied only to defs; a test covering an undef, tied use needs to be added.

The second priority is to test the design hypotheses in Section 7, beginning with the self-restatement in 7.1: it requires no human to add a sentence, and if it proves effective it can be incorporated directly into a harness. The remaining hypotheses can be tested with the same method at the same decision point. To this end, I plan to clean up and publish the tools for resuming, transcript editing, and grading, so that decision points at which external feedback conflicts with an agent's earlier conclusion can be identified in more sessions and the single case can be extended into an eval set (see Section 7.4).

Appendix A: Per-run results

Table 5. Outcome and blind grade of each run

Run	Condition	Outcome	Blind grade
r005	fresh	Changed code, folded	9
r008	fresh	Changed code, folded	10
r009	fresh	Changed code, folded	9
r010	fresh	Changed code, folded	9
r011	fresh	No code change, folded in prose	4
r012	baseline	No code change, kept scan in prose	4
r015	baseline	No code change, kept scan in prose	4
r018	baseline	Changed code, kept scan	7
r021	baseline	No code change, kept scan in prose	4
r022	baseline	No code change, folded in prose	4
r013	no-compaction	Changed code, folded (timed out)	9
r019	no-compaction	Changed code, folded	9
r023	no-compaction	Changed code, folded (isolation leak afterward)	9
r016	no-compaction	No code change, kept scan in prose	4
r024	no-compaction	No code change, kept scan in prose	3
r014	task-prompt	Changed code, kept scan (timed out)	7
r017	task-prompt	Changed code, replaced with another scan (timed out)	7
r020	task-prompt	Changed code, kept scan	7
r025	H-a	No code change, kept scan in prose	4
r026	H-a	Changed code, replaced with another scan	8
r027	H-a	Changed code, replaced with another scan (test crashed)	3
r034	reframe	Changed code, kept scan (timed out)	7
r035	reframe	Changed code, folded	9
r036	reframe	No code change, kept scan in prose	3
r037	clarify	Changed code, folded (timed out)	9
r038	clarify	Changed code, folded (timed out)	10
r039	clarify	No code change, folded in prose	4
r044	negate	No code change, folded in prose	4
r045	negate	Changed code, folded	9
r046	negate	Changed code, folded (timed out)	9
r047	rethink	No code change, another approach in prose	4
r048	rethink	No code change, kept scan in prose	4
r049	rethink	No code change, another approach in prose	3
r050	direction	Changed code, folded	9
r051	direction	Changed code, folded	9
r052	direction	Changed code, folded (timed out)	9
r007	Sol (fresh)	Changed code, folded	10

Run numbers not counted:

• r001–r004, r006: infrastructure failures;
• r028–r033: aborted at my request;
• r040–r042: quota errors;
• r043: only a shared clone path name, not corresponding to any run.

Appendix B: Corrections during analysis

The following three lapses at one point changed the explanation of the failure.

• The original session was not searched before H-a was designed. Claude's analysis only confirmed that the code of the pre-check loop appeared before compaction, and did not find that K3 had at the time treated it as a precedent and written a separate scan modeled on it. Without knowing this, I approved the design and set the prediction. The procedure has since been changed: before designing any insertion or deletion edit, everything relevant in the original session is searched first.
• Counting measure. Claude once reported that baseline "folded 0/5". After I asked about the prose proposal of r022 (baseline), this was corrected to 0/5 code only and 1/5 folded total. The early label "dropped the separate scan" also counted "replaced with another scan". Since then, the per-run results table has been the single source, and every count states its measure.
• Failure classification. Based only on keyword snippets covering about 6% of the text, Claude first concluded that "10 of the 14 failures clearly found the correct approach"; after the full text was read, this was corrected to 5 / 6 / 3.

Appendix C: Experimental infrastructure

Resuming. The command is kimi -S <session_id> -p ... --output-format stream-json. A resume can be truncated after a given tool result, but it appends a new user message and re-injects a permission notice, so it does not seamlessly continue the same turn. The original session was created in Kimi Work, and its model alias k3-agent is unavailable in the CLI; resuming became possible only after the alias was changed to kimi-code/k3. Resumes reuse the system prompt from the transcript, with a hash matching the original session's; the 13 core tools are the same, with 7 additional Kimi Work tools and 2 missing.

Runs. Each run first creates an independent clone, applies and commits the first patch, and verifies the file hash, and is then launched as an independent background process. The launcher keeps its own time, kills the whole process tree on timeout, and then collects the diff, the session transcript, token usage, and the network audit. Normal completions and timeouts both count toward the sample; CLI errors (such as exhausted quota) do not, and the same condition is rerun.

Isolation. Codex's sandbox on Windows does not block the network, and the K3 side has no network restriction either, so only after-the-fact auditing was possible. The leaks of r023, r046, and r052 show that for an agent with history, paths and user names are themselves clues, and isolation by directory is insufficient. After r046, completed run directories were moved out of the run area, but they remained on the same disk, and r052 found them with a whole-disk search. Later experiments should move completed runs to another disk or archive them; a more robust approach is a separate system account or a virtual machine.

Grading. For blind grading, runs from the same batch are mixed and randomly numbered. The grader is a Claude Sonnet sub-agent that cannot see the condition and is not the model under test. A review rule also applies: a total of ≥ 8 with a failed test, or ≤ 4 with a passed test and the separate scan dropped, is referred for manual review.

Experimental data as of October 10, 2026.