Review what the latest change can break

The newest lines are only the starting point

A team adds a cursor to a library catalog so visitors can page through newly acquired books. In the first revision, the service sorts results by acquisition date and uses the book identifier to break ties. Its cursor stores both values. A reviewer checks that the cursor and sort agree. In the next revision, the author switches the tie breaker to title so books acquired on the same day appear alphabetically. The cursor builder stays untouched. A narrow review can see a reasonable change to the sort and miss that the original approval depended on something that is no longer true.

Cognition announced Devin Review 2.0 on November 5. Its release notes say repeated runs on a pull request review the parts that changed by default, and that teams can configure automatic runs and a per-pull-request spend limit. Cognition reports a typical review cost of $0.25 to $0.75; that is the company’s figure, not a quote for a particular repository. The release makes an old review problem more visible: a small set of new lines can invalidate a much larger set of conclusions.

The catalog example is fictional. It is a way to plan review coverage, not a report on how Devin performed or a claim about what context it uses internally. The goal of an incremental pass is to make the next decision easier. That requires starting at the last reviewed revision, tracing the behavior affected by the new change, and knowing when to ask for a wider review.

Source: Cognition: Devin Review 2.0 ↗ · GitHub Docs: Viewing a pull request review ↗ · GitHub Docs: About protected branches ↗

Record what the first approval depended on

Give the first review a useful reference point. Record the revision being reviewed and a short statement of the behavior the reviewer checked. In the catalog, that statement might be: “Page boundaries use acquisition date and book identifier in the same order as the query, and a visitor can continue from the last result.” This records a relationship. “Reviewed catalog files” records only where someone looked.

For an update, distinguish three versions: the target branch the pull request is headed toward, the last reviewed revision, and the current revision. Comparing the last two shows what the author changed since the review. Inspecting the current proposal against its target shows what would land. Both comparisons can matter after a rebase or a related change to the target branch. A reviewer who sees no new edits from the author may still need to check whether a dependency changed elsewhere.

GitHub’s documentation makes the distinction concrete. Its review history can show the version a reviewer saw, and protected branches can be configured to dismiss stale approvals when the reviewed diff changes. Those features help tie an approval to a state. They do not tell you which catalog behavior the person actually verified. A short review note is useful precisely because it supplies the missing explanation.

Keep that note narrow enough to maintain. One or two sentences about the key behavior and the dependent parts are usually more useful than a transcript of the review. When the work is genuinely trivial, record that the change does not affect behavior and why. When a later edit overturns the reason, the note gives the next reviewer a place to begin.

Follow the dependency across the changed line

On the second pass, start with the diff and name the assumptions each material change touches. A change to ordering can affect cursor encoding, cursor decoding, the database predicate for the next page, and tests that assert stable order. A changed file list will not necessarily point to all of them. Searching for the old ordering keys or tracing the code that reads the cursor can reveal relevant code even if no line there was edited.

Work through a concrete boundary. Suppose three books share an acquisition date. Their identifiers put them in one order and their titles in another. If the first page is sorted by title but the next-page filter still compares identifiers, the filter may select a different set than the ordering promises. A record could be skipped or repeated, depending on how the orderings differ. A test whose fixtures give every book a distinct date cannot reveal this mismatch. That is a test case to propose, not a result to claim.

There is a second question if old cursors can survive a deployment. A saved link produced under the previous ordering may arrive at the new code. A team can decide to preserve compatibility, explicitly expire old cursors, or avoid changing the ordering in this revision. Whatever the decision, review the path that enforces it. Merely updating the sort and matching current-page tests does not address a token produced before the change.

Use this pattern on other changes with the same structure. A new default can affect an unchanged caller that omitted a setting. An updated unit can affect an unchanged chart that labels a value. A refactor can move a calculation across a point where it previously happened once. In each case, the useful search question is “Who relies on the old behavior?” rather than “Where did the author touch the code?”

Also ask whether the new tie breaker is sufficient. Two different books can have the same title, so a predictable order may still need the identifier as a final key. If that is the policy, the comparison has three parts: date, title, then identifier. Review how those parts travel together from the query to the cursor and back. Check the page boundary in both directions if the product supports previous as well as next. That work is more useful than asking for a generic second review of every file: the reviewer can name the exact agreement that was broken and the evidence that would show it has been restored.

Set the boundary for a wider pass

Define when a quick follow-up is enough. A wording change in a review comment or a rename confined to one local scope might need little additional investigation. A behavior change to ordering, defaults, persisted values, or a shared interface calls for a dependency pass even when the patch is short. The signal is a changed assumption, not a line count.

Choose a stopping boundary that someone can explain. For the catalog, the reviewer might follow the changed ordering through query, cursor, and client behavior when a cursor is refused. That is wider than the changed lines but narrower than reopening the whole library application. Stop when the relevant relationship has been checked and the consequences of the change are understood. If reviewers cannot find the boundary, that uncertainty is itself a reason to ask for help from an owner who knows the system.

Ask automated reviewers for observations tied to the revision. A useful finding identifies the new change, the still-relevant consumer, and an input that would expose the mismatch. An unhelpful finding says that pagination might break without explaining where. The first gives someone work they can verify. The second can become a long thread with no clear decision. Treat an automated finding as a lead until a person can inspect the code or otherwise establish the consequence.

If a wider pass is needed, choose it explicitly rather than hoping several narrow runs will add up to one. That may mean a manual review of the current proposal, a focused review from the component owner, or another configured analysis if your tooling supports it. Record the scope of the pass and the outcome. The aim is to close the dependency question, not to accumulate more review activity.

Put the spend limit beside a coverage rule

A cap answers how much the automated reviewer may spend. It does not answer what happens when the cap is reached before a material change is checked. Decide that before enabling repeated runs. If a later revision has no automated result, the status shown to the team should distinguish “not reviewed” from “reviewed with no findings.” Budget exhaustion is an operational state, not evidence about the code.

Start by choosing when review can be useful. In a team that pushes several unfinished commits before asking for feedback, reviewing every draft push may spend money on work that will not survive to the next revision. A pass when a proposal first becomes ready can establish a baseline. Later passes can be limited to revisions that change behavior or follow substantive feedback. Teams that genuinely need rapid feedback on every push can choose that deliberately and watch how much review it creates.

Budget at two levels. A per-pull-request cap bounds one thread of revisions. A repository or team allowance helps reveal whether many individually modest reviews add up to more than the team intended. Use actual pilot bills and review records to set internal thresholds rather than extrapolating the vendor’s typical range. Include time spent assessing findings in the decision. A cheap run that sends a maintainer through a long string of irrelevant comments still uses scarce attention.

Reserve a path for consequential changes when the routine allowance is gone. For the catalog that might mean paging in an engineer who maintains the cursor contract. In other repositories, the answer could be delaying merge until a relevant owner has time. Name the person or role that can make that call. Avoid configuring retries that repeatedly attempt the same exhausted check. A clear handoff protects both the budget and the decision the team is about to make.

Judge the pilot by decisions it improves

Choose a small set of already resolved pull requests and include ones with several revisions. Use examples where a later change affected an unchanged dependency, cases where an incremental pass truly was enough, and at least one case where the target branch changed. Have reviewers note the outcomes they expect a useful review to surface before comparing the tool’s output. This makes it harder to mistake an impressive-sounding comment for coverage of the risk you meant to study.

Track which findings led to a verified correction, which were duplicates, and which did not hold up. Also record useful absences: a routine revision that produced no distracting commentary can be a good result if the intended checks actually ran. Separately record expected problems that were not surfaced. A raw count of comments cannot tell you whether the important cross-revision failures are being caught.

Include an exercise where the budget runs out or a pass is deliberately skipped. The point is to see whether a reviewer can tell which revision was examined and recover the remaining work. If a teammate has to reconstruct the answer from billing screens and chat messages, make the review status more explicit. A useful audit trail can be brief: revision, scope, outcome, and next owner for an open issue.

Close the loop by checking the current revision before merging. In the fictional catalog, a good closing note might say that ordering and cursor now use the same keys, that tied acquisition dates were included in the test plan, and that existing cursors have an explicit compatibility or expiry policy. Report any tests actually run separately from the checks still proposed. Then choose which follow-up reviews should be automatic based on what they helped people decide. The aim is to preserve confidence across revisions while keeping both automated spend and human attention finite.