When CodeRabbit Catches What Agents Miss

The calibrator rewrite looked clean. Three agents had reviewed the diff. The unit tests passed. CI was green. The math reviewer agent had walked through the rubric and signed off. The PR was one click from merge when CodeRabbit left a comment that stopped the session cold.

The comment was about a single line. It pointed at a boundary condition in a piecewise scoring branch and asked whether the score on either side of that boundary was actually continuous. In the illustrative values used below, the score jumps upward from 17.6 to 18.4. That is a 0.8-point discontinuity: the function remains increasing at that seam, but it is not continuous. The historical review account was not independently verified for this editorial pass.

This post is about why agents miss math bugs at branch boundaries, why static reviewers catch them, and why the right answer in the agent era is not "use more agents" but "keep the reviewers and take their comments seriously."

The Specific Bug

The calibrator was described as a piecewise linear function of a probability-ratio input. The simplified function below is illustrative rather than a reproduction of live strategy code:

def calibrate(ratio: float) -> float:
    if ratio < 0.15:
        return ratio * 120.0            # segment A: steep
    elif ratio < 0.40:
        return 18.0 + (ratio - 0.15) * 40.0   # segment B: medium
    else:
        return 28.0 + (ratio - 0.40) * 12.0   # segment C: shallow

At the first boundary, the displayed function evaluates to the same value from either segment and is continuous. To illustrate the reported defect without reproducing live coefficients, suppose a prior branch approaches 17.6 while the next begins at 18.4. Crossing the boundary then creates an upward jump of 0.8: monotone, but discontinuous.

In the illustrative example, inputs immediately below and above the boundary produce 17.6 and 18.4. The function is still monotone across the boundary, but nearby inputs receive materially different outputs. That is a discontinuity, not a rounding error.

Why The Agents Missed It

Three separate agents reviewed that file. Two of them were general code reviewers. One was a math reviewer specifically tasked with verifying the scoring rubric. None of them flagged the discontinuity. When I went back and audited the review trace, the pattern was consistent across all three.

Agents reading a piecewise function tend to verify the branches individually. They check that each branch does what its comment says it does. They check that the arithmetic within each branch is correct. What they do not reliably do is evaluate the function at the exact boundary input and compare the outputs on either side. The boundary case is a property that exists between branches, not within one. Agents trained on line-by-line review are very good at line-by-line bugs. They are substantially worse at the bug that lives in the seam.

The unit tests had the same blind spot. The test suite had cases for ratio = 0.10 (segment A), ratio = 0.25 (segment B), and ratio = 0.60 (segment C). It had no case for ratio = 0.15 exactly, or ratio = 0.14999, or ratio = 0.15001. The branches were tested. The seams were not. This is a remarkably common test-design failure and it is why piecewise functions eat scoring systems for lunch.

Why CodeRabbit Caught It

CodeRabbit's comment did not come from running the code. It came from reading the structure of the function and asking a question that agents rarely ask: does this function preserve continuity at the branch boundary? That question is the right one for any piecewise numeric function, and it is the question that makes discontinuity bugs fall out immediately. Once you are asking "does segment A equal segment B at the boundary value?" the arithmetic is trivial. You evaluate both expressions at the boundary and compare. If they differ, you have found the bug.

This is not a question that gets asked by accident. It is a question that gets asked because the reviewer has an explicit mental model of the failure mode and is scanning the code for places where the failure mode could hide. CodeRabbit is tuned for exactly those kinds of structural questions on math-heavy code. Agents focused on implementation correctness are not. Both kinds of review matter, and they catch different categories of bugs. Removing either one leaves a hole.

The Test That Should Have Existed

The reported fix was a correction to the adjoining segment. The process lesson is more important: every piecewise function should have an explicit continuity test. An illustrative test pattern looks like this:

def test_calibrate_continuity_at_boundaries():
    boundaries = [0.15, 0.40]
    epsilon = 1e-9
    for b in boundaries:
        left = calibrate(b - epsilon)
        right = calibrate(b + epsilon)
        assert abs(left - right) < 1e-6, (
            f"Discontinuity at ratio={b}: "
            f"left={left}, right={right}, delta={abs(left - right)}"
        )

Four lines of test logic. A loop over every boundary value in the function. Evaluation just below and just above each boundary. Assertion that the outputs agree to within numerical precision. This test would have caught the 0.8-point discontinuity on the first run. It would have failed the PR. It would have prevented the bug from ever reaching CodeRabbit, because the test would have reached it first.

The discipline here generalizes. For any piecewise numeric function — in a scoring engine, a classifier, a price model, a signal rubric in the InDecision Framework — write a continuity test at every branch boundary. The test takes three minutes to write. It catches a category of bug that unit tests normally miss. And unlike most tests, it does not need to be updated when the function changes: the boundaries are extracted from the function's own structure, and the assertion is a universal property, not a hand-coded expected value.

The Broader Point

The lesson I keep coming back to from this session is not "CodeRabbit is better than agents" or "agents are unreliable." Both of those framings miss the point. The lesson is that review layers catch different bugs, and stripping out the review layer you think is redundant is the fastest way to ship the bug you were not looking for.

Agents are excellent at implementation correctness. They are worse at structural properties that live between lines of code. Static reviewers like CodeRabbit are excellent at structural properties and less reliable at nuanced implementation semantics. Human reviewers catch the bugs that neither category can see because they hold product intent in their heads. You need all three. The cost of running all three is far lower than the cost of a discontinuity bug in a load-bearing scoring function that silently produces the wrong rankings for weeks.

The agent era has made it tempting to strip review layers on the theory that the agent's own correctness check is sufficient. It is not sufficient. It is powerful, it is fast, and it catches most bugs — but the bugs it misses are the structural ones that hide in seams. Those bugs need a different kind of reviewer. Keep CodeRabbit in the loop. Read its comments carefully, especially the ones that sound like questions. The next time CodeRabbit asks whether a boundary is continuous, the answer matters more than the friction of checking it.

The transferable recommendation is to add continuity tests before relying on a piecewise scoring function. The historical deployment and review outcome in this account still requires repository evidence; the boundary-testing method does not.