Methodology
Why Two Audits of the Same Article Can Score Differently
Revylo runs five independent AI models on every article. Here's why scores shift between runs — and what to focus on instead.
By Brian Diamond
Revylo runs five independent AI models on every article. Here's why scores shift between runs — and what to focus on instead.
You audited the same article twice. The overall score moved from 78 to 81. Or from 81 to 76. You didn't change the article. So what happened?
This is normal, and understanding why will save you from chasing phantom improvements.
The short answer
Revylo doesn't run one AI and call it a day. Every audit sends your article through five independent models — the Five Oracles — each with a different architecture, training data, and tolerance for ambiguity. They don't always agree. When they disagree, your score moves.
That's not a bug. That's the product working as designed.
Why five models instead of one?
A single AI model has blind spots. Ask one model whether an article demonstrates first-hand experience and it'll give you an answer — but it might miss what another model catches, or be too generous on a claim that's technically unsupported.
The Five Oracles exist because no single model is reliable enough to grade content quality on its own. By running five and aggregating their votes, Revylo gets closer to how a panel of human editors would actually read your article — some strict, some lenient, some focused on facts, some on voice.
The tradeoff: more reliable signal, more run-to-run variance on edge cases.
What actually varies between runs
Not everything moves. Here's what does and what doesn't.
Overall score (±3–8 points is normal)
Your overall score is a weighted average across eight checks. When one oracle shifts a single check by 5–10 points — which happens regularly on borderline content — the overall moves with it.
An article scoring 78 on one run and 81 on the next, with no edits, is within expected variance. An article scoring 78 then 62 is worth investigating — something structural likely changed in how the oracles read it, or one oracle failed and was retried with different results.
Per-check scores (this is where the real signal lives)
Individual checks are where variance shows up most clearly:
| Check | Typical variance | Why |
|---|---|---|
| Fact Grounding | High | Oracles disagree on whether a claim is "supported" vs "unsupported" when evidence is indirect |
| E-E-A-T | Medium–High | Experience markers are subjective — one oracle sees first-hand testing, another sees generic advice |
| Helpful Content | Medium | "Does this fully answer the query?" has no single correct answer |
| Originality | Low–Medium | Embedding similarity is more deterministic, but threshold interpretation varies |
| Search Intent | Low | Usually stable unless the article sits on a query boundary |
| Brand Voice | Medium | Voice consistency is inherently subjective |
| Internal Linking | Low | Structural analysis, less model-dependent |
| Technical SEO | Low | Mostly rule-based |
The pattern matters more than any single number. If Fact Grounding is red on both runs, that's a real signal. If it's amber on run one and green on run two, the underlying issue is probably still there — one oracle was lenient.
Oracle participation (sometimes an oracle doesn't show up)
Occasionally one of the five oracles fails to respond — a timeout, a rate limit, a parsing error. When that happens, Revylo retries once. If the retry also fails, that oracle is marked as "did not consult" and the remaining four oracles carry the score.
This can shift results noticeably, especially on checks where the missing oracle would have been the tiebreaker. Revylo offers a discounted re-audit when this happens — you shouldn't pay full price for an incomplete panel.
A real example: ChiliStation
We audited the same ChiliStation article three times over two weeks. Here's what actually happened:
| Run | Overall | Fact Grounding | E-E-A-T | Helpful Content |
|---|---|---|---|---|
| Run 1 | 78 | Red (62) | Amber (71) | Green (84) |
| Run 2 | 81 | Amber (68) | Green (76) | Green (86) |
| Run 3 | 79 | Amber (65) | Amber (73) | Green (85) |
Nothing changed on the page between runs. Fact Grounding bounced between red and amber because different oracles weighted the sourcing differently — some counted the manufacturer's spec sheet as sufficient support, others wanted independent verification. E-E-A-T shifted based on whether oracles credited the reviewer's stated testing methodology as first-hand experience.
The overall score moved 3 points. The actionable signal didn't: Fact Grounding and E-E-A-T were consistently the weakest checks across all three runs. That's what needed fixing, not the overall number.
What you should do about it
Don't chase point changes
If your overall score went from 78 to 81 with no edits, that's noise. Don't celebrate it, don't worry about it, don't report it to your client as an improvement.
Do watch per-check patterns across runs
Run the same article 2–3 times if you're on the borderline of a decision (publish vs revise, client sign-off, etc.). Look at which checks are consistently weak, not which ones flipped on a single run.
A check that's red on 2 out of 3 runs is a real problem. A check that's amber, green, amber across three runs is a judgment call — worth improving but not blocking.
Do fix what the Improvement Pack says
The Improvement Pack (the prioritized action list on each scorecard) is generated from the consensus view across oracles, not from whichever oracle happened to be most generous. It's designed to surface issues that multiple models flagged, not edge-case disagreements.
If the Pack says "add a named author byline" on two consecutive runs, add the byline. If it only appears once, it's lower priority.
Do re-audit after you make changes
The whole point of re-auditing is to measure whether your fixes worked. One post-fix audit is enough — you don't need three runs to confirm that adding citations moved Fact Grounding from red to green. That kind of change is large enough to show up reliably.
When variance is a problem (and when to contact us)
Variance becomes a concern when:
- Overall score swings more than 15 points with no article changes and a full oracle panel
- A check flips from green to red (or vice versa) with no edits — not amber to green, but full status changes
- The same oracle fails to consult repeatedly on the same content type
If you see any of these, use the re-audit option or reach out via support. It may indicate a content pattern that's confusing the models in a systematic way — which is useful feedback for us, and usually fixable on your end.
The bottom line
Revylo scores are directional, not precise. They're designed to answer: "Is this article good enough to publish, and if not, what's most worth fixing?" — not "Is this article exactly a 78.4?"
Treat overall scores as bands (red / amber / green), treat per-check patterns across multiple runs as your real signal, and treat the Improvement Pack as your to-do list. The number will move. The patterns won't lie.
Questions about a specific audit? Run a free audit or see the Reference glossary for how each check is scored.