AI Content Quality: What Blind-Judging Our Articles Taught Us

Blind judging taught us that AI content quality is lost to hedging more than to invented facts. Our own reviewer gave articles 8 or more out of 10 that a blind, independent judge rated 4 to 5, mainly because caveats buried the answer. As articles answered first more often, most errors left were contradictions or rules from the wrong place.

This post covers what we did in October 2026, while building the article engine behind BlogAmplify: how the blind tests work, the lessons that changed what runs in production, the ideas we tested and chose not to ship, and what these tests can't tell you. If you want the short version of how an article is made today, it's on our methodology page.

Why we stopped trusting our own reviewer

Every article BlogAmplify writes goes through a review step: a model reads the draft against a list of pass-or-fail checks and scores it, and the draft is revised where it fails. Early on, that reviewer was happy with almost everything. It scored articles 8 or more out of 10. An independent judge, given the same articles without being told where they came from, scored them 4 to 5.

When we looked at why, the reviewer turned out to be the writer's twin rather than the reader's:

• It shared the writer's rules. One rule said an element the ranking pages cover wasn't required if the evidence couldn't support it. That excused exactly the gaps the judge marked down.
• It saw less than the judge. At the time it read a compressed summary of the ranking pages, not the pages themselves, and it reviewed the draft before the images were added.
• It kept running high. Even after the first fixes, a separate review of the pipeline put its scores about 1.3 points above the judge's.

None of this is unusual. A widely cited 2023 study of models used as judges examined "position, verbosity, and self-enhancement biases" in AI judges. A 2024 paper describes self-preference, "where an LLM evaluator scores its own outputs higher than others' while human annotators consider them of equal quality." Google's own guide to helpful content makes the same point for people: beyond assessing your own content, "consider having others you trust but who are unaffiliated with your site provide an honest assessment."

We didn't delete the reviewer, though. In the setup we were testing (a different writer plus the extra audits), we removed the scoring review and its full revision and put a single fix pass in their place. Accuracy dropped from 8.00 to 7.56 and confirmed errors rose from 1 to 4, against none in production's articles that round, even though that version still won 6 of 9 blind comparisons. So the reviewer stays as a checker. What we stopped doing is treating its score as a measure of quality.

How the blind judging works

Each round compares two versions of the same articles: the production engine as the reference, and one change we want to test. The steps are simple enough to copy.

1. Same inputs for both. Each version writes the same keyword for the same business, from the same business sheet. Where a business owner's answers were supplied, both versions get them, and the judge sees them too.
2. Strip anything that identifies a version. Internal scores, costs and timestamps are removed, and images are copied in under random file names.
3. Flip a coin for the labels. Each pair is labelled A and B at random, and the key that says which is which is saved outside the folder the judge reads.
4. Judge with the web open. The judge is a separate AI model. For each pair it looks at what ranks for the keyword, builds a fact-check table of the claims in both versions against their sources, scores eight criteria from 1 to 10 (intent, information gain, accuracy, E-E-A-T, readability, SEO, brand fit and language), names a winner with a margin, and lists confirmed errors.
5. Unblind and count. We tally wins and average scores, and we count confirmed errors separately, because a version can win on writing and still be wrong.

The test set covered nine kinds of business, so a fix that only helped one wouldn't look like progress. Seven were fictional test businesses with supplied facts: heating and cooling, family law, physical therapy, tax and bookkeeping, industrial compressed air, real estate, and a wellness hotel in Austria writing in German. Two were real online businesses in our own accounts: a fashion store and a fashion software company. Keywords ranged from "what size ac unit do i need" to "how long does a hamstring strain take to heal" and "was ist halbpension".

The blinding matters more than it looks. A 2024 study of position bias ran 15 AI judges through more than 150,000 evaluations and found that "position bias is not due to random chance and varies significantly across judges and tasks." A 2025 study on authorship labels found that AI evaluators favoured passages labelled as human-written by 34.3 percentage points, and that attribution labels "lead evaluators to invert assessment criteria." Random labels keep position bias from favouring either version on average, and a hidden key removes the authorship labels. Neither makes a single judgment free of bias, which is one reason one round decides little on its own.

Lesson 1: hedging is the biggest AI content quality problem

The judge's most common complaint wasn't a wrong fact. It was a missing answer. Take an early version of our article for "what size ac unit do i need", which the judge scored 5.1. The manufacturer sizing page it linked to says a 2,000 square foot house typically needs a 3-ton unit. Our article said "A 3-ton unit cannot be confirmed for a 2,000-square-foot Phoenix home from area alone," labelled its own two-homes table as not a sizing recommendation, and gave Phoenix design conditions a whole section without stating the temperature.

The hedging came from our own rules, not from the model's personality:

• "Avoid every number you can't verify." Told this, the writer wrapped useful figures in "it depends" and "consult a professional", and readers left without the number they searched for.
• The evidence funnel threw the answer away. For the hamstring article, the top results agreed on recovery ranges by strain grade. Our research step rejected the clinic pages that gave them as competitors, so the writer never saw the range everyone agreed on, and the number check then refused it.

What changed in production:

• An answer contract. The outline now commits to a direct answer with sourced figures before a word is written, and code checks that the answer comes first and that a "how much" or "how long" question gets a number under it.
• Consensus figures. A figure that two or more of the ranking pages give may be stated as common guidance, so the writer can give the range readers expect.
• Plain arithmetic. Calculations appear inline in plain numbers with each assumption stated once, and common rules of thumb are given as a rough guide instead of being refused.
• Voice checks in code. Code counts deferrals ("needs an assessment") and negations, and flags drafts that lean on them.
• A positive writer brief. The writer's instructions became a short description of a good article plus one example section, instead of a long list of prohibitions.

These changes reduced hedging; they didn't end it. In a later round, the judge still marked down one of our production reference articles on the same keyword for repeating "confirm with Manual J" in nearly every section and answering the 3-ton question with a tautology.

The rule we wrote down for ourselves: never bring back "avoid all unverifiable numbers" without the answer contract next to it. More on how this plays with Google's guidance is in Is AI content bad for SEO?

Lesson 2: the errors left are contradictions and misplaced rules

As articles answered first more often, confirmed errors became rare. Our production reference articles, the same set re-judged round after round, scored between 8.1 and 8.7 out of 10 on accuracy, with 0 to 2 confirmed errors across a round of up to nine articles.

The errors that remained, mostly in versions we were testing, had a pattern. They weren't invented statistics. They were the article disagreeing with itself, or a rule taken from the wrong place:

Production checks the numbers: a figure that doesn't come from the verified evidence, the business's facts or the owner's answers, figures two ranking pages agree on, or the article's own stated assumptions is flagged and revised, and code re-computes worked results (sums, products, percentages, table totals). Invented rates and wrong sums fail that check.

For the rest, we built and tested two extra audits: a consistency check that lists statements contradicting each other, and a rules audit for health, legal and money articles. They helped less than we hoped. Run offline against errors the judges had already confirmed, the checks caught 4 of 19, and 7 of 16 with a stronger model working harder. They find contradictions on the page. They rarely find a domain rule that is wrong, which a judge only confirms by searching the web, and a version of the rules audit that searched the web itself found none of those either.

They also made new errors. Two of one round's confirmed errors were the audits' own fixes: the consistency check "corrected" a business owner's 115°F performance check to the 108°F design temperature, and a rules fix moved a city's last estimated-tax date to one it recalled from memory. We added guards (the audits never change the owner's facts, and a fix that would bring in a figure nobody sourced becomes "remove the wrong part"), and the next round's error count dropped to 1 each. That is the lesson in miniature: an automated fixer needs the same checks as the writer.

Lesson 3: the live site beats the stored profile

In the final round of one series, the one error the judge confirmed in our tested version wasn't the engine's. The software company's stored profile said every plan starts with a free 7-day trial. The judge opened the live pricing page and found a paid trial offered on two plans. The article had repeated what the business profile told it.

Profiles go stale and sites change, so production treats the site as the truth about what a business offers:

• The outline step reads the business's own pages, and where the site contradicts the facts in the profile, the conflict is recorded and shown in the editor.
• A product price that changed on the site since the article was written is updated in the article before it publishes.
• A fact about the business that nobody supplied is left out of the article and filed as a short question for the owner, never posed to readers or guessed.

This works only as well as the latest read of the site. In a later test, our stored read of that same pricing page turned out to be older than the live page, so a stored profile is worth checking whenever your offer changes.

The owner's own knowledge was the biggest single lever we measured. In an early round, a heating and cooling article went from 5.9 to 8.0 with the judge when the owner's answers to a few simple questions were added. That's why the planner drafts up to three short, optional questions per topic, only about facts the site doesn't already state. Articles are written either way. The answers only help when the article uses them, though: the production reference article marked down in Lesson 1 had the same owner answers and left them unused.

Lesson 4: winning a comparison isn't the same as being right

The most tempting result we saw was a different model running every writing step, which won all 9 comparisons in its first round, averaging 7.71 against 7.07, with the biggest gains in intent and readability. The judge also confirmed 11 errors in it, against 1 in our production articles.

A cheaper model for the research step told the same story: it won 5 of 6 comparisons with 4 confirmed errors against none, so it isn't used anywhere.

After several rounds of fixes, the best tested setup (a different writer for the outline, draft and revisions, plus the two audits on the final text) won 9 of 9, averaging 7.81 against 6.88, with readability at 7.44 against 5.78 and 1 confirmed error each. Its accuracy score was still a little lower, 8.00 against 8.33. We kept production as it is, because the extra cost per article was larger than the gain we could measure.

Readable prose with a wrong rule in it is worse for a business than plain prose that's right. So we read the error count first and the win count second.

What is in production, and what we only tested

Two things hold either way. Articles are never held back: what the quality loop couldn't fix is recorded with the article, and owners who want to read first can turn auto-publish off and edit before anything goes live. And the blind judging itself is an offline test: it decides which changes we ship and never runs on a customer's article. Our editorial policy covers disclosure and corrections, and why we built BlogAmplify covers what the product will and won't do.

How to blind-judge your own articles

You don't need our tooling to run the same test on your own content, whether it comes from a writer, an agency or software.

1. Pick 5 to 10 keywords you actually want to rank for, and get two versions of each article.
2. Remove names, dates, file names and anything else that gives a version away.
3. Flip a coin for the A and B labels, and keep the key in a separate file.
4. Give the judge the facts about your business that both writers had.
5. Ask for a fact-check table first and scores second, so the scores rest on checked claims.
6. Use a judge that didn't write either version: a different model family, or a person unaffiliated with your site.
7. If you can, judge each pair twice with A and B swapped, and count a win only when both orders agree. The MT-Bench authors suggest calling it a tie when they don't.
8. Count confirmed errors separately from wins, and treat an error in a price, rule or safety claim as a loss whatever the score.

A scoring rubric you can copy

Our judges scored each criterion from 1 to 10 on one scale: 9 to 10 is better than the best ranking page, 7 to 8 is competitive and publishable after light edits, 5 to 6 is usable but generic or has real defects, and 4 or below shouldn't be published.

A short judge instruction that keeps the order right:

You are judging two articles, A and B, written for the search "[keyword]" for the business described below. First, look at what ranks for that search and note what every top result covers. Second, build a table of the factual claims in both articles, each with its source and whether the source supports it. Then score both articles on the eight criteria, name a winner and a margin, and list only the errors you confirmed against a source.

Google's guidance on generative AI content asks you to "focus on accuracy, quality, and relevance, especially when automatically generating the content." The same page says to fact-check and review AI content before publishing; owners who want that can turn auto-publish off and edit first. A blind test is one honest way to see whether a change moves those qualities, instead of trusting the tool that made it.

What this doesn't show

These results are useful to us, and limited. Read them with these caveats:

• Small samples. Each round had 6 to 9 articles, written once each. A 9 to 0 sweep or 11 errors against 1 is a real signal. A difference of 0.3 in average score is not: the average of the same reference articles, re-judged each round, moved between 6.88 and 7.13 on its own. Each pair was also judged once, in one random order.
• One judge model family. Every judge came from the same family of models, and that family also wrote the challenger in the writer-model tests. Self-preference may have helped the challenger, which is one more reason we weighted confirmed errors over wins.
• Model judges, not readers or rankings. A judge's score isn't a ranking, a click or a customer. These tests don't show how the articles perform in Google or in AI answers. That needs live data over months.
• Mostly fictional businesses, mostly English. Seven of nine businesses were test fixtures with supplied facts, and only one article was in a language other than English.
• Errors only count when found. A confirmed error is one the judge found and checked. An article with none may still have a mistake nobody looked for.

We'll keep judging changes this way before they ship, and we'll write up the next set of results here.