How to detect near-duplicate programmatic SEO pages

A programmatic page family can have 500 unique URLs, 500 unique titles, and 500 unique H1s while still giving users the same page 500 times.

That is the near-duplicate problem.

The mistake is easy to miss when pages are reviewed one at a time. /accountants/austin/ looks plausible by itself. So does /accountants/dallas/.

The risk becomes visible only when you put the pages next to each other and notice that the main answer, proof, examples, and recommendations are unchanged except for the place name.

Near-duplicate auditing is therefore a page-set task. You are testing whether each URL has a reason to exist relative to its siblings and to pages already on the site.

Duplicate and near-duplicate are not the same thing

An exact duplicate repeats the same page or substantially identical main content. Different URL variants can create exact duplicates even when users reach them through different addresses.

A near duplicate is not word-for-word identical. Small substitutions, reordered sentences, light rewriting, or a different entity name may make the text technically different while preserving essentially the same answer.

That distinction matters because simplistic checks miss both sides:

• exact string comparison misses lightly rewritten copies;
• URL, title, and H1 checks can all pass while the body remains almost unchanged;
• a word-count check can say two pages are “long enough” even when all of those words are shared;
• a template can be reused safely when the data underneath it creates genuinely different answers.

Google’s canonicalization documentation says duplicate content on a site is normal in many situations and is not automatically a spam-policy violation. Separately, Google’s spam policies warn about generating many pages primarily to manipulate rankings when those pages contain unoriginal content with little or no value for users.

So the useful audit question is not “What percentage of this page is unique?” It is “Does this URL provide a meaningfully different answer that deserves to be found separately?”

Template reuse is normal. Answer reuse is the risk.

A programmatic SEO system is supposed to reuse structure. That is the point.

The following repetition is usually expected:

• navigation, footer, breadcrumbs, and standard site chrome;
• the same content block order;
• labels, forms, CTAs, disclosures, and legal text;
• a shared explanation that genuinely applies to every member of the family;
• common design components and structured-data patterns.

The dangerous pattern appears in the main answer.

Imagine a location template where every page says the company has experienced staff, offers the same services, follows the same process, and provides the same generic benefits. The only values that change are Austin, Dallas, and Houston.

The pages are structurally different, but the user learns nothing location-specific from choosing one instead of another.

A simple editorial test catches a surprising amount of this: remove the page-specific name from two sibling pages and read the remaining main content. If the pages become interchangeable, the page family needs more than a synonym pass.

That is a diagnostic, not a Google rule. The real fix is to add information that changes because the subject changes.

Structural uniqueness and content uniqueness solve different problems

Programmatic workflows often confuse these because both produce visible differences in a spreadsheet.

Layer	What makes it distinct	What a clean result proves	What it does not prove
Page identity	Unique URL	The row will not resolve to the same address as another row	The page deserves a separate search result
Search-facing structure	Title, H1, canonical, parent, indexability	The page fields are present and not exact copies where the rules check them	The main answer is different
Declared intent	Target query or intent field	The plan expresses who should own a declared target	Different wording can still hide the same intent
Content	Page-specific facts, constraints, relationships, proof, examples, availability, comparisons	The user receives information that changes with the page subject	Google will index or rank the page

A structurally clean page can still be a near duplicate. A content-rich page can still have a duplicate URL or broken canonical. You need both checks because they answer different questions.

This is also why a single site score is weak evidence. A 92/100 average can hide a cluster of 40 pages that should never ship.

What “only change the city/product/name” gets wrong

Entity substitution is not automatically bad. A city, product, integration, category, or person may genuinely change the answer.

The problem is using the entity as the only change.

For a location page, useful differences might include:

• service availability in that location;
• regulations, permits, taxes, climate, or logistics that alter the advice;
• local pricing inputs or operating constraints;
• named proof, projects, inventory, or coverage data;
• relationships to nearby areas or relevant local resources.

For an integration page, the useful differences might be supported triggers, objects, limitations, authentication, setup steps, or workflows that actually vary by integration.

For a comparison page, they might be dimensions where the products differ, the evidence for those differences, and which buyer situation changes the recommendation.

If the source data contains none of this, a writing model cannot manufacture a trustworthy page-specific answer out of a different noun. It can only make repetition harder to notice.

Audit the page family, not a handful of “representative” URLs

The practical unit of review is the family produced by the same page model, plus nearby families that could reuse the same answer.

Use this workflow before publishing.

1. Define what is supposed to change

Write down the page-family contract before reviewing prose.

For example:

Each city page must change service availability, local constraints, proof, and nearby coverage. The city name alone is not a sufficient difference.

That statement gives the auditor something more useful than “make every page unique.”

2. Check identity and structure first

Find deterministic problems before judging content:

• duplicate or invalid URLs;
• duplicate titles and H1s;
• missing required fields;
• unresolved placeholders;
• canonical conflicts;
• missing parent relationships;
• declared intent overlap.

These are cheap to fix and often expose a broken data model before content comparison even starts.

3. Compare the body across the whole supplied set

Then compare the main content, not just metadata.

pSEO Guard’s current content-risk audit can detect body-text near duplicates across the supplied page set and separately catch sibling pages that remain almost the same after page-specific names are discounted. The public near-duplicate rule uses a configured similarity threshold with a default of 0.85.

That number is a product review threshold, not a Google requirement. It does not mean Google requires “15% unique text,” and it should never be converted into a content-writing quota.

The current implementation compares siblings inside the same page family more completely and uses a bounded candidate search across different families. The coverage note matters because a result should not imply that every possible pair was exhaustively compared when it was not.

4. Ask whether the answer changes

Similarity is useful evidence, but it is not the final editorial decision.

For the closest pairs, ask:

• Does the recommended action change?
• Do the facts or constraints change?
• Does the page use subject-specific evidence?
• Would a visitor choose one page over the other for a clear reason?
• If the entity name were removed, could you tell which page you were reading?
• Could one page satisfy both needs without losing anything useful?

If the last answer is yes and the earlier answers are mostly no, two URLs probably do not need to exist.

5. Review page-specific value across the family

Near-duplicate pairs are only one failure pattern. A whole family can share so much vocabulary and evidence that each page contributes very little of its own.

Look for rows whose distinct material is limited to:

• the target name;
• one repeated adjective;
• a templated sentence with a substituted number;
• a rewritten introduction followed by the same body;
• generic claims that could appear on every sibling.

The goal is not to hit an arbitrary “unique content percentage.” The goal is to identify what the page contributes that another member of the family cannot.

6. Keep unmeasured rows visible

If body text is missing, content similarity was not checked.

In workflows where body content is expected, pSEO Guard marks that gap as Review rather than letting an unmeasured content check look like a pass. Very large comparison groups can also exceed bounded pairwise coverage; affected pages remain reviewable instead of being silently called clean.

This is a crucial quality-gate rule:

Unmeasured does not equal passed.

A structurally clean row can be Ready only for the checks that actually ran. The coverage documentation explains the difference between measured, not measured, not compared, and failed content analysis.

7. Compare the proposed family with the existing site

A new page may be unique relative to its siblings and still copy something already published.

The Existing Site Guard compares planned pages with the pages its bounded crawl retrieved. An exact live URL collision is Blocked, a title collision is Review, and a planned body that is too close to retrieved live content is Blocked.

The same coverage rule applies. A page the crawl never retrieved cannot be compared. A live page without usable body text cannot participate in body similarity. Missing coverage is uncertainty, not evidence of uniqueness.

How to decide what happens to a near-duplicate cluster

Once you find a cluster, do not begin by adding adjectives.

Choose the page decision first.

What you found	Better action
Two planned pages answer the same need	Merge the rows and build one stronger page
One planned page has no distinct evidence or task	Remove it from the plan
The pages need separate URLs but their answers are still interchangeable	Redesign the data model and rewrite around genuinely different facts or needs
A planned page duplicates a useful live page	Reuse or improve the existing page rather than create a second copy
Multiple duplicate or very similar URLs must remain reachable	Choose a representative canonical where appropriate
Pages share a template but provide different answers and evidence	Keep the shared structure; template reuse itself is not the problem

A canonical can be correct in the fifth case. It is not a general cure for the first four. Google treats canonical declarations as a preference signal and may select a different canonical, so fixing the page model is still the primary job.

Thin content is not a word-count formula

Near duplicates are often described as “thin,” but thin content should not be reduced to page length.

Google does not publish a universal minimum word count that makes a page useful or indexable. A concise page with the exact facts a user needs can be valuable. A 2,000-word page assembled from the same generic paragraphs as every sibling can still add almost nothing.

pSEO Guard does have a thin_page rule: measured or declared content below the current default 250-word threshold is marked Review. That threshold is a product heuristic, not a Google rule.

The rule’s recommended action is to ask whether the page has enough unique evidence to satisfy its intent, not to pad the page until a counter turns green.

That is the right way to use quantitative checks: as prompts for a decision, not as invented search-engine requirements.

A compact page-family audit framework

Before a page family moves forward, review it through four lenses:

1. Identity: Does every URL identify one intentional page without exact collisions?
2. Intent: Does every page own a user need that is not already better served by a sibling or live page?
3. Answer: Does the main content materially change when the page subject changes?
4. Evidence: Can you point to facts, constraints, relationships, examples, or proof that belong specifically to this page?

Then attach the result to the page itself:

• Ready when the checks that ran do not require action;
• Review when a human decision or incomplete measurement remains;
• Blocked when a finding should stop the page from moving forward.

That is more useful than approving a template once and assuming every row generated from it inherits the approval.

The point of pre-publish duplicate detection

Google can cluster duplicate or very similar pages and choose a representative canonical. That is not a reason to outsource page planning to Google’s indexing system.

By the time a search engine has to decide which of your URLs is representative, you have already paid to generate, review, publish, crawl, maintain, and internally link those URLs.

The cheaper decision is made while they are still rows in a Page Matrix.

Compare the family. Identify what actually changes. Remove pages whose only difference is a token. Keep shared template structure when it helps users. And when the audit could not measure something, leave that uncertainty visible rather than converting it into a green check.