SEO and AEO Automation

What SEO and AEO automation is

The reason this exists as a service is that search work decays quietly. A strategy document is written, two people execute it for six weeks, a launch happens, and nobody looks again until traffic moves. The jobs below are the parts that should never have depended on somebody remembering.

The second reason is that ranking and being cited have become two different outcomes. A page can sit at the top of a results list and never be named in the synthesised answer above it. Those failures have different causes and different fixes, so a system that measures only one of them is reporting half the picture.

Terms this page uses precisely

Answer engine

Any system that returns a synthesised answer instead of a list of links. The practical consequence is that ranking and being named in the answer are separate outcomes that need separate measurement.

Citation

A named reference to your brand, or a link to your page, inside a generated answer. Being present in a retrieval index is not a citation, and neither is ranking well for the query.

Extractable unit

A block that still makes sense when cut out of its page: a definition sentence, a table, a definition list, an FAQ entry. Retrieval works on chunks, so the unit rather than the page is what gets used.

Divergence set

The queries where you rank but are not named in the generated answer for the same intent. This set is the work queue, because the fix is not a ranking fix.

Crawl diff

The difference between two crawls of your own site: pages added, removed, redirected, retitled, or with a changed canonical. Most search incidents are visible here before they appear in analytics.

Who it is for, and who it is not for

This suits a team that already knows what it is trying to rank for and cannot keep up with the execution. It does not suit a team looking for a strategy, because the jobs described here amplify whatever direction you point them in.

This fits you if

• You run a few hundred pages or more, and nobody could tell you today which of them have no internal links pointing at them.
• Search matters to revenue, and a ranking drop currently gets noticed by a person rather than by a system.
• You publish continuously, so briefs and internal links are perpetual work rather than a one-off project.
• Somebody owns search outcomes and will act on a weekly diff rather than filing it.

Do not build this if

• You have forty pages. Read them yourself once a month. Automation costs more than the reading does.
• You want the system to edit live pages without review. On a site that earns money, that is a bad trade at any speed.
• Nobody will act on the output. A weekly report that nobody opens is a running cost with no return.
• The real goal is page volume without editorial control. That is a different build with its own gate, described in content production engine.

What we actually build

Six jobs and one delivery mechanism. Every job is read-only against the outside world, and every write is a proposal.

The crawler and the diff store

A crawl of your own site on a schedule, stored as snapshots rather than as a live view. The diff between snapshots is where incidents surface: a canonical that changed during a release, a template that stopped rendering a heading, three hundred pages that started redirecting. History matters here, because the useful question is always what changed and when.

The integrity checks that run against the live host

Every URL in your sitemap must return 200 rather than a redirect, and every page's canonical must equal the URL it was served at. Both defects are invisible in a browser and both waste crawl budget on a site that has none to spare. We check them against the live host, not against the build directory, because those two disagree more often than anyone expects.

The visibility sampler

Two measurements per tracked query, kept apart: where you rank, and whether you are named in the generated answer. Sampling is repeated because generated answers vary between runs, and a single check is an anecdote. The output is a rate over the sampling window rather than a verdict.

The brief generator and the link proposer

Gaps from the divergence set and the crawl become briefs carrying must-answer questions, the entity to name, and the internal targets. The link proposer scores candidate pairs from paragraph embeddings, filters them hard, and emits a diff. Both jobs stop at a proposal.

link_proposer:
  candidate_source: paragraph_embeddings    # chunk at paragraph, never at page
  top_k: 40

  filters:
    anchor_must_appear_in_sentence: true    # never invent an anchor phrase
    anchor_min_words: 2
    anchor_max_words: 6
    exclude_reciprocal_pairs: true
    target_must_not_be: [homepage, contact, pricing]
    min_semantic_score: 0.62

  caps:
    outbound_new_links_per_page: 3
    inbound_new_links_per_target_per_run: 5
    same_anchor_text_sitewide: 8            # the spam-pattern brake

  boosts:
    orphan_targets: 2.0                     # zero inbound internal links today
    depth_over_3_clicks: 1.5
    recently_updated: 1.2

  output:
    mode: proposal                          # this job never writes
    format: diff
    reviewer: content-lead
    expires_days: 14

The structured data validator and the digest

Structured data is generated from the same record the page renders from, then validated against what the page actually shows. Anything the markup claims and the page does not is a defect, not a detail. The digest then lands where work happens, which is a ticket queue or a channel, not another dashboard nobody opens.

How it works technically

Seven stages. The whole system is deliberately read-only outside your own repository and ticket queue.

1. Crawl. Fetch your own pages on a schedule with a polite rate, store the raw HTML, the rendered text, headings, canonical, title, schema blocks and internal links, keyed by URL and crawl id.
2. Diff. Compare against the previous snapshot. Classify changes: added, removed, redirected, canonical changed, title changed, heading structure changed, internal links lost. Lost internal links are the quietest and most common regression.
3. Integrity. Sample sitemap URLs and assert a 200 on each, assert canonical equals served URL, and compare the newest lastmod per page class so a frozen class is visible.
4. Sample visibility. For each tracked query, record ranked position from your own console data, and separately record whether the brand is named in a generated answer, over repeated runs.
5. Classify gaps. Build the divergence set from queries that rank but are not named, then attach the candidate page and the audit of its extractable units.
6. Propose. Generate briefs for content gaps and link diffs for structure gaps. Both carry the evidence that produced them, so a reviewer can reject the reasoning rather than just the output.
7. Deliver and log. Emit the digest into the ticket queue, and log every job run with its inputs, duration, and what it proposed. Logging discipline for this kind of system is covered in what to log so future you can debug it.

Audit one URL for extractable units. Return JSON only. Do not rewrite the page.

{ "url": "...",
  "primary_entity": "the subject of the page, named as a person would name it",
  "units": [
    { "type": "definition | table | list | faq | steps | comparison",
      "text": "the unit, verbatim, up to 60 words",
      "self_contained": true,
      "entity_named_inside": true,
      "answer_first": true,
      "dated_inside": false,
      "fails": ["entity_named_inside"] }
  ],
  "missing_units": ["a one-sentence definition of the primary entity"],
  "verdict": "citable | partially_citable | not_citable" }

Rules:
- self_contained is false if the unit needs the previous paragraph to make sense.
- entity_named_inside is false when the unit says "it" or "the product" where the
  page title carries the name. This is the most common failure by a distance.
- answer_first is false if the unit describes the topic before answering it.
- dated_inside is true only if a date or version sits inside the unit itself.
- Never invent a unit that is not on the page. An empty units array is a result.

Rate limiting deserves a line of its own. Every external call in this system needs a bounded retry with backoff and a circuit breaker, because a monitoring job that hammers a provider during an outage turns a small problem into a blocked account. The mechanics are in rate limits, retries and backoff.

The Citation Surface Audit

Being named in a generated answer is not a ranking problem with a different label. Retrieval systems work on chunks, so what gets used is a block of your page rather than the page. A page can be authoritative, well linked and useless to an answer engine, because none of its blocks survive being cut out.

One caution. This is a way of making a page easier to use, not a lever that guarantees a result. Engines differ, they change often, and no vendor can promise you a citation. What the audit gives you is a specific, cheap list of defects, which is a better place to be than guessing.

The build process, stage by stage

Four to eight weeks. The crawl and the integrity checks come first, because they usually find something worth fixing before anything else is built.

What you get at handover

The jobs run on your infrastructure against your data, and the history stays yours. That last part matters more than it sounds, because the value of this system is mostly in the comparison against last month.

Attribution deserves honesty at handover. Search outcomes move for reasons this system does not control, so we report what the jobs did and what changed, and we resist the arithmetic that turns one into a claim about the other. The general argument is in measuring whether an automation actually worked.

Where these projects go wrong

Six failure modes, and the first two account for most of the damage.

Treating a single generated answer as a measurement

Ask the same question twice and you can get two different answers with different sources named. A screenshot of one run is not evidence of anything. Sample repeatedly, report a rate with the sample size attached, and never let a single run become a slide in a meeting.

Letting the system write to the site

The moment a job can edit pages, one bad run becomes a live problem across hundreds of URLs, and the rollback is a restore rather than a rejection. Proposals cost a reviewer a few minutes a week. That is the cheapest insurance in the build, and we do not build it the other way.

The internal link engine produces a pattern

Unfiltered proposals converge: the same anchor phrase pointing at the same page from four hundred places, links inserted into sentences that never mentioned the topic, reciprocal pairs everywhere. The caps in the config above exist for this. A sitewide anchor limit and a rule that the anchor must already appear in the sentence remove most of it.

Briefs that average the competition

Generate a brief from the top ten results and you get a specification for the eleventh version of a page that already exists. A brief needs something the others do not have: your data, your named entity, a decision you made and can defend. Otherwise you are automating the production of a page nobody needs.

Structured data generated from a template

Markup that claims a rating, a price or a date the page does not show is a defect and an eventual liability. Generate structured data from the same record the page renders from, then validate the two against each other on every deploy. Never hand-maintain a second copy of the truth.

Sitemaps and canonicals that disagree with the served URL

A sitemap listing URLs that redirect spends a crawler's budget on redirects, and canonicals pointing at the redirecting form tell every reached page that its real address is somewhere else. Both are invisible in a browser. Decide the URL shape once in the build config, and make the page generator, sitemap writer, link writer and canonical writer read that one setting.

A sitemap is a promise. Every URL in it should return 200 on the first request.

What it costs to run once it is live

Model inference is the smallest line here by a wide margin. The costs that matter are engineering maintenance and the human minutes spent reviewing proposals. Run the numbers below on your own assumptions and your provider's published prices today.

The number that comes out is small, and that is the useful finding. Sampling more often is cheap, so the constraint on sampling frequency is provider rate limits and permitted access rather than budget. Decide frequency on what you would actually do differently with a daily signal instead of a weekly one.

How to tell whether you need this

Five questions. Count the yeses.

• Could you name, right now, which of your pages have no internal links pointing at them?
• Would a canonical change shipped in a release be noticed by anything other than a traffic drop weeks later?
• Do you know which queries you rank for and are not named in the answer for?
• Does your search work happen every week, or only in the weeks when nothing is on fire?
• Is there a person who will act on a weekly diff, rather than file it?

One or two yeses means monitoring will tell you things you are not staffed to act on, which is a cost rather than a gain. Three or four is the standard case. Five means you are already running this by hand, and automation converts your best person from an executor into a reviewer.

How to start

The first useful step costs nothing and does not involve us: run the integrity checks on your own site. If they come back clean, you are in better shape than most, and the rest of this is an efficiency conversation.

1. Check your sitemap and canonicalsToday

   Take thirty URLs from your sitemap and confirm each returns 200 rather than a redirect. Then confirm each page's canonical matches the URL it was served at. Any failure here outranks everything else on this page.

2. Send read access and a query listBefore the call

   Analytics, search console, and the twenty queries that actually matter to revenue. Twenty beats two thousand, because the shorter list is the one people will read.

3. A ninety minute scoping callWeek 0

   We walk the current process: who watches what, what gets noticed and how, and what happened the last time rankings moved. That last story tells us what to alert on.

4. A written scope with the limits statedWithin a week

   Jobs, cadences, what we will sample and under what access terms, what the system will never do, and the baseline we measure against. Anything we cannot legitimately access is named in the scope rather than discovered later.

5. One weekly cycle, run by youWeek 8

   Handover means your team receives the digest, approves a link diff, commissions a brief and fixes an integrity failure without us in the room.