Meta’s Oct 30 Crawling Terms: Which Bot to Block

If you buy media on Meta, you have probably seen a version of this week’s advice: Meta’s new crawling terms take effect October 30, so go block meta-externalagent in your robots.txt before the deadline.

That advice is not wrong. It is incomplete in a way that matters, because it names the AI-training crawler and skips the advertising one. Meta’s own crawler documentation lists five distinct user agents, and the one whose stated job is “improving advertising and other business-related products and services” is a separate, separately-blockable agent called meta-externalads. Almost nothing published this week mentions it.

Here is the whole thing, sorted by what you can actually act on in the eleven days left.

What Changed, and What Changed Back

On October 15, Nieman Lab reported that Meta’s updated Commercial Terms — effective October 30 — contained a new clause granting Meta the right to crawl websites associated with any business that uses its products. The clause read, in part:

“You authorise us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyse, collect, retrieve, or crawl the domains, web pages, or URLs… Meta may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of the Meta Company Products.”

Three legal academics who reviewed the language for Nieman Lab read it as an argument Meta could use to scrape news sites for AI training. Cornell’s James Grimmelmann flagged the “test credentials” provision as something that would technically let Meta register as a fake user to get past a registration wall. Meta declined to say whether test credentials would reach paywalled content.

Then, late on Friday, October 16, Meta rewrote the preview. Per The Next Web, the revised terms no longer use the word “train,” and the company’s statement was “We’ve heard you and want to clarify” — its position being that the clause only ever applied to its ads and business products, and that it now says so explicitly. Meta also said it is not changing how its crawlers work.

The News/Media Alliance is not satisfied, and its objection is the sharp one: training was removed from the list of permitted uses, not expressly prohibited. Spokesperson Sam Quigley also noted that the terms still do not say Meta will respect robots.txt.

Two things did not change in the rewrite. The effective date is still October 30. And acceptance is still passive — you agree by continuing to use Meta for business. The stated alternative is to stop.

If You Already Buy Meta Ads, Most of This Is Not New

This is the part that should lower your blood pressure, and it is missing from most of the coverage because most of the coverage was written for publishers rather than advertisers.

Meta’s current Self-Serve Ad Terms, dated August 22, 2025, already grant Meta the right to crawl content on an advertiser’s domains and URLs, and the listed uses there already include “research and training.” The Ad Creative Generative AI Terms have permitted crawling of advertiser-supplied URLs since May 2024. The Product Catalog Terms already allow periodic re-pulls from catalog URLs. If you have run a Meta campaign in the last year, you granted a crawl right then.

What is genuinely new is scope, not permission. The old grant attached to placing an ad order. The new one attaches to business use of Meta’s products at all — running a Page, sitting in Business Suite, managing a catalog. The current Commercial Terms, dated March 2024, have no crawling clause whatsoever. So the population covered expands to include brands with a Page and no spend, agency-managed client Pages that have never run a campaign, and catalog-only sellers.

Meta is also folding the Product Catalog Terms into the Commercial Terms and the Ad Creative Generative AI Terms into the Self-Serve Ad Terms — fewer documents, and one broad grant in place of several narrow ones.

The Five Crawlers, and the One the Advice Misses

Meta’s published crawler documentation is the operative reference here, and it is more specific than the terms are. Five agents, five jobs:

User agent	Documented purpose	robots.txt
meta-externalads	Improving advertising and other business-related products and services	Presented as controllable
meta-externalagent	Training foundation AI models; indexing content directly	Presented as controllable
meta-webindexer	Improving Meta AI search result quality; enables citation and linking	Presented as controllable
meta-externalfetcher	Fetching individual links on a user’s request, for agentic AI features	May bypass
facebookexternalhit	Link previews when a URL is shared on a Meta surface	May bypass for security or integrity checks

The advertising crawler and the AI-training crawler are different bots. That distinction is the whole practical story, and it has two consequences.

First, the “block meta-externalagent” advice is narrower than it sounds. It addresses foundation-model training and leaves the advertising crawl untouched — possibly what you want, but you should know that is what you did.

Second, the reflex of blocking all Meta bots is the one to avoid. Meta documents meta-externalads as serving advertising and business products; it is a reasonable inference that this is the agent reading the landing pages and product URLs behind ad relevance, catalog freshness and automated creative. Blocking it is a decision about your own ad delivery, not a privacy win.

Worth noting what Meta’s own documented example config does: it addresses exactly two agents, meta-externalagent and meta-externalads, and sets both to Allow: / with a single private directory disallowed. The default Meta publishes is permissive for both.

What Meta Has Not Said

Be honest about the gap here, because it determines how confidently you can act.

Meta has not published what degrades if you disallow meta-externalads. There is no documented statement that ads get rejected, that dynamic product ads go stale, or that automated creative quality drops. All three are plausible, and you will see them asserted this week as though they were documented. They are not. The documentation gives the crawler’s purpose and stops.

Meta also has not reconciled the two layers. Its spokesperson points to robots.txt as the control; the terms do not mention robots.txt in either version. Contractual consent and technical blocking are separate systems that never reference each other, and nothing published says which wins. That is the real unresolved question, and no amount of robots.txt editing answers it.

The Case That This Is All Theater

Here is the strongest argument against doing anything at all, and it deserves a hearing because large parts of it are correct.

robots.txt is a request, not an access control. Meta’s own documentation concedes that two of its five agents may ignore it. The terms do not promise to honor it. The rewrite removed a word from a list of uses and surrendered no right. And the only opt-out on offer is to stop using Meta for business, which no advertiser is actually considering. On that reading, an afternoon spent on robots.txt buys a feeling, and the feeling is the product.

That holds as far as it goes. If you think a robots.txt edit is a legal shield, you are wrong, and better to drop that belief before October 30 than after.

Where it breaks down is the assumption that AI training is the only thing at stake. It is not. The per-agent split is an operational decision about your ad accounts that happens to live in the same file. Whether Meta’s advertising crawler can read your pricing pages, your out-of-stock states and your terms is a question about ad quality and brand accuracy, and it would be worth a deliberate answer even if the training debate had never happened. You are not only choosing what Meta learns. You are choosing what Meta knows about you when it writes an ad in your name — and with Advantage+ generating more of that creative on Meta’s initiative, the stale page you forgot about is now a creative input.

That is also the quiet continuity with ChatGPT’s agent campaigns and July’s Google Ads terms update, which expanded URL crawling authority in much the same direction. Three platforms, one pattern: your website is becoming a media-buying input, and the grant arrives through terms you accept by continuing to log in.

What To Do Before October 30

1. Read your current robots.txt before you touch it. If it already carries a blanket User-agent: * disallow on directories your landing pages live under, you may already be blocking the advertising crawler without having decided to.
2. Decide the three agents separately. meta-externalagent (AI training), meta-externalads (advertising and business products), meta-webindexer (Meta AI search visibility and citations). These are three different business questions and a single rule answers none of them well.
3. Keep facebookexternalhit allowed unless you want your shared links rendering without a title or image across Facebook, Instagram, Messenger and WhatsApp.
4. If you block the advertising crawler, treat it as a change and watch it. Meta has not documented the effect, so you are the test. Note the date, then watch disapproval rates, catalog sync health and automated creative quality for two weeks.
5. Agencies: inventory the Pages you administer. The new scope reaches clients who have a Page and no ad spend, and they will not have read any of this. One email beats one surprise.
6. Write down what you decided and why. These terms were edited mid-notice-period once already. The next version is easier to assess against a record of your reasoning than against your memory of a news cycle.
7. Keep your own copy of the performance record. When a platform can change what it reads from your site, and its own AI can change what it writes from that, a schema you control across platforms is what keeps “what changed before this moved?” answerable.

The Takeaway

The headline story — Meta claimed a scraping right, publishers pushed back, Meta edited the clause — is mostly not an advertiser story. Your ad terms granted a crawl right over a year ago, and the rewrite did not take it back.

The advertiser story is the one underneath it. There are five Meta crawlers, not one. The advertising crawler is separately named and separately blockable, Meta’s own example config leaves it wide open, and the effect of closing it is undocumented. That is a real decision with a real deadline, and it is nowhere in the advice going around this week.

Eleven days. Read the file, choose per agent, write down why.

Sources: Search Engine Journal — Meta’s New Terms Expand Crawling Rights Beyond Advertisers (October 2026), Nieman Lab — Meta’s new commercial terms will allow the scraping of news sites, legal experts say (October 15, 2026), The Next Web — Meta drops ‘train’ from its website-crawling terms after backlash (October 2026), News/Media Alliance — Changes to Meta Commercial Terms of Use (October 16, 2026), Meta for Developers — Web Crawlers, Meta — Commercial Terms preview, Meta — Self-Serve Ad Terms preview