# The Yukich Group — mygreenvillescrealestate.com # # ── Search engines ───────────────────────────────────────────────────────── # Googlebot and Bingbot get their own groups, deliberately, even though the # `*` group below already allows everything. # # robots.txt gives a named group PRECEDENCE over `*` — a crawler that matches # a specific User-agent line ignores the wildcard group entirely. That is # normally a footgun, and here it is the point: if someone later adds a # `Disallow` to `*`, whether a deliberate one aimed at some path or a slip # that reads `Disallow: /`, these two groups mean it cannot reach the crawlers # the business depends on. The failure it prevents is the whole site leaving # Google. # # The cost of that precedence is that any `*` rule added later must be # repeated here to apply to Google and Bing. Nothing in this file needs to be: # the paths we keep out of the index are handled with `noindex`, not # `Disallow` — see the note further down for why that is the stronger choice. User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: * Allow: / # ── AI assistants that fetch a page to answer a live question ────────────── # Named explicitly rather than left to fall through `*`, for the same reason # Googlebot is: so a future edit to the wildcard group cannot quietly cut them # off. Being cited by an assistant is a stated goal of this site. # # These are the RETRIEVAL agents — they fetch a page because a person just # asked something, answer from it, and keep nothing. That is a single-session # use, which is not what the MLS GRID AI Use Addendum restricts (see below), # so they are allowed everywhere, including the pages that render listings. # # This is the distinction that matters in the block below, and it is easy to # lose: OAI-SearchBot is not GPTBot, and Claude-SearchBot is not ClaudeBot. # Same companies, different jobs, different rules. User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Perplexity-User Allow: / # ── AI TRAINING crawlers ─────────────────────────────────────────────────── # Welcome on our own writing, and deliberately so: the community guides, the # buying and selling guides, the relocation guide, the blog, and the # about/team pages are ours, and being cited by an assistant is a stated goal # of this site. # # Only the two pages that are NOTHING BUT MLS data are withheld. The MLS GRID # AI Use Addendum (part of the Data License Agreement covering the Spartanburg # feed) prohibits using that data for "AI Training", which it defines to # include "the creation of vector embeddings, retrieval indices, knowledge # graphs, or similar representations that persist beyond a single user # session", and separately bars merging it "into a model's general knowledge # base such that ownership can no longer be identified". # # ── Why /communities/, /areas/ and /lakes are no longer listed here ───────── # They used to be, because functions/_lib/inventory.js embeds live listing # cards into their HTML. Blocking them satisfied the licence by withholding # 19,176 words of our own guides along with the listings — the single best # original writing on the site, none of it MLS-derived. # # The obligation is now met where it belongs, in code: `injectInventory` # checks the User-Agent and SKIPS the listing injection for the four crawlers # named below, so the page they receive carries the full prose and no MLS # record at all. That is stronger than this file, not weaker — robots.txt is # advisory, and a crawler ignoring it previously received the data anyway. # # The suppression is invisible from here. tests/unit/ai-training-suppression.test.mjs # is what holds it up, and the `# @ai-training` annotations below are what # keep that matcher and this file naming the same four crawlers. Adding a # fifth training crawler means adding it in BOTH places; the test fails if # this file names one the code does not know. # # These rules bind only the named AI-training crawlers. Googlebot, Bingbot, # and user-triggered assistant fetches are governed by the * group above and # remain allowed everywhere public. # @ai-training GPTBot User-agent: GPTBot Allow: / Disallow: /listing Disallow: /home-search # @ai-training ClaudeBot User-agent: ClaudeBot Allow: / Disallow: /listing Disallow: /home-search # @ai-training PerplexityBot User-agent: PerplexityBot Allow: / Disallow: /listing Disallow: /home-search # @ai-training Google-Extended User-agent: Google-Extended Allow: / Disallow: /listing Disallow: /home-search # ── Why /account, /saved and /agent are not Disallowed here ──────────────── # They are private, and the instinct is to add them. Doing so would be the # wrong tool and would make two of them MORE likely to appear in Google, not # less. # # `Disallow` forbids the FETCH. `noindex` forbids the INDEX ENTRY. They are # not two strengths of the same thing, and using the first where you want the # second backfires: Google can list a URL it was never allowed to fetch — # titled from whatever links to it — precisely because it could not fetch the # page and read the noindex. Google's own guidance is to allow the crawl when # you want something removed. # # So each of these already carries the directive that actually applies: # # /account, /saved , # and excluded from the sitemap. # /agent/* noindex meta AND an x-robots-tag header, behind # Cloudflare Access — a crawler gets 403 and never sees # a page at all. A Disallow would add nothing except a # public advertisement that the path exists. # /listing (shell) x-robots-tag: noindex via public/_headers. # # tests/unit/seo-thin-pages.test.mjs asserts no noindex page reaches the # sitemap, which is the invariant that actually keeps these out. # ── Why there is no faceted-URL Disallow ─────────────────────────────────── # An SEO audit asked for `Disallow: /home-search?...` on the crawl-budget # argument that faceted search URLs create an infinite space. Checked against # the build before acting, and it does not apply here — twice over: # # 1. Nothing internal links to one. All 147 built pages were grepped: every # internal link to the search page is the bare `/home-search` (451 of # them). # # This used to say browsing produced no new URLs at all, because the # filters were read from the query string and never written back. That # stopped being true on 2026-08-23: the search now writes its filters into # the address bar with replaceState, so that a listing opened from a search # and returned from does not lose it. Filtered URLs are therefore real, # shareable, and linkable from outside. # # It does not change the conclusion, because the reason below was never # "the URLs do not exist" — it is that the ones which do already collapse # onto one canonical. replaceState also creates no history entries and no # crawlable hrefs: Googlebot does not drive the filter bar, so it still # reaches these only if a person shares one. # # 2. The ones a visitor could still share already self-canonicalise. The # canonical is derived from the PATH and ignores the query, verified # against the running Worker: # # /home-search?beds=3&maxPrice=500000 -> canonical .../home-search # /reviews?agent=Brenda%20Yukich -> canonical .../reviews # # So Google already consolidates them. Adding a Disallow would make this # WORSE, not better: a blocked URL can still be indexed url-only when someone # links to it, and blocking the fetch is what stops Google from ever seeing # the canonical that would have consolidated it. At 147 pages there is no # crawl budget to protect either — the audit says so itself. # # That revisit happened — see point 1. The self-canonical still holds, verified # again on 2026-08-23: # # /home-search?beds=3&maxPrice=500000&q=Greenville -> canonical .../home-search # # Revisit again if the canonical ever starts varying by query, which is the # thing that would actually break the consolidation. Sitemap: https://www.mygreenvillescrealestate.com/sitemap-index.xml