Robots.txt for SEO — a practical guide to robots.txt SEO with clear examples you can reuse in real projects.
SEO Tutorial Series (40/102). Prefer one article? Read the complete SEO tutorial.
Short description
robots.txt blocks crawling — not always indexing of URLs discovered elsewhere. Don’t block critical CSS/JS needed for rendering.
robots.txt example
User-agent: *
Disallow: /cart/
Allow: /
Sitemap: https://example.com/sitemap.xml