Create robots.txt and an XML sitemap

Help search engines find every page — and make sure you're not blocking them by mistake.

Updated 25 October 20262 min read1 view0 found this helpful
Applies toWordPress logocPanel logo
On this page

A sitemap lists the pages you want in search results. robots.txt tells crawlers where they may go and where your sitemap is.

A simple robots.txt

Create robots.txt in public_html with cPanel File Manager:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.co.ke/sitemap.xml

Use your own domain and sitemap address. Don't block CSS or JavaScript files — Google needs them to see the page properly.

Create a sitemap

  • WordPress: WordPress makes one automatically at /wp-sitemap.xml. SEO plugins such as Yoast replace it with /sitemap_index.xml.
  • Other sites: most CMSs have a sitemap option; for small static sites, list each page URL in an XML file.

Put your sitemap address in robots.txt as shown above.

Submit to Google

  1. Open Google Search Console and add your domain (verify with a DNS TXT record in cPanel Zone Editor).
  2. Go to Sitemaps, enter the sitemap address and click Submit.
  3. Check back after a few days for pages that couldn't be indexed.

Blocked by mistake?

  • Disallow: / in robots.txt blocks the whole site.
  • A <meta name="robots" content="noindex"> tag hides a page. In WordPress, untick Settings → Reading → Discourage search engines from indexing this site — it's often left on after a site is built.
#SEO#robots.txt#sitemap#Search Console

Was this article helpful?

Your vote helps us write better guides.

Keep reading

Related articles

Still stuck?

Talk to our support team

Open a ticket

Best for anything technical. Every reply is tracked.

Submit a ticket

Call our Nairobi team

+254 726 872 512 · +254 722 917 455

Call now

Message us

Send a question and we'll reply by email or phone.

Contact support
Launch today

Powering Your Online Success!

Domain, hosting, email and SSL from one local team — set up in minutes and backed by people you can call.

  • Free SSL
  • Free migration
  • 99.9% uptime
  • 24/7 expert support

Start with your domain

Check availability, then add hosting in the same order.

Prefer to talk? Call +254 726 872 512