Create robots.txt and an XML sitemap
Help search engines find every page — and make sure you're not blocking them by mistake.

A sitemap lists the pages you want in search results. robots.txt tells crawlers where they may go and where your sitemap is.
A simple robots.txt
Create robots.txt in public_html with cPanel File Manager:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.co.ke/sitemap.xmlUse your own domain and sitemap address. Don't block CSS or JavaScript files — Google needs them to see the page properly.
Create a sitemap
- WordPress: WordPress makes one automatically at
/wp-sitemap.xml. SEO plugins such as Yoast replace it with/sitemap_index.xml. - Other sites: most CMSs have a sitemap option; for small static sites, list each page URL in an XML file.
Put your sitemap address in robots.txt as shown above.
Submit to Google
- Open Google Search Console and add your domain (verify with a DNS TXT record in cPanel Zone Editor).
- Go to Sitemaps, enter the sitemap address and click Submit.
- Check back after a few days for pages that couldn't be indexed.
Blocked by mistake?
Disallow: /in robots.txt blocks the whole site.- A
<meta name="robots" content="noindex">tag hides a page. In WordPress, untick Settings → Reading → Discourage search engines from indexing this site — it's often left on after a site is built.
Was this article helpful?
Your vote helps us write better guides.
