How to Add a Robots txt File to Your WordPress Site for Better SEO

📌If you don’t have a robots txt file in your public_html folder, you can create one manually or use WordPress settings to generate it.

A robots.txt file is essential for managing how search engines crawl and index your WordPress site. If you’re running WordPress site, setting up this file correctly can improve SEO and control access to certain areas of your website.

In this guide, I’ll show you two easy ways to add and edit a txt file in WordPress:

Why is robots txt Important for SEO?

A robots file tells search engine crawlers which pages they can and cannot access. Here are a few reasons why it’s important:

✔ Improves Crawl Efficiency – Prevents search engines from wasting time on unnecessary pages.
✔ Protects Sensitive Areas – Restricts access to admin pages and private directories.
✔ Enhances SEO – Ensures that important content is indexed while keeping low-value pages hidden.

Method 1: Using Yoast SEO to Add Robots txt

Since WordPress doesn’t generate a robots file by default, you can create and manage one easily using Yoast SEO.

Steps to Add txt File via Yoast SEO:

1️⃣ Log in to your WordPress dashboard.
2️⃣ Navigate to SEO > Tools.
3️⃣ Click on File Editor.
4️⃣ If the robots file doesn’t exist, click Create the file.
5️⃣ Add the following rules to your robots file:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-content/uploads/
Sitemap: https://helpzone.blog/sitemap_index.xml

6️⃣ Click Save changes to robots.txt.

That’s it! Your txt file is now live.

Method 2: Robots txt File Manually Uploading via FTP

If you prefer a manual approach, you can upload the txt file directly to your server.

Steps to Upload Manually:

1️⃣ Open Notepad (Windows) or TextEdit (Mac).
2️⃣ Copy and paste the following recommended robot txt file rules:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-content/uploads/
Sitemap: https://helpzone.blog/sitemap_index.xml

3️⃣ Save the file as robots.txt.
4️⃣ Open an FTP client (such as FileZilla) and connect to your website.
5️⃣ Navigate to the public_html directory (your website’s root folder).
6️⃣ Upload the txt file.

How to Check If Robots File is Working

Once you’ve added your file txt, you can check if it’s active by visiting:

👉 https://helpzone.blog/robots.txt

If everything looks good, search engines will start following these instructions.

Final Thoughts

Adding a file is a simple but effective way to optimize your WordPress site for search engines. Whether you use Yoast SEO or manually upload it via FTP, setting it up correctly will improve crawling efficiency, SEO, and site security.

💡 Pro Tip: Keep your file updated whenever you make changes to your site’s structure or sitemap.

Have any questions? Drop a comment below! 🚀