How to Add a robots.txt File to Your Site?

🤖 What is a robots.txt file?

The robots.txt file tells search engine crawlers how to interact with your site. Specifically, it:

• Defines which URLs are allowed to be crawled.

• Defines which URLs are not allowed to be crawled.

• Lives in the root directory of your domain (e.g., https://www.asp.events/robots.txt).

Note: The robots.txt file does not stop pages from being indexed—meta tags control indexing.

🌟 Why have a robots.txt file?

• It is the first file a crawler checks on your website.

• Guides search engines to efficiently crawl your site.

• Excludes sections of your site that you don’t want crawled (like themes or system folders).

• Can point crawlers to your XML sitemap for easier indexing.

🛠 Example Default in Showoff

• User-agent: * → Applies to all crawlers.

• Disallow: /themes/ → Do not crawl the themes folder.

• Disallow: /sites/ → Do not crawl the sites folder.

• Sitemap: ... → Points crawlers to the sitemap with all indexable pages.

✏️ How to Edit robots.txt in Showoff

1. Login to your Showoff back office.

2. Navigate to Settings > SEO Settings.

3. Click the Site robots.txt button (top-right).

4. Edit the file in the robots.txt editor.

5. Click Save Changes to publish.

Warning: Be careful when editing. Incorrect entries can block search engines from your site and impact SEO.