Saturday, March 23, 2019

Custom Robots Txt Generator for Blogger

Admin | March 23, 2019
welcome to the TutBig today I will show you Custom Robots Txt Generator for Blogger Maybe you have heard about the term robots.txt. What is robots.txt? is it necessary in the settings? what if I go away it alone? There are possibly many different questions you have, especially if you are newbie blogger.


If you want to get your blog listed and crawl your pages quickly then you must add this custom robots.txt file in your blogger blog. as well as it's a section of search engine optimization so you should be aware of the terms.  

How It Works?

Robots.txt is a command for the search engine robots to explore or browse pages of our blog. Robots.txt has arguably filtered our weblog from search engines.

Let's say robotic wants to visits a Webpage URL, example, http://www.example.com/about.html. Before it does so, it will take a look at for http://www.example.com/robots.txt, and then it will get admission to the particular webpage.

All the blogs already have a robots.txt file given by means of blogger/Blogspot. By default robots.txt on blogs like this: 
User-agent: Media partners-Google
Disallow:

User-agent: *

Disallow: /search
Allow: /
Sitemap:http://yourblogname.com/atom.xml?redirect=false&start-index=1&max-results=500

What Is The Meaning Of Above Codes?

User-agent: Media partners-Google

This command tells your blog to allow Adsense bots to crawl your blog. If you’re not using Google Adsense on your blog then simply remove this line.
  

Disallow: 

This command prevents Search Engine bots from crawling pages on your blog. 

User-agent: *

All Robot Search Engines / Search engine

Disallow: / search

Not allowed to crawl the search folder, like... / search / label and ... search/search? Updated ... here label is not inserted to search because the label is not a URL who estate towards one specific page. 

Example : 
http://www.tutbig.blogspot.com/search/label/SEOhttp://www.tutbig.blogspot.com/search/SEO

Allow: /

This command tells allow all pages to be crawled, except that written on Disallow above. Mark (/) or less.

Sitemap:   

Sitemap:http://yourblogname.com/atom.xml?redirect=false&start-index=1&max-results=500

How To Prevent Robot On Certain Pages?

To prevent the particular page from Google crawling just disallow this page using Disallow command. For example: if I don't want to index mine about me page in search engines. simply I will paste the code Disallow: /p/about-me.html  right after Disallow: /search.

The code will look like this:.
User-agent:Mediapartners-GoogleDisallow:User-agent:*Disallow: /searchDisallow:/p/about-me.htmlAllow: /Sitemap:http://yourblogname.com/atom.xml?redirect=false&start-index=1&max-results=500 

How To Add Custom Robots.txt File In Blogger?

Note: Before adding a custom robots.txt file in blogger you should keep one thing in your mind that if you are using robots.txt file incorrectly then your entire blog being ignored by search engines.

Step1
Go to your Blogger Dashboard

Step2
Now go to Settings >> Search Preferences >> Crawlers and indexing >> Custom robots.txt >> Edit >> Yes
Custom Robots Txt Generator for Blogger

Step3 
Now paste your custom robots.txt file code in the box.

Step4 
Now click on the Save Changes button.

Step5 
That's all, You are done! now you can see your robot.txt file. To view the robots.txt fine, please type in the browser... 
http://yourblogname.com/robots.txt


It's okay, your blog will still be crawled by search engine robots because as I mentioned before, every blog already possessed default robots.txt. 

I hope you enjoy this post and the photos. Custom Robots Txt Generator for Blogger

Related Poste

34 comments:

  1. We are leading dynamic online solution provider as Pakistan website development company that also provide seo expert services lahore Karachi Rawalpindi

    ReplyDelete
  2. This is very helpful post and illuminating. there is no vulnerability about it.
    Contact us :- https://www.login4ites.com
    https://myseokhazana.com/

    ReplyDelete


  3. I really enjoyed your blog Thanks for sharing such an informative post.
    Website Designing & Development service In Noida

    ReplyDelete
  4. If you want to search more Robots.txt and Google updates then you can check here.

    ReplyDelete
  5. https://m3lomatop.blogspot.com/
    Make money from the Internet

    ReplyDelete
    Replies
    1. Make money from the Internet
      www.m3lomatop.blogspot.com

      Delete
  6. super.... very very super!!!
    https://www.scientificjudgment.com/2019/09/venus-planet-in-tamil.html

    ReplyDelete
  7. Awesome guide. I've added mine but which one should I use? The one of sitemap . xml or the one that shows 1-500?

    ReplyDelete
  8. Thankyou for this blog its really interesting and informative, but there is some errors
    Which need to be recorrect by owner. See i also have some good blogs related to
    Technical services, you can check on my website.

    Avast Login
    garmin.com/express
    avg.com/retail
    bullguard support number

    ReplyDelete
  9. Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here. best generator for travel trailer

    ReplyDelete
  10. Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here. best generator for travel trailer

    ReplyDelete
  11. Sir I just started a new blog:-
    Https://www.WayToidea.in/ please check it.

    ReplyDelete
  12. Thanks for sharing such beautiful information with us and I hope you will share some more information about Custom Robots Txt Generator for Blogger. Please keep sharing!

    Digital marketing
    Digital marketing company in Kochi Kerala
    Web design, development and hosting
    Software Development
    Graphic Design
    Search Engine Optimization

    ReplyDelete
  13. https://www.youtube.com/watch?v=pdXEDqxeV88

    ReplyDelete
  14. We need more such articles that we can read with such enthusiasm.
    Custom Packaging Boxes

    ReplyDelete
  15. I have to search sites with relevant information ,This is a
    wonderful blog,These type of blog keeps the users interest in
    the website, i am impressed. thank you. best eamcet college in ap

    ReplyDelete
  16. Ever been through Telugu News Portal Tolivelugu.com ?
    ToliVelugu News provide Latest Telugu Breaking News, Political News in Telugu, Telangana and AP News Headlines Live Latest Telugu News Online

    ReplyDelete
  17. You pays on line, and save time, attempt and petrol expenses. Your time is valuable.https://www.teddyway.it

    ReplyDelete
  18. Be that as it may, straightforward parts can change from undertaking to extend to guarantee the ideal answer for each is reached.Custom Packaging and Printing Company

    ReplyDelete
  19. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it. custom lapel pins

    ReplyDelete
  20. I really appreciate your method to explaining, I hope to see more posts from your blog. thank you!
    Digital Marketing Course In Kolkata
    Web Design Course In Kolkata
    SEO Course In Kolkata

    ReplyDelete
  21. simultaneous localization and mapping SLAM We have made great strides when it comes to robotics. But where we've come at a standstill is the lack of support to the robots when it comes to finding the location.patents on simultaneous localization and mapping self driving car

    ReplyDelete
  22. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Digital Marketing Training in Chennai
    Digital Marketing Training in Bangalore
    Digital Marketing Training in Delhi
    Digital Marketing Online Training

    ReplyDelete
  23. When your website or blog goes live for the first time, it is exciting. That is until you realize no one but you and your. rapidgator premium link generator

    ReplyDelete
  24. Your work is great and I value you and jumping for some more educational posts. Much obliged to you for sharing incredible data to us. Custom Boxes with logo

    ReplyDelete