Dosto kya aap jante hai ki humare blog me robots.txt file hoti hai jo search crawler ko batati hai ki humare blog ke konse area me enter karke search engine me index karna hai. Jab hum post publish karte hai to ye robots.txt file har post me generate ho jati hai.
Table of Contents
Robots.txt kya hai?
Jaise ki aap apne blog me se search, archive etc ko nahi chahte ki search engine me index ho to inko aap disallow karke search engine crawler ko search engine me show karne se rok sakte hai.
Robots.txt file kaisi hoti hai?
# Blogger Sitemap generated on 2016.03.18
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://yourblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
User-agent: *
Robots.txt file me iska matlab hai ki ye section sabhi robots spiders ya crawler ke liye allow hai.
Disallow: /search
Iska matlab hai ki ki /search root disallow hai search engine spiders isko crawl nhi karega kyunki apne isko disallow kar diya hai.
Allow: /
Robots spiders ke liye ye “/” allow kar diya hai iska matlab aap ke blog me “/” is root ke sabhi url ko search engine index karega.
jaise ki maine koi post likhi hai uska url kuchh is tarah se hai “https://www.techaruby.com/2016/03/blogger-Important-settings.html” to spiders isko search engine me ndex karegi kyunki aap ne Allow: / kiya hai.
Sitemap: https://www.techaruby.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://yourblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500 Sitemap: http://yourblog.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000
Disallow Pages, posts
Agar hum koi specific post ya page robots ke dwara search engine me index hone se rokna chahte hai to us post ya page ko robot.txt file Disallow kar denge.
Disallow: /2015/06/https://www.techaruby.com/2016/02/blogging.html
Jab koi particular post index hone se rokna hai to aapko robots.txt file me upar diye gaye code ke anusar “Disallow: /year/month/post url” add karna hai.
Disallow: /p/about.html
Agar apko koi particular page index hone se rokana hai to aap robot.txt file me “Disallow: /page url without domain name” add karna hai. jaise mene Upar dikhaye gye code ke anusar apne blog me “About us” page ko index hone se disallow kiya hua hai.
Blogger Me custom Robots.txt kaise use karte hai?
Generate xml sitemap
2.Sitemap generate hone ke baad pure site map ko all select karke copy karle.
3. Apne blogger dashbord me jaye aur settings pe click kare.
4. Search preference pe click kare.
5. Custom robot.txt ko edit kare aur yes select kare aur clipboard me copy kiya hua code yahan click karde.
6.Save changes kare.
7.check kare ki custom robot.txt file submit hui ki nahi iske liye apne web browser me apni website url (https://www.techaruby.com/robots.txt) ke bad robots.txt add karke enter kare.
Aap payenge ki apne jo code custom robots.txt me add kiya hai browser me show ho raha hai,
Congratulations apne apne blog me custom robot.txt file acche se submit kardi hai joki apke blog ke SEO (search engine optimization) ke liye bahut important hai.
NYC and very useful information Thanks