Robots.txt kya hai aur blogger me custom robots.txt kaise use karte hai?

यह पोस्ट हाल ही में अपडेट की गयी है : June 10th, 2020

Dosto kya aap jante hai ki humare blog me robots.txt file hoti hai jo search crawler ko batati hai ki humare blog ke konse area me enter karke search engine me index karna hai. Jab hum post publish karte hai to ye robots.txt file har post me generate ho jati hai.

Robots.txt kya hai?

Robots.txt ek simple txt file hoti hai jo search engine spiders ko ye permission deti hai ki hamare blog mese search engine me kya index karna hai aur kya nahi.

Jaise ki aap apne blog me se search, archive etc ko nahi chahte ki search engine me index

ho to inko aap disallow karke search engine crawler ko search engine me show karne se rok sakte hai.

Robots.txt file kaisi hoti hai?

# Blogger Sitemap generated on 2016.03.18
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://yourblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

User-agent: *

Robots.txt file me iska matlab hai ki ye section sabhi robots spiders ya crawler ke liye allow hai.

Disallow: /search

Iska matlab hai ki ki /search root disallow hai search engine spiders isko crawl nhi karega kyunki apne isko disallow kar diya hai.

Allow: /

Robots spiders ke liye ye “/” allow kar diya hai iska matlab aap ke blog me “/” is root ke sabhi url ko search engine index karega.

jaise ki maine koi post likhi hai uska url kuchh is tarah se hai “http://www.techaruby.com/2016/03/blogger-Important-settings.html” to

spiders isko search engine me ndex karegi kyunki aap ne Allow: / kiya hai.

Sitemap:

Sitemap: http://www.techaruby.com/atom.xml?redirect=false&start-index=1&max-results=500

Iska matlab hai ki Apne blog ki sabhi new posts search engine spiders index karega jo is path se.

By default robots 25 posts ko index karta hai aap iski madad se per sitemap 500 posts set kar sakte hai.

Jab apke blog me 500 se jyada posts hoti hai to apko ek aur sitemap add karna jaruri hota hai. Jaise ki 1000 post ke liye sitemap is tarah se hoga.

Sitemap: http://yourblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://yourblog.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000

Disallow Pages, posts

Agar hum koi specific post ya page robots ke dwara search engine me index hone se rokna chahte hai to us post ya page ko robot.txt file Disallow kar denge.

Example:

Disallow: /2015/06/http://www.techaruby.com/2016/02/blogging.html

Jab koi

particular post index hone se rokna hai to aapko robots.txt file me upar diye gaye code ke anusar “Disallow: /year/month/post url” add karna hai.

 Disallow: /p/about.html

Agar apko koi particular page index hone se rokana hai to aap robot.txt file me “Disallow: /page url without domain name” add karna hai. jaise mene Upar dikhaye gye code ke anusar apne blog me “About us” page ko index hone se disallow kiya hua hai.

Blogger Me custom Robots.txt kaise use karte hai?

1.Sabse Pahle aap ko xml sitemap generate karana hai, niche diye gaye link pe click karke site map generate kare.

Generate xml sitemap
2.Sitemap generate hone ke baad pure site map ko all select karke copy karle.

3. Apne blogger dashbord me jaye aur settings pe click kare.

4. Search preference pe click kare.

5. Custom robot.txt ko edit kare aur yes select kare aur clipboard me copy kiya hua code yahan click karde.

6.Save changes kare.

7.check kare ki custom robot.txt file submit hui ki nahi iske liye apne web browser me apni website url (http://www.techaruby.com/robots.txt) ke bad robots.txt add karke enter kare.

Aap payenge ki apne jo code custom robots.txt me add kiya hai browser me show ho raha hai,

Congratulations apne apne blog me custom robot.txt file acche se submit kardi hai joki apke blog ke SEO (search engine optimization) ke liye bahut important hai.

Final words:

Robot.txt file apne blog ke seo friendly banane ke liye bahut hi jaruri hai, Iske dwara hum apne blog ko search engine me submit karke web crawler ya spiders ko control kar sakte hai ki kya search engine me dikhana hai aur kya nahi. par robot.txt file bahut sensitive hoti hai iska wrong use karne pe search engine hamare blog ko index karne se ignore bhi kar sakta hai.

Mujhe ummid hai ki aapko robots.txt kya hota hai aur kaise use karte hai samajh aa gya hoga aur ye bhi samjh aaya hoga ki ye normal si txt file kitna bada kam karti hai aur hamare blog ke liye kitni importance rakhti hai. Agar is post se related koi sawal ho ya kuchh samajh nahi aaya ho to aap jarur comments se puchhe.