Robots.txt File kya Hai: Ise Kaise Add Kare? (Complete Guide 2020)

Robots.txt SEO ke liye bahut hi important hai. Robots.txt search engine ko ye command deta hai ki kon sa page ko show karna hai yak on se page ko block karna hai.

  • Robots.txt file kya hai.
  • Robots.txt file ko use kaise kare.
  • Robots.txt test kaise kare.
  • Aur Robot.txt ke aur kya kya fayade hai.

in sab ke bare me iss article me describe se batane ja raha hu.

Isse ache use karna aana chahiye nahi to ye apke website pe bad effect bhi kar sakta.

Iss article me hum apko step by step sare tips bataye hai.

Aur apko koi problem aaye ya koi step samaj na aaye to comment box me jarur bataye.

 

 Robots.txt File kya hai (What is Robos.txt File)

Robots txt ek Text File hai jiska Extension .txt hota hai.

Robots txt ke help se aap kisi bhi page ya file ko search engine me block kar sakte hai. Audience uss page ya file ko dekh sakte hai per search engine usse ignore karta hai.

Website me aise bahut se “important or useless page” hote hai, jisko block karna aur allow karna bahut jaruri hai.

Agar aap aisa nahi karge to apke website me bahut sare page aur post duplicate ho jayege.

Duplicate page hone se apke website ka ranking bhi down ho sakta hai.

Iss article ko padh ke aap WordPress, Blogger, ya kisi bhi platform me apka website ho aap sare platform me robots txt bana sakte hai.

 

 Robots.txt ko Test kaise kare (How to Test Robots.txt)

Robots.txt test karna bahut hi aasan hai. aap kisi bhi website ki robots txt test kar sakte hai.

Step 1. Browser me koi bhi website ka link open kare.

https://www.yoursite.com/

Step 2. Ab link ke last me robots.txt type kare.

robots.txt link check

Step 3. Ab Enter press kare.

robots txt file check

 

 Robots.txt kam kaise karti hai

Jab Search engine ke Spider (Google or Bing Bots) kisi website pe aati hai page or post ko index karne ke liye, to wo sab se pahale Robots txt ko check karti hai.

Robots txt me jo page Disallow hai usse wo Index nahi karti jiske karan wo page search engine me kabhi show nahi hoga.

Aur jis page me allow likha hota hai wahi search engine me index ho pata hai

 

 Robots.txt ke kay kya fayade hai (Feature of Robots File)

  1. Robots File se aap apne website ko Secure kar sakte ho
  2. Robots File me Sitemap add kar ke website ko kaam time me Search Engine me index kar sakte ho
  3. Duplicate Content page ko Noindex kar sakte like (Category, Author page, Tags, Comment)
  4. Kisi bhi secure page ko Search Engine me hide kar sakte like (index.php)

 

 Website me kon se page block karna chahiye

Apke website me aise bhut se page hai jinko “Search Engine” se hide karna jaruri hai.

For example

  • Category
  • Achieve
  • Tags
  • Comment feed
  • Affiliate links
  • php file
  • Wp-admin
  • Wp-include

Ye sare page ko Noindex karna bahut jaruri hai.

Agar apka website WordPress pe hai to apko ye sare code block karne ki jarurat hai.

Iss article me bataya gaya hai aap Yoast plugin ke help se “Achieves, Author, Tags aur Category” ko Noindex kar sakte hai.

 

 Robots.txt ke kon kon se Term hai

Robot txt syntax

ye sare Robots txt ke Term hai

User-agent: * = iska matlab apke website me jitne bhi file hai (all page, CSS, JavaScript) sab access ho jayega

Allow = Allow likhne se aapka wo page search engine me show hoga, Google ke bots usko crawlers karege.

Disallow = Disallow se apka page block ho jayega. Search engine usko crawler nahi karega

Siteamp = Robots.txt me sitemap ka link add karna bahut jaruri hai. isse apke website ke page aur post jald hi search me index hoge.

 

 Robots.txt Create kaise kare (How to Create Robots.txt)

Robots txt create karna bahut hi aasan hai.

Agar apka website WordPress pe hai

Agar apka site WordPress pe hai to isse create karna bahut hi aasan hai. apko cpanel me jane ki koi jarurat nahi.

Apko step by step samjte hai.

Dashboard → SEO → Tools → File Editor → Create robots.txt file

Step #1. Apne WordPress me Yoast plugin install kare.

Yoast SEO plugin

Step #2. Left side me SEO hai uske TOOLS pe click kare

Yoast Robots txt code

 

Step #3. “File Editor” option pe click kare.

Robots txt yoast

Step #4. Yaha pe niche diye hue code ko copy kar ke yaha add kar de

Yoast robots file edit

User-agent: *
Disallow: /wp-admin/

Disallow: /cg-bin/
Disallow: /comment/feed/
Disallow: /trackback/
Disallow: /xmlrpd.php

Sitemap: http://seoandinfo/sitemap_index.xml

Note: sitemap me mere website ka link hai usko change kar le.


Blogsopt me robots.txt kaise add kare (How to create custom robots.txt file in blogspot?)

Step #1. Sab se pahale Blogsopt Dashboard all posts open kare.

Step #2. Left side me Blogspot ke setting pe click kare.

Step #3. Iske baad Search preference pe click kare.

Step #4. Yaha apko “custom robots txt” milega usko Edit kare.

Step #5. Niche diye hue code ko custom robot.txt ke box me add kar de aur save kar de.

 

User-agent: Mediapartness-Google
User-agent: *

Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /

Sitemap: https://seoandinfo/sitemap.xml

Note: sitemap me maine apne website ka link add kiya hai usko change kar le.

Agar apka website blogspot pe hai to robots.txt create karne ki koi jarurat nahi.

Google Blogsopt automatically apke website ke liye robots file create kar dega.


Agar apka website customized hai

Agar aka website WordPress pe nahi hai to isse create karna thora long process hai.

Step #1. Sab se pahale Notepad open kare aur usme niche bataye ho code ko likhe aur iss file ko robots txt se save kar le.

User-agent: *
Disallow: /index.php/

Step #1. Sab se pahale apne Hosting me Sign in kare

Step #2. Iske baad aap apna server cpanel open kare

Step #3. Uske baad File mabager open kare

Step #4. Yaha apka website ke folder ko open kare, jaha apke website ke sare code ho

Step #5. Iske baad rorbots.txt ko upload kare jo apne suru me banaya tha.

 

 Robots.txt me Sitemap kaise add kare (How to add Sitemap in Robots.txt)

Apne robots txt me sitemap ko jarur add kare.

Three type of sitemap

  • Page and post sitemap
  • Image sitemap
  • Video sitemap

Apke site me jitney bhi site map hai sab ko add robots txt me ad kare.

 

 Google Webmaster me Robots txt file Test kaise kare?

Google webmaster se app free me robots txt test kar sakte hai.

Step #1. Apne Webmaster me Sign in kare.

Step #2. uske baad Left side me Crawl pe click kare.

Step #3. apko robots.txt Tester mil jayega uspe click kare.

Step #4. waha apko RED color me TEST button milega uspe click kare.

Agar apke website me robots file sahi se kam kar raha hai to OK show ho jayega

Yaha aap Goolge Webmaster ke robots txt ke bare me information le sakte hai

 

Robots.txt Important Notes in Hindi

Question: Robots txt ka result kitne dino me show hota hai.

Answer: Robots txt add hone ke baad uska result 2 ya 3 din ke baad pata chalega.


Question: Robots txt file ka use nahi karne pe kya hoga.

Answer: Search engine ke Bots apke website ke sare page ko index kar dekhe, jiske karan apka website hack bhi ho sakta hai. isliye kuch page ko Hide karna jaruri hai.


Question: User-agent: * isme star a matlab kya hai?
Answer: star ‘*‘ ka matlab hai website ke sare page ko search engine ka crawlers index karayega.


Question: Robots txt file me Disallow ka matlab kya hai.
Answer: Disallow ka matlab wo page Search Engine me Index nahi hoga.
For Example: Disallow: /index.php/

 Conclusion

Agar ye article apko acha laga to isse jarur share kare aur iss website ko apne browser me bookmark kar le, kyu ki aise hi new Blogging or Social Media Tips share karte rahege.

Agar apke robots.txt me koi problem aa rahi hai to comment box me jarur bataye.

8 Comments

Add a Comment

Your email address will not be published. Required fields are marked *

1 Shares