What are googlebots ?

What are googlebots?

Defination :

“Googlebot is the search bot software used by Google, which collects documents from the web to build a searchable index for the Google Search engine.”

If a webmaster wishes to restrict the information on their site available to a Googlebot, or another well-behaved spider, they can do so with the appropriate directives in a robots.txt file, or by adding the meta tag <meta name="Googlebot" content="nofollow"/ to the web page.

Note: Googlebot only follows HREF “Hypertext Reference” links–which indicates the URL being linked to–and SRC “Source” links. With a list of webpage URLs, Googlebot will use Web-crawling robots to collect information to build a searchable index for Google’s Indexer.

The function of Googlebot :

Googlebot functions as a search bot to crawl content on a site and interpret the contents of a user’s created robots.txt file (e.g., www.myhost.com/robots.txt). The searchable bots (robots) work by reading Web pages; then, they make the content of the pages available to all Google services (done by Google’s caching proxy).

Note: Googlebot’s requests to Web servers are done by a user-agent string containing “Googlebot,” and requests to a host address contain “googlebot.com.”

Benefits: It is used to find Web pages on the Internet. Search robots will access any file in the root directory and all its subdirectories. Of course, users can set it up to allow or disallow the robots.txt file to Control Search Engine Spiders–a program that travels the Web–to be able to retrieve every page from a Web site.

Pros and Cons of Googlebot :

Pros:

• It can quickly build a list of links that come from the Web.
• It recrawls popular frequently-changing web pages to keep the index current.

Cons:

• It only follows HREFlinks and SRC links.
• It takes up an enormous amount of bandwidth.
• Some pages may take longer to find, so crawling may occur once a month vice daily
• It must be setup/programmed to function properly.