Google ヘルプ - Google Crawler (User Agent) Overview

Google Crawler (User Agent) Overview

Low-quality directory or bookmark site links• Meta tags can be used to provide information to all sorts of clients, and each system processes only the meta tags they understand and ignores the rest. Widely distributed links in the footers or templates of various sites• html AdsBot Checks desktop. Last updated 2021-11-22 UTC. 0 Linux; Android 7. The full user agent string is a full description of the crawler, and appears in the request and your web logs. Google Search interprets delayed meta refresh redirects as temporary redirects. google. section For example: This text can be included in a snippet and this part would not be shown. google. When users search for your site, Google Search results sometimes display a search box specific to your site, along with other direct links to your site. If you want all of Google to be able to crawl your pages, you don't need a robots. example. 1025. All methods described here have about the same response time. Note that if you have large numbers of URLs,. Send feedback Except as otherwise noted, the content of this page is licensed under the , and code samples are licensed under the. You can use this tag on the top-level page of your site to. Using automated programs or services to create links to your site. Are you using a hosted content management platform for example, Blogger or Wordpress? - Paul As long as you don't pass to the buyer of the ad, your pay-per-click PPC advertising links won't violate our guidelines. In some situations, this description is used in the. User agents in robots. 4044. Google can read both HTML and XHTML-style meta tags, regardless of the code used on the page. When Google recognizes that the contents of a page aren't in the language that the user likely wants to read, Google may provide a in search results. Googlebot Full user agent string The Googlebot-News user agent string was. google. Redirecting the links to an intermediate page that is blocked from search engines with a robots. Last updated 2022-02-28 UTC. Prevents various from reading aloud web pages using text-to-speech TTS. デベロッパー• To submit a URL to the index:• You're merging two websites and want to make sure that links to outdated URLs are redirected to the correct pages. User agent token AdsBot-Google-Mobile-Apps Full user agent string AdsBot-Google-Mobile-Apps Caution:. One of the URLs will be the ; which one, depends on signals such as whether the redirect was temporary or permanent. This tag sends the user to a new URL after a certain amount of time, and is sometimes used as a simple form of redirection. To ensure machine readability, the section must be. This table lists information about the common Google crawlers you may see in your referrer logs, and how to specify them in , the , and the. 2623. Googlebot-News• example. Other points to note• In the case of conflicting robots or googlebot meta tags, the more restrictive tag applies. Apache: Consult the , the , and the. Googlebot-Image• Check your platform's support articles. Creating good content pays off: Links are usually editorial votes given by choice, and the more useful content you have, the greater the chances someone else will find that content valuable to their readers and link to it. Googlebot-Image• NGINX: Read about on the NGINX blog. Obeys robots rules. txt file, Google will follow the most specific. General guidelines• For example: There are many on the market. Exchanging money for links, or posts that contain links• Send feedback Except as otherwise noted, the content of this page is licensed under the , and code samples are licensed under the. 2272. Alternate versions of a URL When you redirect a URL, Google keeps track of both the redirect source the old URL and the redirect target the new URL. We'll use your information to improve our algorithmic detection of such links. In general, this gives you the chance to provide your unique and compelling content to a much larger group of users. Google Search interprets instant meta refresh redirects as permanent redirects. People access your site through several different URLs. The following table shows the crawlers used by various products and services at Google:• Redirects are particularly useful in the following circumstances:• Googlebot-Video• html User agent token Mediapartners-Google Full user agent string Mediapartners-Google AdsBot Mobile Web Android Checks Android. Send feedback Except as otherwise noted, the content of this page is licensed under the , and code samples are licensed under the. If the tool finds issues with the page, try to fix them. For details, see the. Presence of this tag indicates to Google that the page is mobile friendly. You removed a page and you want to send users to a new page. The user agent token is used in the User-agent: line in robots. google. This will also ensure that Google keeps the old URL in its results for a longer time. This helps users find your new site and Google may understand this as a crypto redirect. You can create both permanent and temporary redirects on the server side. Keyword-rich, hidden or low-quality links embedded in widgets that are distributed across various sites, for example: Visitors to this page: 1,472 car insurance• To set up a permanent redirect with PHP, use the header function. Use the to request a crawl of individual URLs. 170816. Send documentation feedback Except as otherwise noted, the content of this page is licensed under the , and code samples are licensed under the. html Mobile Apps Android Checks Android app page. Google uses algorithms to determine the optimal crawl rate for each site. If the user clicks the translated title link, all further user interaction with the page is through Google Translate, which will automatically translate any links followed. If you're using a platform like Blogger or Shopify, the platform may already have built-in redirect solutions. For example: We moved! Buying or selling links that pass. Z in the user agent strings in the table, W. Submit a sitemap many URLs at once A sitemap is an important way for Google to discover URLs on your site. example. A sitemap can also include additional metadata about alternate language versions, video, image, or news-specific pages. Page-level meta tags are a great way for website owners to provide search engines with information about their sites. 3071. The following are examples of link schemes which can negatively impact a site's ranking in search results:• The other URL becomes an alternate name of the canonical URL. As with Apache, you have multiple choices to create redirects. Additionally, creating links that weren't editorially placed or vouched for by the site's owner on a page, otherwise known as unnatural links, can be considered a violation of our guidelines. Meta tags are added to the section of your HTML page and generally look like this: Example Books - high-quality used books for children Google understands the following meta tags: Meta tags that Google understands Use this tag to provide a short description of the page. The tag applies to all search engines, while the tag is specific to Google. Crypto redirect Learn more about. You can't request indexing for URLs that you don't manage. html Caution: User agent token Google-Read-Aloud Full user agent strings Current agents:• For details, see the. This meta tag tells Google that you don't want us to provide a translation for this page. User agent token• Link schemes Any links intended to manipulate or a site's ranking in Google search results may be considered part of a link scheme and a violation of Google's. Instant meta refresh redirect: Triggers as soon as the page is loaded in a browser. txt to match a crawler type when writing crawl rules for your site. We recommend using a server-side 301 redirect instead. example. This version number will increase over time to. But if you want more fine-grained control, you can get more specific. google. Inspect the URL using the. Sending someone a "free" product in exchange for them writing it and including a link• Try searching for help articles for example, search for "blogger redirects". Here are a few common examples of unnatural links that may violate our guidelines:• For details, see the. Indicating the link is sponsored adding a to the tag• This includes any behavior that manipulates links to your site or outgoing links from your site. txt file The best way to get other sites to create high-quality, relevant links to yours is to create unique, relevant content that can naturally gain popularity in the Internet community. Our systems prioritize the fast inclusion of high quality, useful content. Google's main crawler is called. google. Be patient and monitor progress using either the or the. example. google. The Google Search Central Twitter is the best place to get the latest updates on bugs and announcements. htaccess file on Apache or setting the redirect headers with server side scripts for example, PHP. The default values are index, follow and don't need to be specified. If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number. Z is actually a placeholder that represents the version of the Chrome browser used by that user agent: for example, 41. Find the content on our new site! Redirects and Google Search Redirecting URLs is the practice of resolving an existing URL to a different one, effectively telling your visitors and Google Search that a page has a new location. Permanent redirects: Show the new redirect target in search results. The 301 and 308 status codes mean that a page has permanently moved to a new location. html Googlebot Image User agent tokens• Don't rely on crypto redirects for letting search engines know that your content has moved unless you have no other choice. Crawl request methods Use the URL Inspection tool just a few URLs Note: You must be an to be able to request indexing in the URL Inspection tool. google. location. Requiring a link as part of a Terms of Service, contract, or similar arrangement without allowing a third-party content owner the choice , should they wish. Temporary redirects: Show the source page in search results. Contact your hosting provider for help with traditional redirects before resorting to crypto redirects. 0 Googlebot News User agent tokens• Don't rely on crypto redirects for letting search engines know that your content has moved unless you have no other choice. Forum comments with optimized links in the post or signature, for example: Thanks, that's great info! Permanent server side redirects If you need to change the URL of a page as it is shown in search engine results, we recommend that you use a permanent server side redirect whenever possible. 0 Linux; Android 6. 19 Google StoreBot User agent token Storebot-Google Full user agent strings• 2272. You will also need to and a. Follow the. span• For details, see the. Temporary server side redirects If you just want to send users to a different page temporarily, use a temporary redirect. Exchanging goods or services for links• 0 Linux; Android 4. div• If you want to block or allow all of Google's crawlers from accessing some of your content, you can do this by specifying Googlebot as the user agent. html AdsBot Mobile Web Checks iPhone. For a full list of values that Google understands, see the. In most cases, the platform submits your new content to search engines automatically. For details, see the. Some crawlers have more than one token, as shown in the table; you need to match only one crawler token for a rule to apply. The following table explains the various ways you can use to set up permanent and temporary redirects, ordered by how likely Google is able to interpret correctly for example, a server side redirect has the highest chance of being interpreted correctly by Google. For all other web servers, check with your server manager or hoster, or search for guides on your favorite search engine for example, search for "LiteSpeed redirects". However, there may be situations where this is not desired. Except as otherwise noted, the content of this page is licensed under the , and code samples are licensed under the. Text advertisements that pass• Controlling crawl speed Each Google crawler accesses sites for a specific purpose and at different rates. Last updated 2022-02-28 UTC. Large-scale article marketing or guest posting campaigns with keyword-rich anchor text links. Links with optimized anchor text in articles or press releases distributed on other sites. Requesting a recrawl multiple times for the same URL or sitemap won't get it crawled any faster. Overview of redirect types While your users generally won't be able to tell the difference between the different types of redirects, Google Search uses redirects as a strong or weak signal that the redirect target should be canonical. This defines the page's content type and character set. The tool will run a live test on the URL to see whether it has any obvious indexing issues, and if not, the page will be queued for indexing. The simplest way to do this is to add a link pointing to the new page accompanied by a short explanation. Caution: The user agent string can be spoofed. While Google attempts to render every URL Googlebot crawled, rendering may fail for various reasons. Similarly, if you want to block some pages from Google altogether, blocking the Googlebot user agent will also block all Google's other user agents. For example, you might want all your pages to appear in Google Search, but you don't want images in your personal directory to be crawled. txt file at all. ビジネス• You can use other meta tags if they are important to your site, but Google will ignore meta tags that it doesn't know. html• Make sure that you surround the value of the content attribute with quotes - otherwise the charset attribute may be interpreted incorrectly. Googlebot follows the redirect, and the indexing pipeline uses the redirect as a weak signal that the redirect target should be canonical. For example: window. You can prevent from passing in several ways, such as:• The W3C. You've moved your site to a new domain, and you want to make the transition as seamless as possible. google. This is particularly useful if you wish to limit indexing of non-HTML files like graphics or other kinds of documents. Learn more about. Labels a page as containing adult content, to signal that it be filtered by SafeSearch results. For example, if you want all your pages to appear in Google Search, and if you want AdSense ads to appear on your pages, you don't need a robots. Select Request indexing. 3945. Inline directives Independently of page-level meta-tags, you can exclude parts of an HTML page from snippets. However, it is not supported by all browsers and can be confusing to the user. com , it's a good idea to pick one of those URLs as your preferred destination, and use redirects to send traffic from the other URLs to your preferred URL. Alternate names are different versions of a canonical URL that users might recognize and trust more. txt Where several user agents are recognized in the robots. Choosing a redirect depends on how long you expect the redirect will be in place and what page you want Google Search to show in search results:• 36 Caution: For user-initiated requests, Google Favicon ignores robots. For example, if a service your site offers is temporarily unavailable, you can set up a temporary redirect to send users to a page that explains what's happening, without compromising the original URL in search results. This tag tells Google not to show the sitelinks search box. Choose the redirect type that works for your situation and site: Redirect types Permanent Googlebot follows the redirect, and the indexing pipeline uses the redirect as a strong signal that the redirect target should be. Video User agent tokens• txt rules. This is the best way to ensure that Google Search and people are directed to the correct page. For example, if a page has both the max-snippet:50 and nosnippet tags, the nosnippet tag will apply. google. Delayed meta refresh redirect: Triggers only after an arbitrary number of seconds set by the site owner. 0 Linux; Android 8. Alternate names may appear in search results when a user's query hints that they might trust the old URL more. If you want to have a , you will have to pick the. If you see a site that is participating in link schemes intended to manipulate ,. 4240. This tag tells the browser how to render a page on a mobile device. Crawling can take anywhere from a few days to a few weeks. If a Google crawler is crawling your site too often, you can. Please note that while the values of the name and content attributes must match exactly what is provided to you including upper and lower case , it doesn't matter if you change the tag from XHTML to HTML or if the format of the tag matches the format of your page. Last updated 2022-02-28 UTC. 0 Desktop User agent token Googlebot Full user agent strings• Send feedback Except as otherwise noted, the content of this page is licensed under the , and code samples are licensed under the. In this case, use robots. Requesting a crawl does not guarantee that inclusion in search results will happen instantly or even at all. This is not an exclusive list of meta tags. Follow your web server's guides:• google. Last updated 2022-02-28 UTC. 36 Google Favicon Caution:. Implement server side redirects The implementation of server side redirects depends on your hosting and server environment, or the scripting language of your site's backend. This is normal and as users get used to the new domain name, the alternate names will fade away without you doing anything.。