Meta-Robots-Tags »noindex« & »nofollow«

The noindex attribute is used to mark URLs that should not be indexed by Google. The nofollow attribute, on the other hand, means that the Googlebot does not follow any links on the corresponding URL.

Noindex directives are the more common of the two directives and are used to avoid duplicate content. They also clean the Google index for pages with little relevant content (thin content) or URLs that are not intended for entry via the search (imprint, data protection, etc.).

The most common implementation is the integration of the meta tag in the <head>:

<meta name=”robots” content=”noindex” />

Noindex by X-Robots-Tag

As an alternative to the <head>, a noindex can also be placed in the HTTP header using an X-Robots-Tag. This is possible, for example. B. for PDF documents without <head>. A corresponding editor in the CMS enables the setting of robot information on URL and directory level.

The information “index, follow” is set by default in the meta robots tag.
Robots tags in the <head> must be placed in the initial HTML and must not be added later using JavaScript.
For each URL, editors must be able to assign the meta robots tags noindex and nofollow independently of one another:

<meta name=”robots” content=”index, follow”>
<meta name=”robots” content=”index, nofollow”>
<meta name=”robots” content=”noindex, follow”>
<meta name=”robots” content=”noindex, nofollow”>

DEV Codierung

Link attributes: “nofollow”, “sponsored”, “ugc”

In contrast to the robots meta tag nofollow, which applies to all links on a URL, the nofollow attribute can be applied to individual links. This prevents the crawlers from following individual links on a URL.

<a href=”…” rel=”nofollow”>…</a>
<a href=”…” rel=”nofollow sponsored”>…</a>
<a href=”…” rel=”nofollow ugc”>…</a>

The nofollow attribute is usually used e.g. B. applied to special utility links such as a login or shopping cart reference as well as links from paid partnerships. The sponsored attribute is used to mark paid links. The ugc attribute distinguishes user generated content.

Editors must be able to provide individual links with each individual attribute using three checkboxes.
The attributes can be combined as required.
This option must be available separately from the function of adding the nofollow robots meta tag to individual URLs.