I received an email notification from the Google Search Console. The email says "Indexed, though blocked by robots.txt".
So I checked the source code of matomo. It seems that crawl is blocked in robots.txt in addition to "robots noindex" in the head tag of the page.
According to this help page, we only need to add tags to block indexing. The help page also says "Don't use a robots.txt file as a means to hide your web pages from Google search results."
So I suggest removing the crawl blocking specification from robots.txt.
These changes were made by #16795. It seems that they changed it with the understanding that Google crawlers will not be able to crawl. This method prevents Google from crawling, but it does show up in Google search results.
Hi @soratako we currently don't have any plans to change this. Is there any particular reason you would want your Matomo to be crawled by search engines? Generally it shouldn't appear in the search results anyway.
@tsteur Is that true? So why? After I received the email, I searched for my domain (a subdomain for matomo like aaa.example.com) and it appeared in Google search results. However, it is only the title of the page and does not display the contents of the page. If you're right, this may be due to other causes. Anyway, I don't want it to appear in search results. Is there any other way to solve this?
Not sure why it would appear @soratako The robots.txt says to not index and as you noticed the meta headers
<meta name="robots" content="noindex,nofollow"> is set too.
Do you think anything else is missing? Or do you mean the
robots.txt shouldn't be there so it can see the
@tsteur Hmm. As far as I read the help page, robots.txt requires the crawler not to access the site. It doesn't say "don't index the site". As a result of using robots.txt, Google crawler cannot access the page and cannot read the meta headers
<meta name="robots" content="noindex,nofollow">. Therefore, it ignores the meta tag and indexes the site. If we just want the site not to appear in search results, we don't need to use robots.txt.
For example, if you search for
demo.matomo.cloud on Google, the matomo site will be displayed in the search results. https://www.google.com/search?q=demo.matomo.cloud This site also has
robots.txt and meta tag
<meta name="robots" content="noindex,nofollow">.
OK thanks for this. @sgiehl do you have any thoughts there?
generally the problem seems to be that robots.txt doesn't prevent you from being listed on the search as it cannot find the meta=noindex because it's blocked to request the site
I'm not able to say anything about possible site effects with Google Ads.
I guess we shouldn't remove the robots.txt completely, as it still should block access to static files that can't be "protected" otherwise.
Also if we allow access to files like
matomo.php I doubt a domain would be removed from the search index anyway.
For content sent by our software we could also consider to send a header like
X-Robots-Tag: noindex, nofollow (See https://developers.google.com/search/docs/advanced/robots/robots_meta_tag?hl=en)
I have the same issue, did anyone try removing the robots.txt?