@soratako opened this Issue on October 18th 2021

Hi!

I received an email notification from the Google Search Console. The email says "Indexed, though blocked by robots.txt".

So I checked the source code of matomo. It seems that crawl is blocked in robots.txt in addition to "robots noindex" in the head tag of the page.

According to this help page, we only need to add tags to block indexing. The help page also says "Don't use a robots.txt file as a means to hide your web pages from Google search results."

So I suggest removing the crawl blocking specification from robots.txt.

These changes were made by #16795. It seems that they changed it with the understanding that Google crawlers will not be able to crawl. This method prevents Google from crawling, but it does show up in Google search results.

@tsteur commented on October 18th 2021 Member

Hi @soratako we currently don't have any plans to change this. Is there any particular reason you would want your Matomo to be crawled by search engines? Generally it shouldn't appear in the search results anyway.

@soratako commented on October 18th 2021

@tsteur Is that true? So why? After I received the email, I searched for my domain (a subdomain for matomo like aaa.example.com) and it appeared in Google search results. However, it is only the title of the page and does not display the contents of the page. If you're right, this may be due to other causes. Anyway, I don't want it to appear in search results. Is there any other way to solve this?

@tsteur commented on October 18th 2021 Member

Not sure why it would appear @soratako The robots.txt says to not index and as you noticed the meta headers <meta name="robots" content="noindex,nofollow"> is set too.

Do you think anything else is missing? Or do you mean the robots.txt shouldn't be there so it can see the meta tag?

@soratako commented on October 19th 2021

@tsteur Hmm. As far as I read the help page, robots.txt requires the crawler not to access the site. It doesn't say "don't index the site". As a result of using robots.txt, Google crawler cannot access the page and cannot read the meta headers <meta name="robots" content="noindex,nofollow">. Therefore, it ignores the meta tag and indexes the site. If we just want the site not to appear in search results, we don't need to use robots.txt.

@soratako commented on October 19th 2021

For example, if you search for demo.matomo.cloud on Google, the matomo site will be displayed in the search results. https://www.google.com/search?q=demo.matomo.cloud This site also has robots.txt and meta tag<meta name="robots" content="noindex,nofollow">.

@tsteur commented on October 20th 2021 Member

OK thanks for this. @sgiehl do you have any thoughts there?

https://github.com/matomo-org/matomo/pull/16795 was generally added to workaround a Google Ads issue. It's not clear if removing the robots.txt could regress something or not. CC @mattab

generally the problem seems to be that robots.txt doesn't prevent you from being listed on the search as it cannot find the meta=noindex because it's blocked to request the site

@sgiehl commented on October 20th 2021 Member

I'm not able to say anything about possible site effects with Google Ads.
I guess we shouldn't remove the robots.txt completely, as it still should block access to static files that can't be "protected" otherwise.
Also if we allow access to files like matomo.js or matomo.php I doubt a domain would be removed from the search index anyway.
For content sent by our software we could also consider to send a header like X-Robots-Tag: noindex, nofollow (See https://developers.google.com/search/docs/advanced/robots/robots_meta_tag?hl=en)

Powered by GitHub Issue Mirror