New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Suggestion: Unblock crawls by robots.txt #18175
Comments
Hi @soratako we currently don't have any plans to change this. Is there any particular reason you would want your Matomo to be crawled by search engines? Generally it shouldn't appear in the search results anyway. |
@tsteur Is that true? So why? After I received the email, I searched for my domain (a subdomain for matomo like aaa.example.com) and it appeared in Google search results. However, it is only the title of the page and does not display the contents of the page. If you're right, this may be due to other causes. Anyway, I don't want it to appear in search results. Is there any other way to solve this? |
Not sure why it would appear @soratako The robots.txt says to not index and as you noticed the meta headers Do you think anything else is missing? Or do you mean the |
@tsteur Hmm. As far as I read the help page, robots.txt requires the crawler not to access the site. It doesn't say "don't index the site". As a result of using robots.txt, Google crawler cannot access the page and cannot read the meta headers |
For example, if you search for |
OK thanks for this. @sgiehl do you have any thoughts there? #16795 was generally added to workaround a Google Ads issue. It's not clear if removing the robots.txt could regress something or not. CC @mattab generally the problem seems to be that robots.txt doesn't prevent you from being listed on the search as it cannot find the meta=noindex because it's blocked to request the site |
I'm not able to say anything about possible site effects with Google Ads. |
I have the same issue, did anyone try removing the robots.txt? |
please excuse me for ressurecting such an old topic, but i have just recently noticed the mamoto comes with robots.txt that allows indexing mamoto.php and some js files why? shouldn't it be just this?
|
Matomo can be configured to track bots as well. By default the tracker will discard requests from bots, though. |
oh ok! thank you for the explanation |
Hi!
I received an email notification from the Google Search Console. The email says "Indexed, though blocked by robots.txt".
So I checked the source code of matomo. It seems that crawl is blocked in robots.txt in addition to "robots noindex" in the head tag of the page.
According to this help page, we only need to add tags to block indexing. The help page also says "Don't use a robots.txt file as a means to hide your web pages from Google search results."
So I suggest removing the crawl blocking specification from robots.txt.
These changes were made by #16795. It seems that they changed it with the understanding that Google crawlers will not be able to crawl. This method prevents Google from crawling, but it does show up in Google search results.
The text was updated successfully, but these errors were encountered: