Hi,

No matter how big of a bug there is in Matomo, it should never be able to bring down your PHP server.
I'd recommend you to look into the php log (and also your webserver logs or php-fpm logs) to find out more about what is going wrong.

Hmm... What could it be then? I just have these lines in de log:
[Wed May 19 13:07:13.555599 2021] [authz_core:error] [pid 1007] [client xxx.xxx.xxx.xxx:58208] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/
[Wed May 19 13:07:13.567639 2021] [authz_core:error] [pid 1004] [client xxx.xxx.xxx.xxx:58212] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/empty
[Wed May 19 13:07:13.588946 2021] [authz_core:error] [pid 1006] [client xxx.xxx.xxx.xxx:58218] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/lang/en.json
[Wed May 19 13:07:21.143921 2021] [authz_core:error] [pid 1006] [client xxx.xxx.xxx.xxx:58454] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/
[Wed May 19 13:07:21.154434 2021] [authz_core:error] [pid 1005] [client xxx.xxx.xxx.xxx:58458] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/empty
[Wed May 19 13:07:21.175353 2021] [authz_core:error] [pid 1003] [client xxx.xxx.xxx.xxx:58464] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/lang/en.json

Hi,

Which web server and which PHP are you using (php-fpm?)?

If you use php-fpm, you might want to look into its error logs (/var/log/php7.4-fpm.log or similar). Maybe there are just not enough child processes to handle the incomming requests.

Apache and I tried, PHP 8.0 nginx and FPM, and PHP 7.4 FPM. I don't think there are more logs but I have to check. Child processes could be the issue but why should this always happen when I open System Check?

the system check actually tries to fetch some files/directories of Matomo via http request. If those directories are not "protected" and the requests are processed by the webserver / php this can cause some additional processes.

But then how to fix it? Is this something Matomo should fix by not running all requests at the same time?

Hi @BigBerny, the check only runs one http request at a time, which means its probably the first one that fails. I'm guessing you're using shared hosting and can't change the web server config? We could disable the individual check, if you'd like to do that, add this file /path/to/matomo/config/config.php with the following contents:

<?php

use Piwik\Plugins\Diagnostics\Diagnostic\RequiredPrivateDirectories;

return [
    'diagnostics.disabled' => array(
        DI\get(RequiredPrivateDirectories::class),        
    ),
];

Though if you're using shared hosting, it might also be helpful to talk to them about this, since Matomo can initiate HTTP requests like this in other places.

same problem here...
running it on plesk, even followed the official install guide to the line...
I can change the webserver-config, but what should I add there?

Same problem here.
When trying to access the system check page, the following happens:

2021-08-08 01:49:41	Error     403  GET /config/config.ini.php HTTP/1.0                                                                                                        
2021-08-08 01:49:41	Error     403  GET /tmp/ HTTP/1.0                                                                                                                         
2021-08-08 01:49:41	Error     403  GET /tmp/empty HTTP/1.0                                                                                                                    
2021-08-08 01:49:41	Error     403  GET /tmp/cache/tracker/matomocache_general.php HTTP/1.0                                                                                    
2021-08-08 01:49:41	Error     403  GET /lang/en.json HTTP/1.0                                                                                                                 
2021-08-08 01:49:41	Access    200  GET /index.php?module=Installation&action=systemCheckPage&idSite=1&period=day&date=yesterday&showtitle=1&random=8985 HTTP/1.0              
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/config/config.ini.php                    
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/                                     
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/empty                                
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/cache/tracker/matomocache_general.php
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/lang/en.json                             
2021-08-08 01:49:45	Access    200  GET /index.php?module=Installation&action=getEmptyPageForSystemCheck HTTP/1.0                                                              

It seems like I sometimes even get the error when opening the settings where the system check is embedded into a panel:

After that I have to wait 10 minutes until I can access the website again...

The code that goes into the config.php above did work but I think that's not the right solution (#17589 (comment)).

The child process settings are configured with ondemand (that's what Plesk shows; Plesk doesn't allow me to modify these settings), so that should not be the problem as far as I understand.

pm.max_children       5
pm.max_requests 
pm                    ondemand
pm.start_servers      1
pm.min_spare_servers  1
pm.max_spare_servers  1

The same thing also happened when opening the DSGVO/GDPR tools.

Marking this for now as a regression for us to look into it if there's anything we can do better as the system check should ideally always work.

I had the sam issue. But use php 7.4.
I missed the last step disabling ModSecurity (https://matomo.org/faq/how-to-install/how-do-i-install-matomo-with-plesk/)

And this helped? Unfortunately, I don't have access to this setting. I can ask the Hoster maybe but before it worked properly.

Yes, that had solved the problem for me.

@BigBerny Could you please check if following the FAQ above helps you? Please ensure to look carefully for the setting or asking your host. If following the FAQ does not help or there is no such setting for you and the host didn't help you either, please let us know.

I suspect, given all the above, we could have a setting that indicates plesk/modsecurity or better, disables some system checks when modsecurity is detected before it crashes and indicates that as a warning. To test this we would either need to install Apache+ModSecurity ourselves or have you test some code for us remotely.

Better yet would be of course, that we avoid this. How @Findus23 points out, Matomo should not be able to crash the webserver in any case, but I guess if there is a bug that puts it into an infinite loop it is something we will want to fix.

However, most of the above is speculative.

In the first instance, please double check the FAQ and whether this can be resolved by using the documentation or otherwise human level. There are too many different configurations we would have to write code for when it is more desirable to concentrate our efforts on high impact, new features and other prominent bugs.

I hope you understand and appreciate @BigBerny that we want to provide the best possible product in as little time as possible. If you have any further ideas or concerns, please don't hesitate to reach out again. Thanks for raising this, as it does affect people.

I checked the FAQ and the only thing which we didn't do is to disable mod_security. I asked our hoster to do that. Don't know if it's possible though since it's shared hosting.

One more thing: It didn't happen in 4.2 so there must be something new which leads to the problem.

The hoster was fast and gave me access to the setting. Unfortunately even after disbling it and restart System Check, it crashed the server and it's offline now...

When opening the admin part it says red "Required Private Directories" by the way.

The server is up again. I ran "./console core:create-security-files" from cronjobs in the meantime. And now I just have a yellow warning:

I did start full System Check again but it does crash again.

I had a conversation with my hoster ("IP-Projects" in Germany) and they told me that ModSecurity was already disabled (but they did not give me access to toggle the setting).
They also could not help me with the problem and do not have more information (they just told me that the .htaccess was not compatible with Apache 2.4 which is like the first search result in google - I hope, you devs checked that already as I'm no expert regarding .htaccess files ;)).

Btw, the Apache server does not crash (at least in my case), it's more like a temporary IP block which lasts around 10 to 15 minutes (in my case) and results in a permanent block after trying many times. I asked my hoster whether they know something about this but I'm not sure whether I will get an answer as they already told me to ask the software manufacturer for help.

My configuration should be the same as written in the FAQ except for the Cron job which should not be relevant for this problem.

Thanks @HCl-not-HCi and @BigBerny for your quick replies.

I checked and the file doing the Private Directories check was introduced between 4.2.1 and 4.3.1 which also introduces the curl command.

Could you both try running this script on your hosting and report if it causes the hanging or other problems?

https://gist.github.com/geekdenz/63c820a004c59100612c9bd81b7fc73c

Please replace the string REPLACE_WITHYOUR_DOMAIN.com with your domain and then send the output unless there are secrets you don't want to share in which case, please describe the output or obfuscate any secrets.

I suspect your host might not resolve the domain correctly locally. If that is the case, we cannot fix the cause with Matomo. However, we could add a setting to work around this problem.

Actually, just thinking about it and looking at some code, you could use a forward proxy to possibly work around this problem, if the domain is the problem:
https://matomo.org/faq/troubleshooting/faq_121/

Note however, it might have an impact on other things in the Matomo installation as well.

@geekdenz How can I run it? If I upload it to the matomo directory and access it (e.g. https://s.mydomain.com/curl.php) I get "403 Forbidden".

Edit: If I just add use echo "test"; in the script, it outputs "test" correctly.

Edit2: The error log says:
[Tue Sep 14 21:22:43.703352 2021] [authz_core:error] [pid 25389] [client XXX.XXX.XXX.XXX:45694] AH01630: client denied by server configuration: /var/www/vhosts/mydomain.com/s.mydomain.com/config/config.ini.php

@BigBerny Thanks for trying this.

403 is actually expected and not wrong in some configurations of the server. What do you mean by "it is crashing"? That you get a 403 Forbidden or that you cannot access it for a while because the request hangs?

403 means that the resource is not allowed to be accessed which can be achieved by configuring the web server and does not mean it crashed. It might mean it is configured in .htaccess files.

If it crashes, could you try accessing with your browser:

• https://s.mydomain.com/tmp/
• https://s.mydomain.com/tmp/empty
• https://s.mydomain.com/tmp/cache/tracker/matomocache_general.php
• https://s.mydomain.com/.git/
• https://s.mydomain.com/.git/config
• https://s.mydomain.com/lang/en.json

Paste the source (obtained by right-clicking in the page and selecting View Source), check if there are any secrets and obfuscate them, into something like https://gist.github.com/ and posting the links here?

If it crashes anywhere above, I think it is a problem from your hosting side. Please let us know how it goes, so we can potentially rule out a bug in Matomo. Good luck! 👍

@geekdenz For me, the script also throws a 403 (including a custom error page) as expected when accessing config.ini.php (similar to the log excerpt I pasted in #17589 (comment)).

Execept for the log entries and the 403 response, nothing else happens for me when running your script (even when holding F5 in the browser to initiate many of these requests).
But when triggering the system check in matomo, my IP gets blocked by the host (the host confirmed that they block clients who try to access forbidden content multiple times).
I tried to reproduce getting blocked without using the system check, but it's pretty random and I cannot reproduce it correctly (I also have to wait every time I was successful^^). I had most luck by spamming the /lang/en.json endpoint (i.e., holding F5 to reload the page continuously). I will try to do some more tests.

All in all, I can say that the problem is most likely a result of too many invalid requests sent by the system check which result in 403 responses. Too many 403 responses result in an IP block in my case.

Just tried it one more time in my browser:

1. Many requests to /config/config.ini.php using your script: No problem
2. Many requests to /tmp: IP block

But still no evidence as I tried 1. just before 2.

The reason I thought the server crashes was that I also can't access Plesk anymore via https://userid.hosttech.eu but I noticed I still can via https://userid.hosttech.eu:8443

It seems I'm blocked from accessing the server (also my other subdomains are affected), except Plesk over port 8443.

So it seems to be an IP block.

It could be Fail2Ban: https://docs.plesk.com/en-US/onyx/administrator-guide/server-administration/protection-against-brute-force-attacks-fail2ban.73381/
Unfortunately, I don't have access to this setting to try to disable it or whitelist my IP.

In that case I agree with @tsteur and a configuration setting to simply not do the HTTP requests suffices.

If the user has this problem at installation, they could just add the setting to config/global.ini.php.

@BigBerny and @HCl-not-HCi

Could you please download these files manually in their raw versions and copy them to the corresponding directories on your hosting:

https://github.com/matomo-org/matomo/blob/m-17589-4.4.1/config/global.ini.php
to config/global.ini.php
https://github.com/matomo-org/matomo/blob/m-17589-4.4.1/plugins/Diagnostics/Diagnostic/RequiredPrivateDirectories.php
to plugins/Diagnostics/Diagnostic/RequiredPrivateDirectories.php

Also, please change

to

disable_http_diagnostics = true

and let us know if this solved the problem for you, so we can apply it to our main branch and update the documentation.

FYI, did you try what @diosmosis pointed out above?

This issue seems to relate
#17430

Patch works 👍

I just don't know if it should be enabled by default. If all Plesk users by default are affected, many will have problems installing it.

I wonder if Plesk or whatever is causing this issue could be detected and the check disabled then.

Could you try to run this script on your server and share whether it reveals any information that could be due to Plesk or related to the issue?

Again, be careful, secrets might be revealed and only share what you believe is OK.

https://gist.github.com/geekdenz/fd8c72fc037d79f306d3a06e5e4784c4

WARNING! May reveal secrets, so be careful with sharing!

Try this instead:

<pre><?php
echo '$GLOBALS = ';
print_r($GLOBALS);

Then there's the case where the system report actually seems to crash see

this might be because of a time out configured in Apache or similar. Say max request time is configured to 30 seconds. Be great to ideally find out if this is due to such a request timeout or whether there is actually an error that isn't caught yet. It looks like we're catching any error during the HTTP request so the problem might be a timeout issue.

@tsteur The error I got there was most likely the same problem as when opening the system check, because I also got blocked and the log contained the same errors (but this error occcurs only sometimes, most likely older system check results get cached).

@geekdenz Your fix is working for me too 👍

I don't think trying to detect Plesk (if even possible) to disable the http check, is a 100% safe solution. As @BigBerny pointed out, this is probably Fail2ban which might not be enabled in all Plesk instances, and much worse, it could be installed in other environments where Plesk is not used.

Nevertheless, I tried the script above. But I don't think there is anything helpful in there:

$GLOBALS = Array
(
    [_GET] => Array
        (
        )

    [_POST] => Array
        (
        )

    [_COOKIE] => Array
        (
            [MATOMO_SESSID] => abc
        )

    [_FILES] => Array
        (
        )

    [_SERVER] => Array
        (
            [USER] => user
            [HOME] => /var/www/vhosts/example.com
            [SCRIPT_NAME] => /issue-tests/server2.php
            [REQUEST_URI] => /issue-tests/server2.php
            [QUERY_STRING] => 
            [REQUEST_METHOD] => GET
            [SERVER_PROTOCOL] => HTTP/1.0
            [GATEWAY_INTERFACE] => CGI/1.1
            [REMOTE_PORT] => 40554
            [SCRIPT_FILENAME] => /var/www/vhosts/example.com/statstest.example.de/issue-tests/server2.php
            [SERVER_ADMIN] => [no address given]
            [CONTEXT_DOCUMENT_ROOT] => /var/www/vhosts/example.com/statstest.example.de
            [CONTEXT_PREFIX] => 
            [REQUEST_SCHEME] => https
            [DOCUMENT_ROOT] => /var/www/vhosts/example.com/statstest.example.de
            [REMOTE_ADDR] => <IP address 1>
            [SERVER_PORT] => 443
            [SERVER_ADDR] => <IP address 2>
            [SERVER_NAME] => statstest.example.de
            [SERVER_SOFTWARE] => Apache
            [SERVER_SIGNATURE] => 
Apache Server at statstest.example.de Port 443


            [PATH] => /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
            [HTTP_COOKIE] => MATOMO_SESSID=abc
            [HTTP_SEC_FETCH_USER] => ?1
            [HTTP_SEC_FETCH_SITE] => none
            [HTTP_SEC_FETCH_MODE] => navigate
            [HTTP_SEC_FETCH_DEST] => document
            [HTTP_UPGRADE_INSECURE_REQUESTS] => 1
            [HTTP_DNT] => 1
            [HTTP_ACCEPT_ENCODING] => gzip, deflate, br
            [HTTP_ACCEPT_LANGUAGE] => de,en-US;q=0.7,en;q=0.3
            [HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
            [HTTP_USER_AGENT] => Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0
            [HTTP_CONNECTION] => close
            [HTTP_X_ACCEL_INTERNAL] => /internal-nginx-static-location
            [HTTP_X_REAL_IP] => <IP address 1>
            [HTTP_HOST] => statstest.example.de
            [proxy-nokeepalive] => 1
            [HTTPS] => on
            [PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY] => 0
            [PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY] => 0
            [UNIQUE_ID] => unique id
            [FCGI_ROLE] => RESPONDER
            [PHP_SELF] => /issue-tests/server2.php
            [REQUEST_TIME_FLOAT] => 1631709261.1153
            [REQUEST_TIME] => 1631709261
        )

    [_REQUEST] => Array
        (
        )

    [_ENV] => Array
        (
        )

    [GLOBALS] => Array
 *RECURSION*
)

Would it be worth creating a release 4.4.2 @tsteur ?

Also, I merged into 4.x-dev, but it is not general Diagnostic, because we know it works as is. I think it is good enough though because it resolves a corner case for users using Plesk or Fail2Ban or similar.

We should probably create an FAQ.

Would it be worth creating a release 4.4.2 @tsteur ?

Not sure I understand for what?

Next planned release will be 4.5.0. Possibly in around 2 or 2.5 weeks time

@tsteur I branched off 4.4.1 here:

#18001

So, that fix has no release. I guess we could just have users wait for the 4.5.0 release if it will happen in 2-3 weeks.

I was having this same problem. Whenever I ran a diagnostic I lost access to the website. I am running on Plesk and found in Fail2ban it had banned my ip. Turned Fail2Ban off and everything runs normally.

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/ip-address-blocked-after-opening-matomo-err-connection-timed-out/48045/2