@BigBerny opened this Issue on May 19th 2021

When opening the Admin area since 4.3 it shows a few more issues like "Required Private Directories" and a new database issue (can't see it right now, since server is down). When I open System Check to find out more about it, the page doesn't load and the whole server is not available anymore (ERR_CONNECTION_REFUSED). I can't even access Plesk.

It's reproducible and didn't happen in 4.2. Which log might be useful to find out what's going wrong?

Expected Behavior

Opening System Check should open it

Current Behavior

Opening System check crashes server.

Steps to Reproduce (for Bugs)

Open System check

Your Environment

Can't access it right now but what I know:

  • Matomo Version: 4.3
  • PHP Version: 8.0
  • Server Operating System:
  • Additionally installed plugins:
@Findus23 commented on May 19th 2021 Member

Hi,

No matter how big of a bug there is in Matomo, it should never be able to bring down your PHP server.
I'd recommend you to look into the php log (and also your webserver logs or php-fpm logs) to find out more about what is going wrong.

@BigBerny commented on May 19th 2021

Hmm... What could it be then? I just have these lines in de log:
[Wed May 19 13:07:13.555599 2021] [authz_core:error] [pid 1007] [client xxx.xxx.xxx.xxx:58208] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/
[Wed May 19 13:07:13.567639 2021] [authz_core:error] [pid 1004] [client xxx.xxx.xxx.xxx:58212] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/empty
[Wed May 19 13:07:13.588946 2021] [authz_core:error] [pid 1006] [client xxx.xxx.xxx.xxx:58218] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/lang/en.json
[Wed May 19 13:07:21.143921 2021] [authz_core:error] [pid 1006] [client xxx.xxx.xxx.xxx:58454] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/
[Wed May 19 13:07:21.154434 2021] [authz_core:error] [pid 1005] [client xxx.xxx.xxx.xxx:58458] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/empty
[Wed May 19 13:07:21.175353 2021] [authz_core:error] [pid 1003] [client xxx.xxx.xxx.xxx:58464] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/lang/en.json

@Findus23 commented on May 19th 2021 Member

Hi,

Which web server and which PHP are you using (php-fpm?)?

If you use php-fpm, you might want to look into its error logs (/var/log/php7.4-fpm.log or similar). Maybe there are just not enough child processes to handle the incomming requests.

@BigBerny commented on May 19th 2021

Apache and I tried, PHP 8.0 nginx and FPM, and PHP 7.4 FPM. I don't think there are more logs but I have to check. Child processes could be the issue but why should this always happen when I open System Check?

@sgiehl commented on May 19th 2021 Member

the system check actually tries to fetch some files/directories of Matomo via http request. If those directories are not "protected" and the requests are processed by the webserver / php this can cause some additional processes.

@BigBerny commented on May 19th 2021

But then how to fix it? Is this something Matomo should fix by not running all requests at the same time?

@diosmosis commented on May 19th 2021 Member

Hi @BigBerny, the check only runs one http request at a time, which means its probably the first one that fails. I'm guessing you're using shared hosting and can't change the web server config? We could disable the individual check, if you'd like to do that, add this file /path/to/matomo/config/config.php with the following contents:

<?php

use Piwik\Plugins\Diagnostics\Diagnostic\RequiredPrivateDirectories;

return [
    'diagnostics.disabled' => array(
        DI\get(RequiredPrivateDirectories::class),        
    ),
];

Though if you're using shared hosting, it might also be helpful to talk to them about this, since Matomo can initiate HTTP requests like this in other places.

@patermatzka commented on July 27th 2021

same problem here...
running it on plesk, even followed the official install guide to the line...
I can change the webserver-config, but what should I add there?

@HCl-not-HCi commented on August 8th 2021

Same problem here.
When trying to access the system check page, the following happens:

2021-08-08 01:49:41 Error     403  GET /config/config.ini.php HTTP/1.0                                                                                                        
2021-08-08 01:49:41 Error     403  GET /tmp/ HTTP/1.0                                                                                                                         
2021-08-08 01:49:41 Error     403  GET /tmp/empty HTTP/1.0                                                                                                                    
2021-08-08 01:49:41 Error     403  GET /tmp/cache/tracker/matomocache_general.php HTTP/1.0                                                                                    
2021-08-08 01:49:41 Error     403  GET /lang/en.json HTTP/1.0                                                                                                                 
2021-08-08 01:49:41 Access    200  GET /index.php?module=Installation&action=systemCheckPage&idSite=1&period=day&date=yesterday&showtitle=1&random=8985 HTTP/1.0              
2021-08-08 01:49:41 Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/config/config.ini.php                    
2021-08-08 01:49:41 Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/                                     
2021-08-08 01:49:41 Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/empty                                
2021-08-08 01:49:41 Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/cache/tracker/matomocache_general.php
2021-08-08 01:49:41 Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/lang/en.json                             
2021-08-08 01:49:45 Access    200  GET /index.php?module=Installation&action=getEmptyPageForSystemCheck HTTP/1.0                                                              

It seems like I sometimes even get the error when opening the settings where the system check is embedded into a panel:

system check error

After that I have to wait 10 minutes until I can access the website again...

The code that goes into the config.php above did work but I think that's not the right solution (https://github.com/matomo-org/matomo/issues/17589#issuecomment-844217389).

The child process settings are configured with ondemand (that's what Plesk shows; Plesk doesn't allow me to modify these settings), so that should not be the problem as far as I understand.

pm.max_children       5
pm.max_requests 
pm                    ondemand
pm.start_servers      1
pm.min_spare_servers  1
pm.max_spare_servers  1

The same thing also happened when opening the DSGVO/GDPR tools.

@tsteur commented on August 8th 2021 Member

Marking this for now as a regression for us to look into it if there's anything we can do better as the system check should ideally always work.

@wi-wissen commented on August 19th 2021

I had the sam issue. But use php 7.4.
I missed the last step disabling ModSecurity (https://matomo.org/faq/how-to-install/how-do-i-install-matomo-with-plesk/)

@BigBerny commented on August 20th 2021

And this helped? Unfortunately, I don't have access to this setting. I can ask the Hoster maybe but before it worked properly.

@wi-wissen commented on August 23rd 2021

Yes, that had solved the problem for me.

@geekdenz commented on September 12th 2021 Contributor

@BigBerny Could you please check if following the FAQ above helps you? Please ensure to look carefully for the setting or asking your host. If following the FAQ does not help or there is no such setting for you and the host didn't help you either, please let us know.

I suspect, given all the above, we could have a setting that indicates plesk/modsecurity or better, disables some system checks when modsecurity is detected before it crashes and indicates that as a warning. To test this we would either need to install Apache+ModSecurity ourselves or have you test some code for us remotely.

Better yet would be of course, that we avoid this. How @Findus23 points out, Matomo should not be able to crash the webserver in any case, but I guess if there is a bug that puts it into an infinite loop it is something we will want to fix.

However, most of the above is speculative.

In the first instance, please double check the FAQ and whether this can be resolved by using the documentation or otherwise human level. There are too many different configurations we would have to write code for when it is more desirable to concentrate our efforts on high impact, new features and other prominent bugs.

I hope you understand and appreciate @BigBerny that we want to provide the best possible product in as little time as possible. If you have any further ideas or concerns, please don't hesitate to reach out again. Thanks for raising this, as it does affect people.

@BigBerny commented on September 13th 2021

I checked the FAQ and the only thing which we didn't do is to disable mod_security. I asked our hoster to do that. Don't know if it's possible though since it's shared hosting.

One more thing: It didn't happen in 4.2 so there must be something new which leads to the problem.

@BigBerny commented on September 13th 2021

The hoster was fast and gave me access to the setting. Unfortunately even after disbling it and restart System Check, it crashed the server and it's offline now...

When opening the admin part it says red "Required Private Directories" by the way.

@BigBerny commented on September 13th 2021

The server is up again. I ran "./console core:create-security-files" from cronjobs in the meantime. And now I just have a yellow warning:
image

I did start full System Check again but it does crash again.

@HCl-not-HCi commented on September 13th 2021

I had a conversation with my hoster ("IP-Projects" in Germany) and they told me that ModSecurity was already disabled (but they did not give me access to toggle the setting).
They also could not help me with the problem and do not have more information (they just told me that the .htaccess was not compatible with Apache 2.4 which is like the first search result in google - I hope, you devs checked that already as I'm no expert regarding .htaccess files ;)).

Btw, the Apache server does not crash (at least in my case), it's more like a temporary IP block which lasts around 10 to 15 minutes (in my case) and results in a permanent block after trying many times. I asked my hoster whether they know something about this but I'm not sure whether I will get an answer as they already told me to ask the software manufacturer for help.

My configuration should be the same as written in the FAQ except for the Cron job which should not be relevant for this problem.

@geekdenz commented on September 13th 2021 Contributor

Thanks @HCl-not-HCi and @BigBerny for your quick replies.

I checked and the file doing the Private Directories check was introduced between 4.2.1 and 4.3.1 which also introduces the curl command.

Could you both try running this script on your hosting and report if it causes the hanging or other problems?

https://gist.github.com/geekdenz/63c820a004c59100612c9bd81b7fc73c

Please replace the string REPLACE_WITHYOUR_DOMAIN.com with your domain and then send the output unless there are secrets you don't want to share in which case, please describe the output or obfuscate any secrets.

I suspect your host might not resolve the domain correctly locally. If that is the case, we cannot fix the cause with Matomo. However, we could add a setting to work around this problem.

@geekdenz commented on September 13th 2021 Contributor

Actually, just thinking about it and looking at some code, you could use a forward proxy to possibly work around this problem, if the domain is the problem:
https://matomo.org/faq/troubleshooting/faq_121/

Note however, it might have an impact on other things in the Matomo installation as well.

@BigBerny commented on September 14th 2021

@geekdenz How can I run it? If I upload it to the matomo directory and access it (e.g. https://s.mydomain.com/curl.php) I get "403 Forbidden".

Edit: If I just add use echo "test"; in the script, it outputs "test" correctly.

Edit2: The error log says:
[Tue Sep 14 21:22:43.703352 2021] [authz_core:error] [pid 25389] [client XXX.XXX.XXX.XXX:45694] AH01630: client denied by server configuration: /var/www/vhosts/mydomain.com/s.mydomain.com/config/config.ini.php

@geekdenz commented on September 14th 2021 Contributor

@BigBerny Thanks for trying this.

403 is actually expected and not wrong in some configurations of the server. What do you mean by "it is crashing"? That you get a 403 Forbidden or that you cannot access it for a while because the request hangs?

403 means that the resource is not allowed to be accessed which can be achieved by configuring the web server and does not mean it crashed. It might mean it is configured in .htaccess files.

If it crashes, could you try accessing with your browser:

Paste the source (obtained by right-clicking in the page and selecting View Source), check if there are any secrets and obfuscate them, into something like https://gist.github.com/ and posting the links here?

If it crashes anywhere above, I think it is a problem from your hosting side. Please let us know how it goes, so we can potentially rule out a bug in Matomo. Good luck! 👍

@HCl-not-HCi commented on September 14th 2021

@geekdenz For me, the script also throws a 403 (including a custom error page) as expected when accessing config.ini.php (similar to the log excerpt I pasted in https://github.com/matomo-org/matomo/issues/17589#issuecomment-894723948).

Execept for the log entries and the 403 response, nothing else happens for me when running your script (even when holding F5 in the browser to initiate many of these requests).
But when triggering the system check in matomo, my IP gets blocked by the host (the host confirmed that they block clients who try to access forbidden content multiple times).
I tried to reproduce getting blocked without using the system check, but it's pretty random and I cannot reproduce it correctly (I also have to wait every time I was successful^^). I had most luck by spamming the /lang/en.json endpoint (i.e., holding F5 to reload the page continuously). I will try to do some more tests.

All in all, I can say that the problem is most likely a result of too many invalid requests sent by the system check which result in 403 responses. Too many 403 responses result in an IP block in my case.

@HCl-not-HCi commented on September 14th 2021

Just tried it one more time in my browser:

  1. Many requests to /config/config.ini.php using your script: No problem
  2. Many requests to /tmp: IP block

But still no evidence as I tried 1. just before 2.

@geekdenz commented on September 14th 2021 Contributor

Thanks @HCl-not-HCi for your feedback.

So, the IP block. Does it make your browser look like it is looping, i.e. taking a long time to load? That would indicate that Matomo will hang as well for a while for you because the host is blocking your browser, not because Matomo crashed. Also, if your IP is temporarily blocked, I believe it means it is not a Matomo issue that is causing it. However, it does affect users like yourself. So, I suggest you give me your exact Matomo version (e.g. 4.3.1) and I create a patch and documentation for that and I'll merge it into our main branch if it works for you.

I will create the files downloadable for you and then you'll need to upload them using an FTP client or similar.

Hope this helps.

@HCl-not-HCi commented on September 14th 2021

I can't tell you the exact message at the moment because I'm on my phone right now. But as I remember, the browser tries to load the page for some time and then times out. This not only happens to matomo but for all services on the webspace for me (it's definitely a block by the hosting provider, not a crash).

But I cannot tell what happens during the system check - whether it's just the number of requests or if there is another error e.g., in the .htaccess file leading to the IP block. All I have is the log output in one of my previous posts.

Thank you, I'm happy to help. The version does not really matter as long it's >= 4.3. On my test instance, I currently have 4.4.1 installed.

@tsteur commented on September 14th 2021 Member

fyi sounds like we're maybe dealing with different issues here.

Btw, the Apache server does not crash (at least in my case), it's more like a temporary IP block which lasts around 10 to 15 minutes (in my case) and results in a permanent block after trying many times.
But when triggering the system check in matomo, my IP gets blocked by the host (the host confirmed that they block clients who try to access forbidden content multiple times).

For this the solution would be either to change server configuration, or to disable that specific system check. We could add a config setting for this @geekdenz

Then there's the case where the system report actually seems to crash see
image

this might be because of a time out configured in Apache or similar. Say max request time is configured to 30 seconds. Be great to ideally find out if this is due to such a request timeout or whether there is actually an error that isn't caught yet. It looks like we're catching any error during the HTTP request so the problem might be a timeout issue.

@BigBerny commented on September 14th 2021

The curl script didn't make the server "crash" but System Check does. Actually, I'm not sure if actually crashes but it starts loading System Check (only

, empty page, DevTools say "Failed to load resource: net::ERR_HTTP2_PING_FAILED), timesout and when I try to access Matomo again I get "(ERR_CONNECTION_REFUSED)".

After checking again, I think you're right. It seems to be an IP ban. I can still access Matomo on my phone (over LTE, different IP).

Can test the other URLs tomorrow if needed.

@BigBerny commented on September 14th 2021

The reason I thought the server crashes was that I also can't access Plesk anymore via https://userid.hosttech.eu but I noticed I still can via https://userid.hosttech.eu:8443

It seems I'm blocked from accessing the server (also my other subdomains are affected), except Plesk over port 8443.

So it seems to be an IP block.

@BigBerny commented on September 14th 2021

It could be Fail2Ban: https://docs.plesk.com/en-US/onyx/administrator-guide/server-administration/protection-against-brute-force-attacks-fail2ban.73381/
Unfortunately, I don't have access to this setting to try to disable it or whitelist my IP.

@geekdenz commented on September 15th 2021 Contributor

In that case I agree with @tsteur and a configuration setting to simply not do the HTTP requests suffices.

@geekdenz commented on September 15th 2021 Contributor

If the user has this problem at installation, they could just add the setting to config/global.ini.php.

@geekdenz commented on September 15th 2021 Contributor

@BigBerny and @HCl-not-HCi

Could you please download these files manually in their raw versions and copy them to the corresponding directories on your hosting:

https://github.com/matomo-org/matomo/blob/m-17589-4.4.1/config/global.ini.php
to config/global.ini.php
https://github.com/matomo-org/matomo/blob/m-17589-4.4.1/plugins/Diagnostics/Diagnostic/RequiredPrivateDirectories.php
to plugins/Diagnostics/Diagnostic/RequiredPrivateDirectories.php

Also, please change
https://github.com/matomo-org/matomo/blob/45eec5ac60ed35e6d345a48e22ca9ce6df7cd7df/config/global.ini.php#L822

to

disable_http_diagnostics = true

and let us know if this solved the problem for you, so we can apply it to our main branch and update the documentation.

@geekdenz commented on September 15th 2021 Contributor

FYI, did you try what @diosmosis pointed out above?

This issue seems to relate
https://github.com/matomo-org/matomo/pull/17430

@BigBerny commented on September 15th 2021

Patch works 👍

@BigBerny commented on September 15th 2021

I just don't know if it should be enabled by default. If all Plesk users by default are affected, many will have problems installing it.

@geekdenz commented on September 15th 2021 Contributor

I wonder if Plesk or whatever is causing this issue could be detected and the check disabled then.

Could you try to run this script on your server and share whether it reveals any information that could be due to Plesk or related to the issue?

Again, be careful, secrets might be revealed and only share what you believe is OK.

https://gist.github.com/geekdenz/fd8c72fc037d79f306d3a06e5e4784c4

@geekdenz commented on September 15th 2021 Contributor

WARNING! May reveal secrets, so be careful with sharing!

Try this instead:

<pre><?php
echo '$GLOBALS = ';
print_r($GLOBALS);
@HCl-not-HCi commented on September 15th 2021

Then there's the case where the system report actually seems to crash see
image

this might be because of a time out configured in Apache or similar. Say max request time is configured to 30 seconds. Be great to ideally find out if this is due to such a request timeout or whether there is actually an error that isn't caught yet. It looks like we're catching any error during the HTTP request so the problem might be a timeout issue.

@tsteur The error I got there was most likely the same problem as when opening the system check, because I also got blocked and the log contained the same errors (but this error occcurs only sometimes, most likely older system check results get cached).

@geekdenz Your fix is working for me too 👍

I don't think trying to detect Plesk (if even possible) to disable the http check, is a 100% safe solution. As @BigBerny pointed out, this is probably Fail2ban which might not be enabled in all Plesk instances, and much worse, it could be installed in other environments where Plesk is not used.

Nevertheless, I tried the script above. But I don't think there is anything helpful in there:

Output (some parts changed because of secrets) ``` $GLOBALS = Array ( [_GET] => Array ( ) [_POST] => Array ( ) [_COOKIE] => Array ( [MATOMO_SESSID] => abc ) [_FILES] => Array ( ) [_SERVER] => Array ( [USER] => user [HOME] => /var/www/vhosts/example.com [SCRIPT_NAME] => /issue-tests/server2.php [REQUEST_URI] => /issue-tests/server2.php [QUERY_STRING] => [REQUEST_METHOD] => GET [SERVER_PROTOCOL] => HTTP/1.0 [GATEWAY_INTERFACE] => CGI/1.1 [REMOTE_PORT] => 40554 [SCRIPT_FILENAME] => /var/www/vhosts/example.com/statstest.example.de/issue-tests/server2.php [SERVER_ADMIN] => [no address given] [CONTEXT_DOCUMENT_ROOT] => /var/www/vhosts/example.com/statstest.example.de [CONTEXT_PREFIX] => [REQUEST_SCHEME] => https [DOCUMENT_ROOT] => /var/www/vhosts/example.com/statstest.example.de [REMOTE_ADDR] => [SERVER_PORT] => 443 [SERVER_ADDR] => [SERVER_NAME] => statstest.example.de [SERVER_SOFTWARE] => Apache [SERVER_SIGNATURE] => Apache Server at statstest.example.de Port 443 [PATH] => /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [HTTP_COOKIE] => MATOMO_SESSID=abc [HTTP_SEC_FETCH_USER] => ?1 [HTTP_SEC_FETCH_SITE] => none [HTTP_SEC_FETCH_MODE] => navigate [HTTP_SEC_FETCH_DEST] => document [HTTP_UPGRADE_INSECURE_REQUESTS] => 1 [HTTP_DNT] => 1 [HTTP_ACCEPT_ENCODING] => gzip, deflate, br [HTTP_ACCEPT_LANGUAGE] => de,en-US;q=0.7,en;q=0.3 [HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 [HTTP_USER_AGENT] => Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0 [HTTP_CONNECTION] => close [HTTP_X_ACCEL_INTERNAL] => /internal-nginx-static-location [HTTP_X_REAL_IP] => [HTTP_HOST] => statstest.example.de [proxy-nokeepalive] => 1 [HTTPS] => on [PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY] => 0 [PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY] => 0 [UNIQUE_ID] => unique id [FCGI_ROLE] => RESPONDER [PHP_SELF] => /issue-tests/server2.php [REQUEST_TIME_FLOAT] => 1631709261.1153 [REQUEST_TIME] => 1631709261 ) [_REQUEST] => Array ( ) [_ENV] => Array ( ) [GLOBALS] => Array *RECURSION* ) ```
@geekdenz commented on September 15th 2021 Contributor

Would it be worth creating a release 4.4.2 @tsteur ?

Also, I merged into 4.x-dev, but it is not general Diagnostic, because we know it works as is. I think it is good enough though because it resolves a corner case for users using Plesk or Fail2Ban or similar.

We should probably create an FAQ.

@tsteur commented on September 15th 2021 Member

Would it be worth creating a release 4.4.2 @tsteur ?

Not sure I understand for what?

Next planned release will be 4.5.0. Possibly in around 2 or 2.5 weeks time

@geekdenz commented on September 16th 2021 Contributor

@tsteur I branched off 4.4.1 here:

https://github.com/matomo-org/matomo/pull/18001

So, that fix has no release. I guess we could just have users wait for the 4.5.0 release if it will happen in 2-3 weeks.

Powered by GitHub Issue Mirror