@DaAwesomeP
I have newly installed Matomo via docker with FPM and Nginx. I was able to log in after setting it up and everything worked fine, but upon attempting to login the next day I see this error:
And in the Nginx logs I see:
FastCGI sent in stderr: "PHP message: Error in Matomo: Session must be started before any output has been sent to the browser; output started in php://input/1" while reading upstream, client: <REDACTED>, server: <REDACTED>, request: "POST /index.php?module=Login HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "<REDACTED>", referrer: "https://<REDACTED>/index.php"If I check the "Remember Me" box and login, then I see:
And in Nginx:
FastCGI sent in stderr: "PHP message: Error in Matomo: You must call Zend_Session::regenerateId() before any output has been sent to the browser; output started in php://input/1" while reading upstream, client: <REDACTED>, server: <REDACTED>, request: "POST /index.php?module=Login HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "<REDACTED>", referrer: "https://<REDACTED>/index.php"This is the latest Matomo v4.1.0. Matomo/PHP/FPM and MariaDB are in Docker, Nginx is running directly on Debian 10.
Hmmm restarting the docker container seemed to have resolved it. Strange.
@morganrallen
If your situation is like mine, it will happen again. I seem to run into this issue ever couple of weeks. Sure, restarting the docker container gets it going again, but it's not a fix.
Yup, you're exactly right. Also, a new issue I have been having that is also solved by restarting it and probably related: if I am already logged in, then the main page just doesn't load entirely and is blank. It loads the navbar and left side, but not the main widgets. Settings page works fine.
yup, this matches what I am experiencing.
Some additional feedback. In the 'post logged in state' where the navigation portions of the page load, but the data/content portions don't, there is also JS errors being thrown on the console.
Possibly unhandled rejection: null 3 index.php:224:202
e https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:224
get https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:195
g https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
$digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
$apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
v https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
onload https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
(Async: EventHandlerNonNull)
Pg https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
s https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:204
b https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
$digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
$apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
v https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
onload https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
(Async: EventHandlerNonNull)
Pg https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
s https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:204
b https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
$digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
$apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
c https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
invoke https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:140
c https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
Wc https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
Ee https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:116
<anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:446
i https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
fireWith https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
ready https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
J https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
(Async: EventListener.handleEvent)
promise https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
<anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
<anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
<anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
In addition, when trying to log back in from this error state, a URL fragment is injected into the page, just after the BODY opening tag.
form_login=morgan&form_nonce=xxxxxxxxxxxxxxxxxx&form_redirect=https%3A%2F%xxxxxxxxxxxx%2Findex.php%3Fmodule%3DCoreHome%26action%3D%26period%3Dday%26date%3Dyesterday&form_password=xxxxxxxxxxxxxxxxxx It does this weather Remember is check or not
This is happening very frequently. At this point I SSH in every time I need to access Matomo because I will probably need to restart it to access it or at some point while accessing it. Are there further steps we can take to debug?
So... I appear to have fixed my instance while chasing down a different issue, it's a bit of a stretch but worth looking into.
Long story short, we kept being infected by the kinsing malware due to a misconfiguration of our docker-compose file. Are any of the following true....
- you have
kinsingorkdevtmpfsi(note thei,kdevtmpfsis a real process) processes running on the system? - you have port
9000exposed on your host system ( < this was our mistake) - you have your docker daemon port exposed on the host (2375 I think)
my understanding is the kinsing malware deletes a bunch of temp files on start and this could be causing session corruption. Again, it's a stretch but it seems to be working here.
Holy cow well this is a big deal. The only ports we have open are 22 (SSH w/ fail2ban & the Mozilla security spec) and 443 (Nginx running outside of Docker, fully patched). All other ports blocked with UFW. The good thing is that the malware appears to be contained within the Matomo docker container. I am suspicious that the container may have shipped with it, but that seems unlikely.
Aha. UFW + Docker was the problem. Luckily this server was not in production yet, so I can wipe it and start over to be sure.
So this appears to be the culprit?
UFW + Docker was the problem
Would be helpful for other if you elaborate on this a bit more.
Both Docker and UFW use iptables to manage traffic. However, Docker does not follow the rules set by UFW. So closing ports with UFW won't actually affect Docker. There isn't really an official solution to this problem. There are some workarounds, but they all have their issues (no internet in containers, no IPv6, etc.). My solution is to ensure that nothing is listening on 0.0.0.0 from Docker, but this is annoying nonetheless.
did find evidence of the malware?
Yup, it was contained in the Matomo container (Docker did its job), but I am going to wipe the machine anyway.