@DaAwesomeP opened this Issue on December 24th 2020

I have newly installed Matomo via docker with FPM and Nginx. I was able to log in after setting it up and everything worked fine, but upon attempting to login the next day I see this error:
image
And in the Nginx logs I see:

FastCGI sent in stderr: "PHP message: Error in Matomo: Session must be started before any output has been sent to the browser; output started in php://input/1" while reading upstream, client: <REDACTED>, server: <REDACTED>, request: "POST /index.php?module=Login HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "<REDACTED>", referrer: "https://<REDACTED>/index.php"

If I check the "Remember Me" box and login, then I see:
image
And in Nginx:

FastCGI sent in stderr: "PHP message: Error in Matomo: You must call Zend_Session::regenerateId() before any output has been sent to the browser; output started in php://input/1" while reading upstream, client: <REDACTED>, server: <REDACTED>, request: "POST /index.php?module=Login HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "<REDACTED>", referrer: "https://<REDACTED>/index.php"

This is the latest Matomo v4.1.0. Matomo/PHP/FPM and MariaDB are in Docker, Nginx is running directly on Debian 10.

@DaAwesomeP commented on December 24th 2020

Hmmm restarting the docker container seemed to have resolved it. Strange.

@morganrallen commented on January 5th 2021

If your situation is like mine, it will happen again. I seem to run into this issue ever couple of weeks. Sure, restarting the docker container gets it going again, but it's not a fix.

@DaAwesomeP commented on January 5th 2021

Yup, you're exactly right. Also, a new issue I have been having that is also solved by restarting it and probably related: if I am already logged in, then the main page just doesn't load entirely and is blank. It loads the navbar and left side, but not the main widgets. Settings page works fine.

@morganrallen commented on January 5th 2021

yup, this matches what I am experiencing.

@morganrallen commented on January 5th 2021

Some additional feedback. In the 'post logged in state' where the navigation portions of the page load, but the data/content portions don't, there is also JS errors being thrown on the console.

Possibly unhandled rejection: null 3 index.php:224:202
    e https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:224
    get https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:195
    g https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
    $digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
    $apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
    k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
    v https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    onload https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    (Async: EventHandlerNonNull)
    Pg https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    s https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:204
    b https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
    k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
    $digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
    $apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
    k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
    v https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    onload https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    (Async: EventHandlerNonNull)
    Pg https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    s https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:204
    b https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
    k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
    $digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
    $apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
    c https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
    invoke https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:140
    c https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
    Wc https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
    Ee https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:116
    <anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:446
    i https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    fireWith https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    ready https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    J https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    (Async: EventListener.handleEvent)
    promise https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    <anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    <anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    <anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4

In addition, when trying to log back in from this error state, a URL fragment is injected into the page, just after the BODY opening tag.

form_login=morgan&form_nonce=xxxxxxxxxxxxxxxxxx&form_redirect=https%3A%2F%xxxxxxxxxxxx%2Findex.php%3Fmodule%3DCoreHome%26action%3D%26period%3Dday%26date%3Dyesterday&form_password=xxxxxxxxxxxxxxxxxx 

It does this weather Remember is check or not

@DaAwesomeP commented on January 8th 2021

This is happening very frequently. At this point I SSH in every time I need to access Matomo because I will probably need to restart it to access it or at some point while accessing it. Are there further steps we can take to debug?

@morganrallen commented on January 8th 2021

So... I appear to have fixed my instance while chasing down a different issue, it's a bit of a stretch but worth looking into.
Long story short, we kept being infected by the kinsing malware due to a misconfiguration of our docker-compose file. Are any of the following true....

  • you have kinsing or kdevtmpfsi (note the i, kdevtmpfs is a real process) processes running on the system?
  • you have port 9000 exposed on your host system ( < this was our mistake)
  • you have your docker daemon port exposed on the host (2375 I think)

my understanding is the kinsing malware deletes a bunch of temp files on start and this could be causing session corruption. Again, it's a stretch but it seems to be working here.

@DaAwesomeP commented on January 8th 2021

Holy cow well this is a big deal. The only ports we have open are 22 (SSH w/ fail2ban & the Mozilla security spec) and 443 (Nginx running outside of Docker, fully patched). All other ports blocked with UFW. The good thing is that the malware appears to be contained within the Matomo docker container. I am suspicious that the container may have shipped with it, but that seems unlikely.

@DaAwesomeP commented on January 8th 2021

Aha. UFW + Docker was the problem. Luckily this server was not in production yet, so I can wipe it and start over to be sure.

@morganrallen commented on January 8th 2021

So this appears to be the culprit?

UFW + Docker was the problem

Would be helpful for other if you elaborate on this a bit more.

@DaAwesomeP commented on January 8th 2021

Both Docker and UFW use iptables to manage traffic. However, Docker does not follow the rules set by UFW. So closing ports with UFW won't actually affect Docker. There isn't really an official solution to this problem. There are some workarounds, but they all have their issues (no internet in containers, no IPv6, etc.). My solution is to ensure that nothing is listening on 0.0.0.0 from Docker, but this is annoying nonetheless.

@morganrallen commented on January 8th 2021

did find evidence of the malware?

@DaAwesomeP commented on January 8th 2021

Yup, it was contained in the Matomo container (Docker did its job), but I am going to wipe the machine anyway.

This Issue was closed on January 9th 2021
Powered by GitHub Issue Mirror