Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot log in second time after fresh install v4.1.0 #17019

Closed
DaAwesomeP opened this issue Dec 24, 2020 · 14 comments
Closed

Cannot log in second time after fresh install v4.1.0 #17019

DaAwesomeP opened this issue Dec 24, 2020 · 14 comments
Labels
answered For when a question was asked and we referred to forum or answered it.

Comments

@DaAwesomeP
Copy link

I have newly installed Matomo via docker with FPM and Nginx. I was able to log in after setting it up and everything worked fine, but upon attempting to login the next day I see this error:
image
And in the Nginx logs I see:

FastCGI sent in stderr: "PHP message: Error in Matomo: Session must be started before any output has been sent to the browser; output started in php://input/1" while reading upstream, client: <REDACTED>, server: <REDACTED>, request: "POST /index.php?module=Login HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "<REDACTED>", referrer: "https://<REDACTED>/index.php"

If I check the "Remember Me" box and login, then I see:
image
And in Nginx:

FastCGI sent in stderr: "PHP message: Error in Matomo: You must call Zend_Session::regenerateId() before any output has been sent to the browser; output started in php://input/1" while reading upstream, client: <REDACTED>, server: <REDACTED>, request: "POST /index.php?module=Login HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "<REDACTED>", referrer: "https://<REDACTED>/index.php"

This is the latest Matomo v4.1.0. Matomo/PHP/FPM and MariaDB are in Docker, Nginx is running directly on Debian 10.
@DaAwesomeP
Copy link
Author

Hmmm restarting the docker container seemed to have resolved it. Strange.

@tsteur tsteur added the answered For when a question was asked and we referred to forum or answered it. label Dec 27, 2020
@morganrallen
Copy link

If your situation is like mine, it will happen again. I seem to run into this issue ever couple of weeks. Sure, restarting the docker container gets it going again, but it's not a fix.

@DaAwesomeP
Copy link
Author

Yup, you're exactly right. Also, a new issue I have been having that is also solved by restarting it and probably related: if I am already logged in, then the main page just doesn't load entirely and is blank. It loads the navbar and left side, but not the main widgets. Settings page works fine.

@DaAwesomeP DaAwesomeP reopened this Jan 5, 2021
@morganrallen
Copy link

yup, this matches what I am experiencing.

@morganrallen
Copy link

Some additional feedback. In the 'post logged in state' where the navigation portions of the page load, but the data/content portions don't, there is also JS errors being thrown on the console.

Possibly unhandled rejection: null 3 index.php:224:202
    e https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:224
    get https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:195
    g https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
    $digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
    $apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
    k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
    v https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    onload https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    (Async: EventHandlerNonNull)
    Pg https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    s https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:204
    b https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
    k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
    $digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
    $apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
    k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
    v https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    onload https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    (Async: EventHandlerNonNull)
    Pg https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:208
    s https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:204
    b https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:202
    k https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:236
    $digest https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:248
    $apply https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:251
    c https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
    invoke https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:140
    c https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
    Wc https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:117
    Ee https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:116
    <anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:446
    i https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    fireWith https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    ready https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    J https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    (Async: EventListener.handleEvent)
    promise https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    <anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    <anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4
    <anonymous> https://xxxxxxxxxx/index.php?module=Proxy&action=getCoreJs&cb=b67469782ae943dac831248efa7dd345:4

In addition, when trying to log back in from this error state, a URL fragment is injected into the page, just after the BODY opening tag.

form_login=morgan&form_nonce=xxxxxxxxxxxxxxxxxx&form_redirect=https%3A%2F%xxxxxxxxxxxx%2Findex.php%3Fmodule%3DCoreHome%26action%3D%26period%3Dday%26date%3Dyesterday&form_password=xxxxxxxxxxxxxxxxxx

It does this weather Remember is check or not

@DaAwesomeP
Copy link
Author

This is happening very frequently. At this point I SSH in every time I need to access Matomo because I will probably need to restart it to access it or at some point while accessing it. Are there further steps we can take to debug?

@morganrallen
Copy link

So... I appear to have fixed my instance while chasing down a different issue, it's a bit of a stretch but worth looking into.
Long story short, we kept being infected by the kinsing malware due to a misconfiguration of our docker-compose file. Are any of the following true....

  • you have kinsing or kdevtmpfsi (note the i, kdevtmpfs is a real process) processes running on the system?
  • you have port 9000 exposed on your host system ( < this was our mistake)
  • you have your docker daemon port exposed on the host (2375 I think)

my understanding is the kinsing malware deletes a bunch of temp files on start and this could be causing session corruption. Again, it's a stretch but it seems to be working here.

@DaAwesomeP
Copy link
Author

Holy cow well this is a big deal. The only ports we have open are 22 (SSH w/ fail2ban & the Mozilla security spec) and 443 (Nginx running outside of Docker, fully patched). All other ports blocked with UFW. The good thing is that the malware appears to be contained within the Matomo docker container. I am suspicious that the container may have shipped with it, but that seems unlikely.

@DaAwesomeP
Copy link
Author

Aha. UFW + Docker was the problem. Luckily this server was not in production yet, so I can wipe it and start over to be sure.

@morganrallen
Copy link

morganrallen commented Jan 8, 2021 •
edited

So this appears to be the culprit?

UFW + Docker was the problem

Would be helpful for other if you elaborate on this a bit more.

@DaAwesomeP
Copy link
Author

Both Docker and UFW use iptables to manage traffic. However, Docker does not follow the rules set by UFW. So closing ports with UFW won't actually affect Docker. There isn't really an official solution to this problem. There are some workarounds, but they all have their issues (no internet in containers, no IPv6, etc.). My solution is to ensure that nothing is listening on 0.0.0.0 from Docker, but this is annoying nonetheless.

@morganrallen
Copy link

did find evidence of the malware?

@DaAwesomeP
Copy link
Author

Yup, it was contained in the Matomo container (Docker did its job), but I am going to wipe the machine anyway.

@erlangparasu
Copy link

https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/initial-access-techniques-in-kubernetes-environments-used-by/ba-p/3697975

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@morganrallen @tsteur @DaAwesomeP @erlangparasu