Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multi server environment in maintenance mode shouldn't result in 503 response #18054

Closed
MrIsak opened this issue Sep 23, 2021 · 5 comments · Fixed by #18862
Closed

Multi server environment in maintenance mode shouldn't result in 503 response #18054

MrIsak opened this issue Sep 23, 2021 · 5 comments · Fixed by #18862
Assignees
@peterhashair
Labels
Bug For errors / faults / flaws / inconsistencies etc. Help wanted Beginner friendly issues or issues where we'd highly appreciate community's help and involvement.
Milestone
4.9.0

Comments

@MrIsak
Copy link

MrIsak commented Sep 23, 2021

In the FAQ https://matomo.org/faq/new-to-piwik/faq_134/ it is written
"When Matomo is used behind a load balancer, here are the steps to follow:
..... in your config.ini.php, below the section [General], add the following line: multi_server_environment=1. This setting lets Matomo know you’re using multiple servers....."

When publishing a web-service via loadbalancers, you use a health-monitor that checks that the web-application answers as expected. As described in https://matomo.org/faq/how-to/faq_20278/

When activating maintenance_mode, the Reporting API and Core platform changes status-code from 200 to 503.
This will cause the loadbalancers to stop sending traffic to the web-server. And the maintenance page is not reachable for the visitors/clients.

Expected Behavior

When multi_server_environment = 1 (or new property indicating that there is a loadbalancers in front of matomo) is set.
And maintenance_mode = 1 is set, the status code from Reporting API and Core platform should still be http 200.
So clients are displayed with the maintenance page.

Current Behavior

Reporting API and Core platform returns 503 when maintenance_mode is set to 1.

Possible Solution

When using multi_server_environment ( or new property), and maintenance_mode is set to 1. Keep the response code 200.

Steps to Reproduce (for Bugs)

  1. maintenance_mode = 0
  2. curl -ik 'https://matomo.example.com/index.php?module=API&method=API.getMatomoVersion&token_auth=<token>
    Results in a 200 OK
  3. maintenance_mode = 1
  4. same command as in step 2 results in a 503 Service Unavailable.

Your Environment

  • Matomo Version: 4.4.1
  • PHP Version: PHP 7.4.3
  • Server Operating System: Ubuntu 20.04.3
@MrIsak MrIsak added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Sep 23, 2021
@tsteur tsteur added Bug For errors / faults / flaws / inconsistencies etc. Help wanted Beginner friendly issues or issues where we'd highly appreciate community's help and involvement. and removed Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. labels Sep 23, 2021
@tsteur tsteur added this to the 4.8.0 milestone Sep 23, 2021
@tsteur
Copy link
Member

tsteur commented Sep 23, 2021

Thanks for reporting this @MrIsak makes sense.

Although 503 is the best response code, I wonder if we maybe want to return an HTTP 200 in these cases in general (eg in case someone is using it behind a load balancer but doesn't have this config flag set). Could also only return http 200 when this config is set as suggested.

We should also update the docs in https://matomo.org/faq/how-to/faq_20278/ and mention that if Matomo is put in maintenance mode, then it will return a 503 unless multi_server_environment is configured.

@heurteph-ei
Copy link

heurteph-ei commented Sep 24, 2021 •
edited

Sorry, but: If the server is in maintenance (whereas other are not), it is preferable the load-balancer does not sent any request to the server during its maintenance... So it should reply 503
But it's true that if all servers are in maintenance mode, the maintenance page should be available...

@tsteur
Copy link
Member

tsteur commented Sep 26, 2021

@heurteph-ei I assume that servers would be maybe terminated once the 503 comes up and then new ones would be launched which then fail again etc and eg a running Matomo DB upgrade triggered on an instance would be terminated. And generally nobody would see the Matomo Maintenance page. @MrIsak is that maybe the case for you?

@MrIsak
Copy link
Author

MrIsak commented Sep 27, 2021 •
edited

@heurteph-ei That would be the "general" recommendation for loadbalanced environments. But due to how motomo works. ALL matomo servers will be in maintenance to prevent clients from using the GUI and have a bad user experience. Which means, no traffic will be forwarded to the enviroment, since the maintenance mode returns 503, not 200, as it says in
https://matomo.org/faq/how-to/faq_20278/

@MrIsak
Copy link
Author

MrIsak commented Sep 27, 2021

@tsteur , not sure I understand you correct. But yes. Since the recommendation in a multi-server-enviroment is to sync config.ini between servers; all servers will be in maintenance-mode at the same time. This will cause the loadbalancers to not forward traffic to matomo since the health-checks response is 503. And as you say, no one will see the maintenance-page.

@tsteur tsteur modified the milestones: 4.8.0, 4.9.0 Feb 9, 2022
@peterhashair peterhashair self-assigned this Feb 28, 2022
@peterhashair peterhashair mentioned this issue Mar 1, 2022
Merged
11 tasks
@justinvelluppillai justinvelluppillai changed the title maintenance_mode + multi_server_environment should not result in response code 503 Multi server environment in maintenance mode shouldn't result in 503 response Apr 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@peterhashair peterhashair

Labels
Bug For errors / faults / flaws / inconsistencies etc. Help wanted Beginner friendly issues or issues where we'd highly appreciate community's help and involvement.
Projects
None yet
Milestone
4.9.0
Development

Successfully merging a pull request may close this issue.

With multi_server_environment enable maintain won't send 503 as response matomo-org/matomo
4 participants
@tsteur @peterhashair @MrIsak @heurteph-ei