503 Service Unavailable after upgrade to 4.13.0 #20119

Expected Behavior

I expected matomo to work without a 503 error.

Current Behavior

Matomo returns http error 503 Service Unavailable

Possible Solution

No idea?

Steps to Reproduce (for Bugs)

• While using version 4.12.3
• Upgrade by using the Automatic method to version 4.13.0
• Watch all pages return 503 Service Unavailable

Context

I just wanted to upgrade.

Here is part of the Apache log:

"GET /matomo/index.php HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"
"GET /matomo/index.php?module=MultiSites&action=index&idSite=9&period=day&date=today HTTP/1.1" 503 299 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"
"GET /matomo/index.php?module=MultiSites&action=index&idSite=9&period=day&date=today HTTP/1.1" 503 299 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"

Here is part of the Error log:

AH01067: Failed to read FastCGI header
AH01075: Error dispatching request to :

The first request is to the index.php which redirects to the next URL, which fails with 503. The third request is me hitting refresh page.

Out of curiosity, I looked at the parts of the code that produce a 503 error, they are:

$ grep -ri "sendResponseCode(503" | grep '\.php:'
plugins/CoreUpdater/CoreUpdater.php:                Common::sendResponseCode(503);
core/FrontController.php:            Common::sendResponseCode(503);
core/Tracker/Response.php:            Common::sendResponseCode(503);

Your Environment

• Matomo Version: 4.13.0
• PHP Version: PHP 7.4.30
• Server Operating System: AlmaLinux 8.7
• Additionally installed plugins: None
• Browser: Firefox 108
• Operating System: Fedora Linux 37

Below is the output from console plugin:list

+----------------------+-------------------+---------------+
| Plugin               | Core or optional? | Status        |
+----------------------+-------------------+---------------+
| Diagnostics          | Core              | Activated     |
| BulkTracking         | Core              | Activated     |
| CoreConsole          | Core              | Activated     |
| ScheduledReports     | Core              | Not activated |
| UserCountryMap       | Core              | Activated     |
| Live                 | Core              | Activated     |
| PrivacyManager       | Core              | Not activated |
| ImageGraph           | Core              | Activated     |
| Annotations          | Core              | Not activated |
| MobileMessaging      | Core              | Not activated |
| Overlay              | Core              | Activated     |
| SegmentEditor        | Core              | Activated     |
| Insights             | Core              | Activated     |
| Morpheus             | Core              | Activated     |
| Contents             | Core              | Activated     |
| Resolution           | Core              | Activated     |
| Login                | Core              | Activated     |
| DevicePlugins        | Core              | Activated     |
| Heartbeat            | Core              | Activated     |
| UserId               | Core              | Not activated |
| ProfessionalServices | Core              | Not activated |
| Marketplace          | Core              | Not activated |
| RssWidget            | Core              | Not activated |
| IntranetMeasurable   | Core              | Activated     |
| TwoFactorAuth        | Core              | Activated     |
| GeoIp2               | Core              | Activated     |
| Tour                 | Core              | Not activated |
| DBStats              | Core              | Not activated |
| PagePerformance      | Core              | Activated     |
| CustomDimensions     | Core              | Not activated |
| CoreUpdater          | Core              | Activated     |
| Feedback             | Core              | Not activated |
| VisitorInterest      | Core              | Activated     |
| Transitions          | Core              | Activated     |
| CoreAdminHome        | Core              | Activated     |
| UsersManager         | Core              | Activated     |
| SitesManager         | Core              | Activated     |
| Installation         | Core              | Activated     |
| Monolog              | Core              | Activated     |
| Intl                 | Core              | Activated     |
| CorePluginsAdmin     | Core              | Activated     |
| CoreHome             | Core              | Activated     |
| WebsiteMeasurable    | Core              | Activated     |
| CoreVisualizations   | Core              | Activated     |
| Proxy                | Core              | Activated     |
| API                  | Core              | Activated     |
| Widgetize            | Core              | Not activated |
| LanguagesManager     | Core              | Activated     |
| VisitTime            | Core              | Activated     |
| Actions              | Core              | Activated     |
| Dashboard            | Core              | Activated     |
| MultiSites           | Core              | Activated     |
| Referrers            | Core              | Activated     |
| UserLanguage         | Core              | Activated     |
| DevicesDetection     | Core              | Activated     |
| Goals                | Core              | Activated     |
| Ecommerce            | Core              | Activated     |
| SEO                  | Core              | Activated     |
| Events               | Core              | Activated     |
| UserCountry          | Core              | Activated     |
| VisitsSummary        | Core              | Activated     |
| VisitFrequency       | Core              | Activated     |
| CoreVue              | Core              | Activated     |
| CustomVariables      | Optional          | Not activated |
+----------------------+-------------------+---------------+

Below is the output from console diagnostics:run

Database abilities:
    - OK UTF8mb4 charset
    - WARNING LOAD DATA INFILE
Hint: Using LOAD DATA INFILE by updating your PHP and MySQL software and ensuring your database user has the FILE privilege speeds up Matomo's archiving process a lot.
You should fix this problem if your Matomo server tracks high-traffic websites (e.g. over 100,000 pages per month).
    - OK CREATE TEMPORARY TABLES
    - OK Changing transaction isolation level
Max Packet Size: WARNING It is recommended to configure a 'max_allowed_packet' size in your MySQL database of at least 64MB. Configured is currently 16MB.
2 warnings detected

Thank you!

@LauraTaylorUK Are you able to see any error message in the matomo log file (matomo/tmp/logs/matomo.log)?

I can't find any logs under matomo/tmp/logs, the directory only has two files:

index.htm
index.php

I couldn't find a solution, so I deleted my matomo installation and re-installed v4.13.0 from scratch, but I kept the same config.ini.php file and my SQL database. Things are back to normal.

Maybe something went terribly bad during the upgrade procedure?

Here is the broken matomo directory, diff'ed with the working matomo directory:

$ diff --exclude=tmp -ru ~/broken/ ~/public_html/working/
Only in broken/config: .htaccess
Only in broken/: favicon.ico
Only in broken/js: .htaccess
Only in broken/lang: .htaccess
Only in broken/misc: GeoLite2-City.mmdb
Only in broken/misc: internal-docs
Only in broken/misc/log-analytics: .gitignore
Only in broken/misc/log-analytics: .travis.yml
Only in broken/misc: phpstorm-codestyles
Only in broken/misc/user: .gitkeep
Only in broken/misc/user: .htaccess
Only in broken/plugins: CustomVariables
Only in broken/plugins: .htaccess
Only in broken/plugins: Provider
Only in broken/vendor/tecnickcom/tcpdf: .github

Interestingly, I see a lot of htaccess files missing from the working matomo directory, but I don't think they are the cause of the 503 errors.