Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After upgrading matomo from 3.13 to 4.5, can't process archive #18281

Closed
MyIcecream opened this issue Nov 8, 2021 · 13 comments
Closed

After upgrading matomo from 3.13 to 4.5, can't process archive #18281

MyIcecream opened this issue Nov 8, 2021 · 13 comments
Labels
invalid For issues or pull requests that are no longer relevant to Matomo core. Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced.

Comments

@MyIcecream
Copy link

MyIcecream commented Nov 8, 2021

After upgrading matomo from 3.13 to 4.5 then can't process archive. The log stays on this page for three days without moving (FYI, I have 36w+ domains on my server), any idea?

INFO [2021-11-08 08:53:46] 30799 ---------------------------
INFO [2021-11-08 08:53:46] 30799 NOTES
INFO [2021-11-08 08:53:46] 30799 - Async process archiving supported, using CliMulti.
INFO [2021-11-08 08:53:46] 30799 - Reports for today will be processed at most every 3600 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO [2021-11-08 08:53:46] 30799 - Archiving was last executed without error 349 days 0 hours ago
INFO [2021-11-08 08:53:47] 30799 - Will process all 360546 websites
INFO [2021-11-08 08:53:47] 30799 ---------------------------
INFO [2021-11-08 08:53:47] 30799 START
INFO [2021-11-08 08:53:47] 30799 Starting Matomo reports archiving...
DEBUG [2021-11-08 08:53:47] 30799 Applying queued rearchiving...

  • Matomo Version:4.5.0
  • PHP Version:7.4.21
  • Server Operating System:Centos7
@MyIcecream MyIcecream added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Nov 8, 2021
@sgiehl
Copy link
Member

sgiehl commented Nov 8, 2021

Hi @MyIcecream. could you check how many entries are in the table archive_invalidations? And also if the is an entry in the options table with the name ReArchiveList and maybe how big that option_value is.

@MyIcecream
Copy link
Author

MyIcecream commented Nov 9, 2021

Hi, Thanks for replying: Do you mean these?
mysql> select count() from matomo_archive_invalidations;
+----------+
| count(
) |
+----------+
| 40187578 |
+----------+
mysql> select max(length(option_value)) from matomo_option;
+---------------------------+
| max(length(option_value)) |
+---------------------------+
| 5033211 |
+---------------------------+

@sgiehl
Copy link
Member

sgiehl commented Nov 9, 2021

Seems like there are quite too many invalidation records. Might be caused be the high number of websites.
@diosmosis do you think it would be safe to clean the table and maybe the option and restart the archiving?

@MyIcecream
Copy link
Author

MyIcecream commented Nov 9, 2021

If I empty this table, will I lose data? such as all domains, just reset the archive?

@sgiehl
Copy link
Member

sgiehl commented Nov 9, 2021

I guess cleaning the table matomo_archive_invalidations, might only cause that some archives aren't triggered correctly. But that can be fixed by invalidating the archives manually later if needed.

@MyIcecream
Copy link
Author

What should I do?

@sgiehl
Copy link
Member

sgiehl commented Nov 9, 2021

Stop any archiving processes. Afterwards try running a TRUNCATE TABLE matomo_archive_invalidations and then trigger the archiving again.

@diosmosis
Copy link
Member

This seems similar to this issue: #17474 caused by updating from 3.X to one of the later 4.X's that had upgrade fixes for certain bugs in earlier 4.X releases. Truncating archive_invalidations and clearing ReArchiveList may solve the issue.

@MyIcecream
Copy link
Author

MyIcecream commented Nov 10, 2021

SELECT LENGTH(option_value) FROM matomo_option WHERE option_name = 'ReArchiveList';
+----------------------+
| LENGTH(option_value) |
+----------------------+
| 117 |
+----------------------+
SELECT option_value FROM matomo_option WHERE option_name = 'ReArchiveList';
+-----------------------------------------------------------------------------------------------------------------------+
| option_value |
+-----------------------------------------------------------------------------------------------------------------------+
| a:1:{i:0;s:99:"{"idSites":"all","pluginName":"VisitFrequency","report":null,"startDate":1606089600,"segment":null}";} |
+-----------------------------------------------------------------------------------------------------------------------+

Do you mean I can only execute these sql?
mysql> DELETE FROM matomo_option WHERE option_name = 'ReArchiveList';
mysql> TRUNCATE TABLE matomo_archive_invalidations;

@diosmosis
Copy link
Member

@MyIcecream yes, those two queries look right.

@MyIcecream
Copy link
Author

It worked, thank you all @diosmosis @sgiehl

@diosmosis
Copy link
Member

Glad to hear it worked! I'll close the issue.

@justinvelluppillai justinvelluppillai added the invalid For issues or pull requests that are no longer relevant to Matomo core. label Nov 29, 2021
@MatomoForumNotifications

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/matomo-on-premise-scaling/45262/4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
invalid For issues or pull requests that are no longer relevant to Matomo core. Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced.
Projects
None yet
Development

No branches or pull requests

5 participants