New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
archive process trouble with parameter N #16409
Comments
Hi @Jinzol thanks for creating the issue. Did you invalidate the reports before running this command? AFAIK we added some code to change |
Well I did not invalidate the reports before running because I want to use a custom N with archiving. I have problem that the data of the month august is not correct while the archives says its correct. We duplicate the data to an other environment with its own archiving process and there is a difference in results. What I think what is happening is that my archiving process takes sometimes to long (48h+) and the next archiver run will skip data because N is to small. You get small gaps in the processed data and that gives different results. Thats why I want to always increase the value of N. But that my theorie :-) |
If you don't invalidate the reports upfront, Matomo should usually not rearchive these reports even if it uses eg |
Well I can give that a try and I will report it back. I takes some time to reprocess the data. But in the meantime I still don't understand why the archivers have finished correctly and the processed data is wrong. Or my raw data is incorrect but with the mirror environment it seems that is not the issue. We have seen more deadlocks on the database. What kind of effect does that have of the archivers? Or how can I prevent them. I would like to say to my colleague's the archiver is done and the data is trustworthy. The problem with invalidate reports is that I know something is wrong and now I don't know what went wrong. |
@Jinzol let us know when you have an update. It's hard to say if/what went wrong there. It also depends maybe how it all works with your mirror environment etc. Is it happening repeatedly or just once? What kind of deadlocks are triggered? Do you have more information eg what kind of query is having the issue? |
Hi @tsteur, well it took some time but I reprocessed august and it seems fixed.
The last one took 80768 seconds or 22,43 hours. I hope it gets abit faster with day,week,month and n=2. I think my problem started that some archive run didn't finish correctly. I will add the logging of the deadlock.
It seems it happens around ~5 a day. |
Thanks. Be good to create a new issue for the deadlock if there isn't one already for that particular deadlock (we can also close as duplicate if needed). be great to watch if the problem happens again, otherwise we can maybe close the issue now and reopen if the problem happens again? You will maybe also want to write the output of the archiving somewhere like
|
Hi I want to process some specific data for a specific segment for specific time.
It seems the parameter " --force-date-last-n" does not work anymore and default is used.
The code what I use is
/opt/rh/rh-php72/root/usr/bin/php /srv/www/matomo/console core:archive --force-date-last-n=45 --force-idsites=3 --force-periods=day --force-idsegments=450 --concurrent-requests-per-website=1
When I check the logging of this piece of code it shows
INFO [2020-09-08 15:34:37] 26330 Will pre-process for website id = 3, period = day, date = last2 INFO [2020-09-08 15:34:37] 26330 - pre-processing all visits
It should be last45 in this case and it has worked.
We run Matomo 3.13.5
How can I force to run the previous 45 days instead the last 2. Date range is not really an option.
I hope you can help me :-)
The text was updated successfully, but these errors were encountered: