Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Piwik segmented search with archived data (cron) doesn't show some periods #8436

Closed
akrus opened this issue Jul 27, 2015 · 21 comments
Closed
Labels
Bug For errors / faults / flaws / inconsistencies etc. duplicate For issues that already existed in our issue tracker and were reported previously. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Milestone

Comments

@akrus
Copy link

akrus commented Jul 27, 2015

Hi,

I have Piwik running (latest stable version) and real-time segmented search is quite slow. I've changed this to use archived data (cron) and executed full re-archive using 'piwik/console core:archive --url=http://piwik -vv -
-force-all-periods=315576000 --force-all-websites --force-date-last-n=10000'. This took one day to complete, but as I can see on the graphs, e.g. Visits over Time, in one segmented search there is no data between Apr 2014 and Jun 2015 (so I have Mar 2014 and Jul 2015), in another - Jan-Jun 2015 is missing and so on. Why this could happen? Executing console core:archive again doesn't change anything. New days are appearing fine though.

See screenshots attached.
piwik_daily
piwik_monthly
piwik_segment_settings

@tsteur
Copy link
Member

tsteur commented Jul 27, 2015

but as I can see on the graphs, e.g. Visits over Time, in one segmented search there is no data between Apr 2014 and Jun 2015 (so I have Mar 2014 and Jul 2015), in another - Jan-Jun 2015 is missing and so on

I'm not sure if I understand the issue right. It is normal that two different segmented searches result in different data. It depends on the query and data whether it is normal that there is no data or not. Or do you mean when executing the same segmented search multiple times you get a different result?

Can you let us know the segments that you use?

@akrus
Copy link
Author

akrus commented Jul 27, 2015

Same segmented search with "segmented reports are processed in real time (default)" has all the information available, see graph below:
piwik_realtime_daily
When I change this to use archives - the data is missing (graphs in first post) and I can't find out why as no errors appear during archiving.

@bodomic
Copy link

bodomic commented Jul 30, 2015

That's a good question, I experience the same issue too. Realtime works fine, archived one is patially lacking data.

@tsteur
Copy link
Member

tsteur commented Jul 30, 2015

I have no clue how this can happen right now. Maybe someone else does? It would be probably beneficial to get access to a database or to get a dump of a database to debug this issue - if possible. If that's possible for someone, please email us at "hello at piwik.org" and leave a comment here (in case it goes into spam folder)

@quba
Copy link
Contributor

quba commented Jul 30, 2015

Do you monitor archiving logs?

Such situation may happen if there were archiving errors in the past (e.g. memory limit and others). It's possible that there are incomplete archives marked as done.

There are 2 solutions to confirm:

Hope that helps.

@akrus
Copy link
Author

akrus commented Jul 30, 2015

Okay, increased memory_limit to 2G, dropped data for 2015_02 and launched archiving again, let's see...

@akrus
Copy link
Author

akrus commented Jul 30, 2015

Now it archived fine, how to deal with this situation without reading logs all the time?

@quba
Copy link
Contributor

quba commented Jul 30, 2015

Make sure that you have memory_limit set properly (2G should be fine but it depends on your traffic).

You can also set a basic monitoring (even cron job) that will grep the log file searching for errors and send an e-mail if needed.

@akrus
Copy link
Author

akrus commented Sep 3, 2015

Alright, I dropped all tables in MySQL related to archives, re-indexed everything and now the data is there, but I have another problem - whenever I add any new segmented search (e.g. new region) and switch it to use 'pre-processed data' it shows nothing and re-indexing doesn't help, the only solution is to again clear everything and initialize it from scratch.
Doesn't look like correct behavior...

@RMastop
Copy link
Contributor

RMastop commented Sep 3, 2015

I have the same issue here,
The default value of process_new_segments_from = "beginning_of_time" is still in place.
However I see data not being archived.

One solution is to delete the tables, but for large sites this is not really an option.
You don't want to delete a month of archived data for 50+ sites because of one segment failing.
Invalidate the data is not the easiest solution, because you have to create a URL with comma separated dates or a bash file that calls the URL for every date you would like to invalidate. (It would help to be able to invalidate per range, and segment)

@mattab
Copy link
Member

mattab commented Sep 18, 2015

whenever I add any new segmented search (e.g. new region) and switch it to use 'pre-processed data' it shows nothing and re-indexing doesn't help, the only solution is to again clear everything and initialize it from scratch.

@akrus Can you explain exactly the step to reproduce? it seems to work fine for us

I have the same issue here, The default value of process_new_segments_from = "beginning_of_time" is still in place. However I see data not being archived.

@RMastop do you still have this issue? how can we reproduce this?

@mattab mattab added the Bug For errors / faults / flaws / inconsistencies etc. label Sep 20, 2015
@mattab mattab modified the milestones: 2.15.0, Short term Sep 20, 2015
@RMastop
Copy link
Contributor

RMastop commented Dec 9, 2015

Hi @mattab,
We reproduced the issue. Will write the details about it to Piwik.pro support.

@mattab mattab modified the milestones: 2.16.x (LTS), Mid term Apr 11, 2016
@mattab mattab added the Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. label Apr 11, 2016
@mattab
Copy link
Member

mattab commented Apr 11, 2016

Hi @RMastop @akrus Could you please try again with 2.16.1 stable? we fixed a related issue recently in #9468 #10022 which may have fixed this issue. let us know please if you are still affected or if it's fixed using 2.16.1...

@akrus
Copy link
Author

akrus commented Apr 11, 2016

Upgraded now, should I execute full re-index or usual cron job should fix this?

@mattab
Copy link
Member

mattab commented Apr 11, 2016

Maybe wait 24 hours and see if it fixed itself already. If some old data is still not processed, you can re-process it all with the command in: https://piwik.org/faq/log-analytics-tool/faq_17994/

If you are still seeing the issue we'll investigate more and get to the bottom of it...

(added Major tag since when data doesn't show or is not tracked properly in Piwik, we consider it a major issue.)

@RMastop
Copy link
Contributor

RMastop commented Apr 11, 2016

Hi @mattab,

I'm sorry, I'm not allowed to connect to the instance where it happened.
I'll try to contact someone to validate.

Cheers,
Richard

@mattab
Copy link
Member

mattab commented Jul 18, 2016

whenever I add any new segmented search (e.g. new region) and switch it to use 'pre-processed data' it shows nothing and re-indexing doesn't help, the only solution is to again clear everything and initialize it from scratch.

@akrus could you please create a new issue? I will mark this issue as solved for now, but if you still experience this issue in Piwik 2.16.2 or later, please open a new issue and we will investigate!

@mattab mattab closed this as completed Jul 18, 2016
@mattab mattab added the duplicate For issues that already existed in our issue tracker and were reported previously. label Jul 18, 2016
@XinfinityoO
Copy link

I m having similar issues ..i cant see old records

@mattab
Copy link
Member

mattab commented Sep 27, 2016

@XinfinityoO please create a new issue and write down as many details as possible

@voarsh
Copy link

voarsh commented Dec 3, 2018

I am having a similar issue

@voarsh
Copy link

voarsh commented Dec 5, 2018

Solution:
Invalidate plugin + delete all numeric and BLOB tables.

Run:
The query to process all sites, all periods, segments. Took 3 days to process 9 GBs of data....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. duplicate For issues that already existed in our issue tracker and were reported previously. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Projects
None yet
Development

No branches or pull requests

8 participants