Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Queued tracking: when importing visits on a day different from the visit's day, invalidate the archived reports #6850

Closed
mattab opened this issue Dec 14, 2014 · 4 comments
Closed

Queued tracking: when importing visits on a day different from the visit's day, invalidate the archived reports #6850

mattab opened this issue Dec 14, 2014 · 4 comments
Assignees
@tsteur
Labels
Bug For errors / faults / flaws / inconsistencies etc.
Milestone
2.10.0

Comments

@mattab
Copy link
Member

mattab commented Dec 14, 2014

The goal of this issue is to make the QueuedTracking logic more reliable, in particular when some queued tracking requests were not imported into the Tracking API in real time.

Use case to reproduce issue

For example let's imagine there was a problem on the Tracking API server or database which was down. During this time 100,000 pending requests are queued into Redis. While the system is down the core:archive command is running, creating archives of "zero visit" for "yesterday". When the system comes back up the 100k requests will be imported into "yesterday". The yesterday's reports will still show zero visit data because Piwik considers the day was archived already. So we are seeing gaps in our data!

Proposed solution
@mattab mattab changed the title Queued tracking: when importing visits on a day different from the visit day, invalidate the day Queued tracking: when importing visits on a day different from the visit's day, invalidate the archived reports Dec 14, 2014
@mattab mattab added the Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. label Dec 14, 2014
@mattab mattab added this to the Piwik 2.10.0 milestone Dec 14, 2014
@tsteur
Copy link
Member

tsteur commented Dec 17, 2014

the selected lines point to custom vars? Maybe create a link to the tag of the latest version instead of master? edit: found it!

@tsteur tsteur self-assigned this Dec 17, 2014
@tsteur tsteur added Bug For errors / faults / flaws / inconsistencies etc. and removed Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. labels Dec 17, 2014
@tsteur
Copy link
Member

tsteur commented Dec 17, 2014

I was thinking about this for a while and I think it has to be solved in the tracker itself somehow. Or in the archiver. The fact that we implemented it in log importer and that we'd have to reimplement it again shows there is something wrong. Also I think when using the cdt tracker param to set a custom time in the past it wouldn't invalidate an already finished archive which then shows it definitely has to be solved in the tracker or archiver.

@mattab
Copy link
Member Author

mattab commented Dec 17, 2014

I think when using the cdt tracker param to set a custom time in the past it wouldn't invalidate an already finished archive

it's definitely a bad bug that has created confusion before in the forums, and also partly covered in #6683 - would be really nice for consumers of Tracking API to have this bug solved as it will make their life much easier!

@mattab
Copy link
Member Author

mattab commented Dec 17, 2014

I was thinking about this for a while and I think it has to be solved in the tracker itself somehow.

👍 it makes the most sense in Tracker itself. it has to be fast to do but CoreAdminHome.invalidateArchivedReports is not that fast - maybe we cache the fact that some day has been invalidated? (we'd need to clear the cache when the day is actually being archived, to make sure subsequent calls to the Tracker will again re-invalidate archived report.

@mattab mattab mentioned this issue Dec 17, 2014
Closed
@tsteur tsteur mentioned this issue Dec 18, 2014
Merged
@tsteur tsteur closed this as completed Dec 22, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tsteur tsteur

Labels
Bug For errors / faults / flaws / inconsistencies etc.
Projects
None yet
Milestone
2.10.0
Development

No branches or pull requests
2 participants
@tsteur @mattab