Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New plugin: provide simple UI to invalidate old reports #8942

Closed
mattab opened this issue Oct 7, 2015 · 7 comments
Closed

New plugin: provide simple UI to invalidate old reports #8942

mattab opened this issue Oct 7, 2015 · 7 comments
Assignees
Labels
c: New plugin For features that probably will not be added to Matomo, but could be implemented as plugins. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Oct 7, 2015

Our goal is to create a new open source plugin that will let Super Users easily invalidate any report in Piwik. This will provide a user friendly way to make this frequent action.

Proposal

  • add a new menu entry in Administration > Invalidate reports (maybe there's a better location?)
  • the menu entry and functionality is only available to Super Users (since it can result in rather big performance issues if someone invalidates a very big long range on a large piwik DB)
  • the page displays:
    • Title Invalidate historical reports
    • Description:

You can invalidate all historical reports for a specific website, or for all websites, by clicking the button "Invalidate historical reports". When you invalidate historical reports, they will be re-processed from the raw logs the next time archiving will run. This is useful when you want to force Piwik to re-process historical data for all reports, for example when:

  • you created a new Custom Report and want the Custom Reports to be processed for all historical data
  • you created a new Funnel and want your Funnel reports to be processed for all historical data
  • you have modified the raw visitor information (for example by importing new visitor logs in the past) and want these changes to raw logs reflected in all your reports.
  • Website selector (would default on "All websites" maybe?) and segment editor.
  • a button "Invalidate all historical reports"
  • on click on the button, display a popover:

You are about to invalidate all historical reports, which will cause Piwik to re-process all historical reports the next time archiving run. Re-processing all historical reports may take a long time. Are you sure you want to invalidate all historical reports for [website $WebsiteName|All websites]?

  • if a segment was also selected, also append and for segment $segmentName
  • when no segment was selected, append and for all Segments.
  • on submit:
    • the API CoreAdminHome.invalidateArchivedReports is called with those parameters that user confirmed
    • For the date parameter, we can use $websiteCreationTime,today to invalidate all reports (method getCreationDateFor)
    • If "All websites" was selected, we can use $earliestWebsiteCreationTime,today to invalidate all reports (use method getMinMaxDateAcrossWebsites)
    • we set period to false so that all periods are invalidated
    • on successful submit, write a message such as:

Reports have been successfully invalidated. These reports will be reprocessed the next time reports archiving will run.
The CoreAdminHome.invalidateArchivedReports API returned:
$API_OUTPUT_HERE

  • ideally include a UI test for the initial page load, and for the page after successful submit of an invalidation
  • Update FAQ: https://piwik.org/faq/how-to/faq_155/ to link to this new plugin as an alternative to the API solution
  • host plugin at piwik/plugin-InvalidateReports or so

Notes

@gaumondp
Copy link

gaumondp commented Oct 8, 2015

Invalidate reports (maybe there's a better name?)

From a user POV, I would say we don't want to "invalidate" anything. We really want to "recreate reports" or re-generate", "reconstitute" reports. The action to invalidate is a side-effect hidden under the hood.

Warning popover

I know we can't estimate time or CPU needed but maybe add a generic warning for exemple if the period is more than X days if entire DB is > 1 GB ?

@RMastop
Copy link
Contributor

RMastop commented Oct 8, 2015

I started to write a response this morning. The only reason I have used the invalidate feature, is because I needed to recreate an historic report, or archive a new segment for a certain period. It's more about the need to force archiving a segment or period. New segments are not archived from the very first date of tracking, for performance reasons. By being able to force archiving a fine grained set of data, we keep the load to a minimum and still be able to provide the requester their reports.

@mattab mattab added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: New plugin For features that probably will not be added to Matomo, but could be implemented as plugins. labels Dec 23, 2015
@mattab mattab added this to the Short term milestone Dec 23, 2015
@mattab mattab modified the milestones: Backlog (Help wanted), 3.2.1 Nov 10, 2017
@mattab
Copy link
Member Author

mattab commented Nov 10, 2017

Hi @sgiehl Updated the description/spec, could you take a look?

@sgiehl sgiehl self-assigned this Nov 10, 2017
@sgiehl
Copy link
Member

sgiehl commented Nov 11, 2017

  • For the date parameter, we can use $websiteCreationTime,today to invalidate all reports (method getCreationDateFor)
  • If "All websites" was selected, we can use $earliestWebsiteCreationTime,today to invalidate all reports (use method getMinMaxDateAcrossWebsites)

We can do that for the first version, but we defiantly need to do that better way later. If purging old log data is active, invalidating old reports could remove data that can't be reprocessed. So maybe it would be better to check for which day log data is available and leave all older data untouched...

@mattab
Copy link
Member Author

mattab commented Nov 12, 2017

So maybe it would be better to check for which day log data is available and leave all older data untouched...

It should already be implemented, see method removeDatesThatHaveBeenPurged and findOlderDateWithLogs, but it would be great to double check this works as expected in this case?

@sgiehl
Copy link
Member

sgiehl commented Nov 13, 2017

Plugin will soon be published: https://github.com/piwik/plugin-InvalidateReports

@sgiehl sgiehl closed this as completed Nov 17, 2017
@mattab
Copy link
Member Author

mattab commented Nov 19, 2017

Plugin available here: https://plugins.piwik.org/InvalidateReports

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: New plugin For features that probably will not be added to Matomo, but could be implemented as plugins. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

4 participants