Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Make it flat" actions/pages report eats server resources when there's a large amount of data #5098

Closed
tassoman opened this issue May 5, 2014 · 10 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Performance For when we could improve the performance / speed of Matomo.
Milestone

Comments

@tassoman
Copy link
Contributor

tassoman commented May 5, 2014

Ciao, for year 2013 range report some of our websites have counted >2.74 M visits and >6.64 M page views.

When user opens the Actions/Pages report for the whole 2013 and tries to "Make it flat" he can't get the flattened report neither "excluding low population" because the icon wheel stills twisting with no feedback.

Server side, Apache's child uses 100% cpu, then dies in about 2 minutes because memory was exhausted (3Gb in staging environment). Before this death, it was dying by PHP's default max_execution_time setting (now raised to 300s for testing purposes).

Is there any chance to get "Make it flat" working also when we have a lot of data, or just dying with feedback in case of big amount of data, or finally could it be optionally disabled when data is too much?

@mattab
Copy link
Member

mattab commented May 6, 2014

Thanks for the report, that's a known issue.

See related #4768 Custom date range reports should archive much faster

@tassoman
Copy link
Contributor Author

tassoman commented May 7, 2014

Ciao Matt,
I've read the related ticket. I could understand the link between the tickets is related to Actions.get API call with flat=1 that happens in this event I've explained in this ticket. I'm correct?

You see we had the same problem using the old 1.x version, now we're trying to upgrade our big installation to latest 2.x stable to find this solution.

Could you say ticket #4768 will be closed in the next release 2.3.x?

Thanks for your appreciated support. Pierluigi.

@mattab
Copy link
Member

mattab commented May 8, 2014

Could you say ticket #4768 will be closed in the next release 2.3.x?

I don't think so, because its complex. But this "issue" was already there with 1.x (it was even slower then).

@tassoman
Copy link
Contributor Author

tassoman commented May 8, 2014

I see, I took a brief tour of the commits and I've seen how many critical areas of the software are involved. I dunno I could be helpful with submissions in this issue because I don't feel so skilled enough but we'd like having it working and we could test solutions with a big amount of data at home.

An idea could be storing user's UI request, showing him an explaining note that report will be available asap.
Then run the requests queue during slow times (maybe nightly?) then mailing user when report is done.

Building reports using a decoupled crontab could help avoiding max_execution_time hang in the user's UI, and keeping low server load daytime.

The problem is the server needs to archive and log 24x7 and our archive job is longer than 1hr so we have already slowed to one job each 4 hours.

@tassoman tassoman added this to the 2.x - The Great Piwik 2.x Backlog milestone Jul 8, 2014
@mattab mattab removed the P: normal label Aug 3, 2014
@mattab mattab modified the milestones: Mid term, Short term Oct 11, 2014
@pouyana
Copy link

pouyana commented Jan 16, 2015

Hi Matt,
I just read all the linked topics, here but I didn't find a way to get the flat=1 reports done, on a website with more than 1Million visitors a month and also the date=last30 and period=month and so on didn't help either. Can adding hardware resources help here?

@mattab mattab modified the milestones: Piwik 2.13.0, Short term Feb 7, 2015
@mattab mattab modified the milestones: Piwik 2.12.0, Piwik 2.13.0 Mar 9, 2015
@mattab
Copy link
Member

mattab commented Mar 9, 2015

Hi guys,
can you please test Piwik 2.12.0-b1 ? It brings huge performance improvements to flatten requests and likely this issue is already fixed! Please confirm :-)

@pouyana
Copy link

pouyana commented Mar 9, 2015

Hi I tried to use the 2.12.0-b1 but, I get JavaScript error in assoc. with Angular.js:

Error: [$injector:unpr] http://errors.angularjs.org/1.2.28/$injector/unpr?p0=piwikProvider%20%3C-%20piwik%20%3C-%20piwikNotificationDirective
z/...

@tsteur
Copy link
Member

tsteur commented Mar 9, 2015

This should be unrelated to this change. Do you mind creating an issue for this?

@tassoman
Copy link
Contributor Author

Hello there, we're planning a fully upgrade of our Piwik installation from 2.4 to the latest stable 2.1* on the staging server using a dump provided by production servers. We'll keep you informed on this asap thank you for caring of it!

@mattab
Copy link
Member

mattab commented Mar 11, 2015

This is fixed as far as we know - please comment here if you still have issues using 2.12.0-b1 or later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Performance For when we could improve the performance / speed of Matomo.
Projects
None yet
Development

No branches or pull requests

4 participants