Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory usage and processing time skyrocketing in the referrers widget #3968

Closed
anonymous-matomo-user opened this issue Jun 1, 2013 · 8 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. worksforme The issue cannot be reproduced and things work as intended.
Milestone

Comments

@anonymous-matomo-user
Copy link

Hi,

for one site I got a big problem with showing the keywords widget when requesting the data for one month (April 2013 in this case).

I can only show the keywords if I allow more than 10 minutes of maximum execution time and more than 2 gigabytes of RAM.
Browser based archiving is disabled, cronjob for archiving is running fine. No special settings otherwise ... .
That's not a high volume site, 2.8k visits / month, running Piwik 1.11.1 right now.

Call from the widget is like the following:
<domain>/index.php?module=Referers&action=getKeywords&widget=1&viewDataTable=tableAllColumns&enable_filter_excludelowpop=0&flat=1&filter_sort_column=&include_aggregate_rows=1&idSite=49&period=month&date=2013-04-01&token_auth=XXX

@mattab
Copy link
Member

mattab commented Jun 3, 2013

If you setup the Cron as explained in: http://piwik.org/setup-auto-archiving/

The the reports should be pre-processed. Make sure you run the cron every hour. Also in the General Settings set a timeout of 3000 seconds. This should ensure that the reports are not processed more often than needed. If they take few minutes to load, it means the archiving is triggered.

@anonymous-matomo-user
Copy link
Author

Sorry but cron archiving is set up correctly and web archiving is disabled. If web archiving is disabled it should never run into it, right?
Also its only doing that for the keywords, nothing else.
The cron job is running every 6 hours and the timeout is set to 21.600 seconds because of that.

@mattab
Copy link
Member

mattab commented Jun 3, 2013

Sorry didnt see that you ts77 created the ticket!

I really don't understand the problem (low volume site, + pre-processed, should never take 10 minutes)

Can you try upgrade to 1.12 ? this might fix it.

Also you can enable [log] in the config.ini.php and add &debug=1 in the URL to see if it is triggering archiving?

finally, could work to try with XHPROF to see where it's spending so much time.

@anonymous-matomo-user
Copy link
Author

Thanks for the pointers!

I got some debugging, and it recognizes it as being archived, still using LOTS of memory, just an excerpt here:

[2013-06-04 11:41:06] [eee8136d] [6.03 Mb] Already processed, fetching idArchive = 1188 (idSite=49), for month (2013-04), plugin Referers, Visits = 2998
[2013-06-04 11:41:06] [eee8136d] [9.71 Mb] Already processed, fetching idArchive = 1188 (idSite=49), for month (2013-04), plugin Referers, Visits = 2998
[2013-06-04 11:41:06] [eee8136d] [9.86 Mb] Already processed, fetching idArchive = 1188 (idSite=49), for month (2013-04), plugin Referers, Visits = 2998
[...many similar lines here...]
[2013-06-04 11:46:52] [eee8136d] [2,401.32 Mb] Already processed, fetching idArchive = 1188 (idSite=49), for month (2013-04), plugin Referers, Visits = 2998

What could be possibly wrong to get that many results? Shouldn't there be just one entry per month?

@mattab
Copy link
Member

mattab commented Jun 8, 2013

Sorry for delay, contact me by email for urgent matters in future.

There seems to be some kind of loops going on here. Do you flat=1 or expanded=1 maybe ?

@anonymous-matomo-user
Copy link
Author

Yes, as the request above has shown its using flat=1. Digging deeper thats the "flat" view of the keywords widget.
Is that how its supposed to work? If so, is there a way to disable that completely?

@mattab
Copy link
Member

mattab commented Jun 18, 2013

To disable flattener try the following:

+++ core/API/DataTableManipulator/Flattener.php (revision )
@@ -45,6 +45,7 @@
      */
     public function flatten($dataTable)
     {
+        return $dataTable;
         if ($this->apiModule == 'Actions' || $this->apiMethod == 'getWebsites') {
             $this->recursiveLabelSeparator = '/';
         }

Does it fix the issue?

In any case it would be good if we could understand this bug, as I have not seen it before (or even heard in the forums)

@mattab
Copy link
Member

mattab commented Sep 6, 2013

Please reopen if the bug persist in 2.0 codebase from git.

@anonymous-matomo-user anonymous-matomo-user added this to the 2.0 - Piwik 2.0 milestone Jul 8, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. worksforme The issue cannot be reproduced and things work as intended.
Projects
None yet
Development

No branches or pull requests

2 participants