Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

All websites dashboard - improve performance when thousands of websites #6809

Closed
mgazdzik opened this issue Dec 4, 2014 · 6 comments
Closed

All websites dashboard - improve performance when thousands of websites #6809

mgazdzik opened this issue Dec 4, 2014 · 6 comments
Assignees
@tsteur
Labels
c: Performance For when we could improve the performance / speed of Matomo. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone
2.13.0

Comments

@mgazdzik
Copy link
Contributor

mgazdzik commented Dec 4, 2014

Currently on instance with really big number of websites (i.e. 20k+) Multisites dashboard tends to time out. It's caused by loading all sites into JS in one API call.
maybe there should be pagination implemented server-side so each API call only retrieves portion that would be displayed at one time to the user to speed up this view.
@mattab mattab added Bug For errors / faults / flaws / inconsistencies etc. c: Performance For when we could improve the performance / speed of Matomo. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. and removed Bug For errors / faults / flaws / inconsistencies etc. labels Dec 4, 2014
@mattab mattab changed the title Multisites - improve performance for high sites number All websites dashboard - improve performance for high sites number Multisites Dec 12, 2014
@mattab mattab added this to the Short term milestone Dec 12, 2014
@mattab mattab changed the title All websites dashboard - improve performance for high sites number Multisites All websites dashboard - improve performance when thousands of websites Dec 18, 2014
@mattab mattab mentioned this issue Dec 18, 2014
Closed
@tsteur
Copy link
Member

tsteur commented Dec 23, 2014

BTW: Paging does not really need to be implemented. filter_limit and filter_offset works out of the box.

A problem is rather that our API does not return eg the number of total websites, etc like:

"metadata": {
     "resultset": {
           "count": 329,
           "offset": 25,
           "limit": 25,
           "page": 2,
           "next": "url to next", "previous": "url to prev..."
     }
},
"results": []

Also I think the websites group plugin does need all the websites in the browser currently to aggregate the group stats correctly. Otherwise it would be easy to do

@asafyish
Copy link

+1

@tsteur
Copy link
Member

tsteur commented Feb 8, 2015

FYI: I just performed a test with 4k websites and the problem seems not to be the JS etc. The bigger problem seems to be server side even with browser archiving disabled it takes easily 15 seconds on the server. Even if we do use paging it would still need that time. I'll have a look to see if there's a quick fix possible to make it a bit faster. Maybe we'd have to create an archive for MultiSites.getAll as well (if possible, don't know)

@tsteur
Copy link
Member

tsteur commented Feb 8, 2015

A lot of time is actually spent to generate the datatable, but also a lot of time is spent to format the datatable when having eg 4k rows (= 4k websites). So using filter_limit=10 would improve performance quite a bit as well. So we probably have to update the dashboard UI to use paging AND improve speed of report generation etc. (eg via archiving MultiSites.getAll)

@mattab
Copy link
Member

mattab commented Feb 9, 2015

Maybe we'd have to create an archive for MultiSites.getAll as well (if possible, don't know)

+1 makes sense!

tsteur added a commit that referenced this issue Feb 10, 2015
@tsteur
refs #6809 a couple of performance improvements for the all websites …
6c0db64 
…report.

I tested it with 4k websites. The change made it only a couple of seconds faster
but requesting 4k websites would still take 4-8 seconds even if only 10 sites of
them are actually requested. For example before we did request each site 3 times
(12k times in total and triggered 12k events etc) which should be no longer the case.
The last `clearSiteCache` was there to free some memory but I think it shouldn't
be problematic to remove it. Also I removed one 'sort' as a lot of time is wasted
there. It should be in theory sorted afterwards anyway again. Need to see test
results whether this change is good or not.
tsteur added a commit that referenced this issue Feb 10, 2015
@tsteur
refs #6809 make sure we do not set sites if they are already set
51391dd
tsteur added a commit that referenced this issue Feb 10, 2015
@tsteur
refs #6809 fix a couple of system tests, the sites matching pattern w…
8e54620 
…ere not correctly returned
tsteur added a commit that referenced this issue Feb 10, 2015
@tsteur
refs #6809 this should fix the broken test if MultiSites.getAll resul…
8fa4eb6 
…ts in only one requested site but uses multi period
@tsteur tsteur mentioned this issue Feb 10, 2015
Merged
@mattab
Copy link
Member

mattab commented Apr 7, 2015

it may be complicated to fix and maintain the feature Website Groups, but It would be very nice to solve this performance issue for 3.0.0 or before!

@tsteur tsteur self-assigned this Apr 8, 2015
@tsteur tsteur modified the milestones: Piwik 2.13.0, Short term Apr 8, 2015
tsteur added a commit that referenced this issue Apr 8, 2015
@tsteur
refs #6809 started working on applying the limit server side, unfortu…
e6d4132 
…nately this way it does not work as the sort would be wrong
@tsteur tsteur mentioned this issue Apr 17, 2015
Merged
tsteur added a commit that referenced this issue Apr 21, 2015
@tsteur
refs #6809 started working on applying the limit server side, unfortu…
00f147b 
…nately this way it does not work as the sort would be wrong
@mattab mattab mentioned this issue Jun 12, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tsteur tsteur

Labels
c: Performance For when we could improve the performance / speed of Matomo. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Milestone
2.13.0
Development

No branches or pull requests
4 participants
@tsteur @mattab @asafyish @mgazdzik