@diosmosis opened this Issue on April 24th 2012 Member

Currently the All Websites dashboard will work well when there are hundreds or even thousands of websites, but after a point it becomes unusable. It should, however, be able to scale to any number of websites.

There are currently many barriers to scaling this area, including:

  • Lack of server-side pagination means all site data must be queried at once.
  • The design of the Array Archive types will result in unnecessary object creation (in this case several objects per website).
  • There's no way to archive metrics for all websites, so summary & evolution data has to be calculated when queried.

These issues will be addressed in this ticket.

@mattab commented on August 7th 2012 Member

What is the status of this ticket, and how much work do you estimate is left to finish it?

@diosmosis commented on August 8th 2012 Member

Still quite a bit of work left. I have to finish the refactoring of the Archive types, then commit the changes, piece by piece. One of the other two changes may not be as hard to implement. I'll send you a patch of what I've done so far so you can take a look (w/o having to review it).

@mattab commented on March 11th 2013 Member

Please contact us if you are interested in this fix!

@pebosi commented on May 24th 2013 Contributor

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

@diosmosis commented on May 24th 2013 Member

Replying to pebosi:

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

Hi @pebosi, the dashboard should be usable for ~1000 websites on any period except range, but ONLY if browser-initiated archiving is disabled (controlled by the first setting in the General Settings admin page). Is this disabled for you?

@diosmosis commented on July 6th 2013 Member

Fixed by Archive.php refactor. When archiving is disabled and there are 20,000 sites each w/ visits, MultiSites.getAll takes 12-14s to return the result on my laptop. Memory usage is at 118 MB.

@anonymous-matomo-user commented on July 6th 2013

In 3a3517e48823a4d7dc521e5a754c51ddbcf5d210: Refs #3124, add benchmark for MultiSites.getAll and benchmark fixture that adds 20,000 sites to the DB then tracks one pageview per site.

@mattab commented on July 6th 2013 Member

Awesome Benaka, truly beautiful!

This Issue was closed on July 6th 2013
Powered by GitHub Issue Mirror