Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limit amount of sites in SitesManager. #5892

Closed
wants to merge 8 commits into from
Closed

Limit amount of sites in SitesManager. #5892

wants to merge 8 commits into from

Conversation

czolnowski
Copy link
Contributor

Add offset and limit parameter to API methods which is used to fetching sites.

@czolnowski
Copy link
Contributor Author

Referenced to: #3838

@mattab
Copy link
Member

mattab commented Aug 1, 2014

Thanks for PR!

is there reason you added offset/paging to several APIs ie getSitesWithAdminAccess, getSitesWithViewAccess, getSitesWithAtLeastViewAccess ? it seems that the Manage Websites screen in #3838 only uses getSitesWithAdminAccess so maybe the parameter could be added only to this API?

@mattab mattab modified the milestones: Piwik 2.5.0, Piwik 2.6.0 Aug 3, 2014
@mattab mattab closed this Aug 6, 2014
@julienmoumne
Copy link
Member

After discussing the SitesManager scalability issue during the meet-up, we concluded client side paging could be implemented as a first step.

If we later find client side paging does not scale enough, server side paging could be considered.

@czolnowski
Copy link
Contributor Author

We tested SitesManager with 20.000 sites and backend doesn't respond, because of memory limit. For bigger instances of piwik we have implement backend limiting and offseting, as well filtering too.

@mattab
Copy link
Member

mattab commented Aug 6, 2014

@czolnowski was there a memory error from server? because loading 20k websites in an array sounds cheap on memory eg. could fit in 200kb?

@mattab mattab added the wontfix label Aug 15, 2014
@czolnowski
Copy link
Contributor Author

@mattab: Problem is when you want to print this using ResponseBuilder. It cause memory error.

@mattab
Copy link
Member

mattab commented Aug 25, 2014

can you paste the memory error with backtrace? maybe we can make it less memory hungry as it shouldn't cause a memory error I think.

@czolnowski
Copy link
Contributor Author

Here is one of logs from nginx:

2014/08/25 12:41:35 [error] 8310#0: *536 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388580 bytes) in /home/piwik/vhosts/sites-manager.mypiwik.pro/core/DataTable/Renderer/Xml.php on line 202
PHP message: PHP Stack trace:
PHP message: PHP   1. {main}() /home/piwik/vhosts/sites-manager.mypiwik.pro/index.php:0
PHP message: PHP   2. require_once() /home/piwik/vhosts/sites-manager.mypiwik.pro/index.php:47
PHP message: PHP   3. Piwik\FrontController->dispatch() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/dispatch.php:34
PHP message: PHP   4. Piwik\FrontController->doDispatch() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/FrontController.php:86
PHP message: PHP   5. call_user_func_array() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/FrontController.php:576
PHP message: PHP   6. Piwik\Plugins\API\Controller->index() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/FrontController.php:576
PHP message: PHP   7. Piwik\API\Request->process() /home/piwik/vhosts/sites-manager.mypiwik.pro/plugins/API/Controller.php:33
PHP message: PHP   8. Piwik\API\ResponseBuilder->getResponse() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/Request.php:217
PHP message: PHP   9. Piwik\API\ResponseBuilder->renderValue() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/ResponseBuilder.php:76
PHP message: PHP  10. Piwik\API\ResponseBuilder->handleArray() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/ResponseBuilder.php:137
PHP message: PHP  11. Piwik\API\ResponseBuilder->getRenderedDataTable() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/ResponseBuilder.php:375
PHP message: PHP  12. Piwik\DataTable\Renderer\Xml->render() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/ResponseBuilder.php:230
PHP message: PHP  13. Piwik\DataTable\Renderer\Xml->renderTable() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/DataTable/Renderer/Xml.php:36
PHP message: PHP  14. Piwik\DataTable\Renderer\Xml->renderArray() /home/piwik/vho

And database:

mysql> select count(*) from piwik_site;
+----------+
| count(*) |
+----------+
|    20001 |
+----------+
1 row in set (0,00 sec)

@mattab
Copy link
Member

mattab commented Aug 25, 2014

Ok so it looks like we have some very inefficient code in there... maybe we need a little XHprof analysis to figure out why and maybe we can find solution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.
Projects
None yet
Milestone
2.7.0
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@czolnowski @mattab @julienmoumne @tsteur