Make widgets load faster on Piwik dashboard, and sparklines load faster on All Websites #7104

ThaDafinser · 2015-01-29T16:07:50Z

tl;dr:

Change https://github.com/piwik/piwik/blob/master/core/FrontController.php#L470 with this code (its an ugly hack to just show the gain)

Session::start();
if($module != 'CoreHome' && $module != 'Dashboard'){
    Session::writeClose(true);
}

Somehow i always felt that the GUI, especially the dashboard, is quite slow.

I now measured it and and remembered an easy solution when i comes to speed up AJAX requests.
Since Piwik starts the session for each request, all pending requests are queued by php if i remember correctly https://github.com/piwik/piwik/blob/master/core/FrontController.php#L470
(Reason: every request could write to the session and so it should be queued to have consistent session data)

So if you disable the ability to write session, the queueing is not needed anymore.

_Note_
This is an ugly hack to just show the performance gain! Of course each action (or plugin) have to decide if it dont need to write anytime

without hack

with hack

.

sgiehl · 2015-01-29T19:52:47Z

+1 for refactoring the session handling.
always closing the session and only reopening it where required seems to be a good way.

ThaDafinser · 2015-01-30T07:15:47Z

I dont know if that would be correct.

I close the session it in my application for some AJAX requests or longer exports where i know the session is only read.

If you always close the session and reopen it, the behaviour could be strange (already had gone through that)

Example: 2 requests are submitted from the user at the same time
#1) modify the session at the end (takes 1s to process)
#2) reads the session and modify another var (takes 0,5s to process)

After the 2 requests, #1 have won and the change from request #2 is lost.

The only "clean" solution in my mind, is to close it at the Plugin or explicit action level.

ThaDafinser · 2015-01-30T07:26:02Z

Just to keep in mind: For the little benchmark i used a production system, which have 2 servers: one for apache/php and one for mysql (each one got 4 CPU and 8GB memory).

Maybe a not so "powerful" setups could get problemes with the number of requests

tsteur · 2015-02-01T23:31:33Z

The problem will be to know which requests are session read only and which ones are read/write...

I presume we don't use the session for API reports, eg we could close it immediately again for calls against the Metadata API (API.getProcessedReport, API.getReportMetadata) as eg. the mobile app sometimes requests many processed reports at once. Other API calls might use the session for some reason.

In the UI the performance benefit would be more or less only relevant for the dashboard within the "Dashboard Report page" when many widgets are requested at once. We could close the session directly if &widget=1 is set in the URL but we'd have to check this. Eg we need to make sure possible Nonce/CSRF tokens are invalidated if needed etc.
Also we save notifications in the session, meaning if a Widget triggers a "persistent" notification it would not be actually persistent. We could ignore this case I presume...

It would be nice to test this on some more (production) servers. Eg on my local instance there is not really a performance benefit but this has different reasons...

tsteur · 2015-02-01T23:56:21Z

I did another test on a production system. I reloaded the pages many times to make sure it is not a random result because of something else.

Without session close (load event after 8.8s):

With session close (load event after 7.8s):

As one can see most reports load faster especially the ones further below (they have to wait longer). Some requests take only 1 second instead of 3. For some other the benefit is not as big (because they didn't have to wait that long of course). One was actually able to see that reports were loading a bit faster and more in "parallel".

BTW: Interesting was too see that getSearchEngines seems to be pretty slow!

tsteur · 2015-02-02T03:09:30Z

Re slow searchEngines report see #7119 . Prepared a pull request...

mattab · 2015-02-09T04:30:32Z

If we make Piwik use Database sessions by default rather than file sessions, would this solve the concurrent session issue and slowness?
(I don't think there's a particular reason to use the file sessions handler)

tsteur · 2015-02-09T04:49:00Z

Nope, it wouldn't I think

mattab · 2015-02-09T06:40:33Z

Is this issue mid term or do you think it is possible to solve it without big effort?

ThaDafinser · 2015-02-10T09:22:17Z

I think this should be done partly in short term for some views.

e.g my "All websites dashboard" also needs nearly 2min to load all evolutionGraphs from the 37 websites.

With Session::writeClose(); i'm down to 20s....the more important thing is that the further click e.g. on the detail of a webpage is not blocked i can directly go forward.

See the PR #7166

tsteur · 2015-02-10T19:39:32Z

I think we can work on it in Piwik 2.12.0 but would only do it for widgets (win for Piwik) and getProcessedReport (win for Piwik and Piwik Mobile) if possible see comment. We need to check whether they need any kind of sessions though eg re Nonce invalidation etc. Otherwise easy to implement I reckon.

Edit: Sparklines might benefit from it as well!

mattab · 2015-02-10T21:48:53Z

Added to 2.12 as per your note!

tsteur · 2015-02-14T01:45:23Z

FYI: I noticed for all API calls we already do not even start the session. So we do not have to care about API.getProcessedReport. Still interesting to close session again for &viewDataTable=sparkline and &widget=1

…ajax requests are made within widget or when sparklines are requested

tsteur · 2015-02-14T02:45:49Z

Nonces shouldn't be a problem as we use TTL based expiration and not hops based (which are usually more "secure"). If we try to write an Nonce and session was already closed an exception will be triggered and shown in the UI (as anyone could append widget=1 anywhere).

Otherwise only notifications can be a potential problem I think but those widget and sparkline requests shouldn't trigger any. In an implementation I am working on I do check whether there is a referrer so in case one opens it directly we would still display/register those notifications. I don't check the referrer for security as anyone can send any referrer anyway. For example we do need to check the referrer in case the URL parameter widget=1 will be somehow forwarded in URL's and if this parameter was forwarded to a link to the admin an exception would be shown as Nonce would not be writable. It's hard to explain right now :) Just saying I do check referrer for a reason, not for security :)

If referrer information is not there (eg was filtered because of whatever) the session would just be not closed but everything would still work.

For the widget requests that also have a token_auth set we would not even need to start a session but then there is the risk that somewhere a SessionNamespace is instantiated - which would then start the session a bit delayed - and we'd have the problem again. Eg in Plugin/Controller on most requests a session would be started for notifications.

…ted from dashboard or allwebsites

tsteur · 2015-02-14T03:01:46Z

I will maybe do another test for this on our demo test server to check whether there is a performance improvement. @ThaDafinser maybe you can give this change a test as well? https://github.com/piwik/piwik/compare/7104

ThaDafinser · 2015-02-16T07:37:10Z

@tsteur i use now your patch. For now it seems to work.

If i get problems, i will inform you 👍

…ion. Sometimes, eg when module=API, we do not start a session. If any code part during such a request uses SessionNamespace although we did not start a session, Zend_Session_Namespace will start a Zend_Session which does not respect our configuration. It would use default Zend Session options and therefore probably file based sessions. Therefore, we force the creation of a "Piwik session". If we have created the session before it won't be started again.

mattab · 2015-02-19T23:19:23Z

Kuddos @ThaDafinser for the suggestion and @tsteur for the power fix!

tsteur added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: Performance For when we could improve the performance / speed of Matomo. labels Feb 1, 2015

ThaDafinser mentioned this issue Feb 10, 2015

Issue #7104 #7166

Closed

mattab added this to the Piwik 2.12.0 milestone Feb 10, 2015

ThaDafinser mentioned this issue Feb 13, 2015

"Action" caching #7198

Closed

tsteur added a commit that referenced this issue Feb 14, 2015

refs #7104 directly close session again to prevent any blocking when …

1c34cc2

…ajax requests are made within widget or when sparklines are requested

tsteur added a commit that referenced this issue Feb 14, 2015

refs #7104 make sure to close session for sparklines only when reques…

6b8d9f9

…ted from dashboard or allwebsites

tsteur added a commit that referenced this issue Feb 14, 2015

refs #7104 fix typo

ef3c466

tsteur mentioned this issue Feb 19, 2015

Faster dashboard and sparklines by closing the session early #7252

Merged

mattab added a commit that referenced this issue Feb 19, 2015

refs #7104 Rename vars and put logic into method

41544b4

mattab modified the milestones: Piwik 2.11.1, Piwik 2.12.0 Feb 19, 2015

mattab closed this as completed Feb 19, 2015

mattab changed the title ~~Up to 500% (and more) faster piwik GUI with a few lines...~~ Make widgets load faster on Piwik dashboard Feb 23, 2015

mattab changed the title ~~Make widgets load faster on Piwik dashboard~~ Make widgets load faster on Piwik dashboard, and sparklines load faster on All Websites Feb 23, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make widgets load faster on Piwik dashboard, and sparklines load faster on All Websites #7104

Make widgets load faster on Piwik dashboard, and sparklines load faster on All Websites #7104

ThaDafinser commented Jan 29, 2015

sgiehl commented Jan 29, 2015

ThaDafinser commented Jan 30, 2015

ThaDafinser commented Jan 30, 2015

tsteur commented Feb 1, 2015

tsteur commented Feb 1, 2015

tsteur commented Feb 2, 2015

mattab commented Feb 9, 2015

tsteur commented Feb 9, 2015

mattab commented Feb 9, 2015

ThaDafinser commented Feb 10, 2015

tsteur commented Feb 10, 2015

mattab commented Feb 10, 2015

tsteur commented Feb 14, 2015

tsteur commented Feb 14, 2015

tsteur commented Feb 14, 2015

ThaDafinser commented Feb 16, 2015

mattab commented Feb 19, 2015

Make widgets load faster on Piwik dashboard, and sparklines load faster on All Websites #7104

Make widgets load faster on Piwik dashboard, and sparklines load faster on All Websites #7104

Comments

ThaDafinser commented Jan 29, 2015

without hack

with hack

sgiehl commented Jan 29, 2015

ThaDafinser commented Jan 30, 2015

ThaDafinser commented Jan 30, 2015

tsteur commented Feb 1, 2015

tsteur commented Feb 1, 2015

tsteur commented Feb 2, 2015

mattab commented Feb 9, 2015

tsteur commented Feb 9, 2015

mattab commented Feb 9, 2015

ThaDafinser commented Feb 10, 2015

tsteur commented Feb 10, 2015

mattab commented Feb 10, 2015

tsteur commented Feb 14, 2015

tsteur commented Feb 14, 2015

tsteur commented Feb 14, 2015

ThaDafinser commented Feb 16, 2015

mattab commented Feb 19, 2015