New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Detect when stats load slowly and explain to user to disable Browser trigger archiving in Admin #6210
Comments
Two small notes:
I'm not sure what is default date selection but for sure having "yesterday" in general would be better default than "today". |
Today we got again some user feedback of frustration at the slowness of the
In some way it's true that it's broken by design because we do not tell users if they don't look in the settings or the docs. Because we all know the issue can be solved by setting up the cron, I've added a |
from forum feedback: http://forum.piwik.org/read.php?2,125969,page=1#msg-126260
Maybe we could also double check the wording in installer. Right now it says "setup the cron if you have a high traffic website" or so, instead we could make it clear that it's important for fast Piwik experience to setup the cron for most websites (except for very low traffic Piwik.) |
FYI: not sure how this could be implemented... Could we hook into our ajax global queue, and/or the AngularJS http queue, to detect if any pending request is taking more than N ~ 15 seconds? Here is something maybe relevant for this problem: for our UI tests we had to check whether all HTTP requests are finished. When all HTTP are received by the browser then only we can take a screenshot as we know everything is loaded. This wasn't working well in PhantomJS and this is why we had to use our global ajax queue and angular HTTP queue. See the magic code in the UI test helper code: https://github.com/piwik/piwik/blob/2.16.0-rc1/tests/lib/screenshot-testing/support/page-renderer.js#L503-L520 |
It be fine as long as we do this easy and simple as a diagnostic check for example. Eg this check runs when Piwik is installed, and we'd always recommend disabling browser archiving (since it always makes it faster and should always be used). For segment archiving we could also add another warning if enabled and explain the trade off (eg longer wait times vs on demand data and not needing to preconfigure segments). checking how long a UI call takes etc won't be that helpful. There is also already a check for this in the "Tour" plugin, and we will be sending out emails mentioning this (we are working on this as part of a different issue). Later when we go into onboarding improvements we'll see what else we can do. This feature we can generally easily add any time though so it's not high prio for Matomo 4 and we need to make sure to spend as little time on it as possible. |
See e.g. https://forum.matomo.org/t/slow-dashboard-for-sharepoint-2019/37071?u=lukas for an example |
Sounds good to have a diagnostics
fyi: additionally to the system check that people may not see (unless they are super users or so, and visit the Admin page). After all if someone has only 10 or 100 visits a day, and don't use segments eg., it's probably mostly loading fast enough for them and wouldn't need to always see a notice at the top. So the idea be to show a message only when it's clearly too slow and they wouldn't know why (maybe even all users would see it like "Reports seem to load slowly. Consider notifying a Super User so they can setup auto-archiving of reports."). But yes this could be done in a later step (and the diagnostic check above added in the meantime.) |
If you want any slow detection, then I'll move it out into a different milestone. Also slow is relative and depends on many factors. Generally, we should always recommend cron archiving and most super users would see it eventually. If people have only few visits, they are likely on some slow hoster etc so we should always recommend to get the best Matomo experience as otherwise people will say things are slow to others. These days normal expectations are that things load in 200ms. |
Also keep in mind that we will be sending out emails about this etc as part of onboarding etc and will further work on it then. |
When cron archiving is used, date ranges should be fast to archive, correct? Since we're just summing up other periods. UX ideas:
@tsteur what do you think? |
they can be still slow in general but depends on the range and the report. It's bit hard though to instead recommend using the other periods as it's really a base functionality to have custom ranges and often you do need to select a specific range. Overall it should be still somewhat fast enough that it isn't needed to show anything but it might also depend on the server. Since #6672 is done I wouldn't do anything as there isn't really a proper workaround available. Regarding segments can we maybe add this to the system check as well for that setting? For the existing system check maybe we could rename it or create a new one as I wonder if we could also show to every user with at least Generally I'm not seeing the point of measuring any time as even 1s or 0.5s server time can be perceived as slow these days and people would even at 0.5s talk negatively about Matomo's performance. That's why I'm thinking we should always make people aware of this performance matter guaranteeing best performance possible. Of course measuring time does make sense when it is obviously slow and takes say > 5s to load. It could be only a temporary issue though if it's on a shared hoster or so and it can get quickly complicated when measuring time. That's why I'm thinking ideally we find away to guarantee that people become aware of CLI archiving. Not sure if we are already showing something during the install but could eg add another step when installing just informing people about this topic? There be no action and only a "next" button but the text would explain they could improve performance? |
To my understanding, this is for individual segments. For example, a segment that is not pre-processed and takes a long time to load, gets this notification displayed, so the user knows they can make it auto-archive.
This isn't the check I was talking about, it was:
This is possible, but seems like extra work since we'd need to store whether the notification has been clicked away or not (eg, in the session).
This check for time is only for segments that are not pre-processed. Everything else we don't have a clear action for users to take. |
OK we could suggest that if browser archiving is disabled (otherwise it wouldn't work). Just to keep in mind it might get a bit difficult. eg a segment for I think also only users with at least admin access can create segments to pre-process via CLI currently. So regular view and write users that create a segment we wouldn't be able to show that information AFAIK. I understand this is the case as it can cause a lot of storage and processing time when preprocessing a segment through CLI on the other side maybe view/write users should still be able to set segments to CLI archiving since it is otherwise simply slow @mattab ? (unless I'm understanding the code wrong). I'm thinking we should maybe also change the default to CLI processing maybe (if browser archiving is disabled)? @mattab Of course both default settings have their pros and cons but thinking performance the resulting experience is quite critical and there's usually no need to process segments on demand to get results faster while it is not acceptable to have a slow Matomo. System checkI see @diosmosis . didn't even notice that one in the system check. Only saw this one which already was a warning
Yeah we would need to store this in a user setting or user flag or something and only show it once. We might not need this for now I would say but can always add it later.
Yep indeed. See message above a lot of settings will need to be taken into account and user roles. I'm thinking though we could avoid a lot of that by simply making segments archiving through CLI the default and let also users with at least view access create segments to be processed in the background. (anonymous is not allowed to create segments anyway). Then we could simply focus on getting people to disable browser archiving in the first place (eg make them aware in installer etc) |
yes, all segment settings only for super users. (noticed this particular setting can be disabled even for super users via
just tested and it looks like it's just super users who can change any of the segment settings
Sounds like a good idea. But some people use segments only for filtering Visits log and technically don't require them pre-processed. That might be an edge case though so it'd make sense to change the default. However that might also quickly create some problems for people who create many segments via API or even via the UI and don't realise the impact it will have on the server archiving process. basically it will likely break the archiving whenever enough people create enough segments (or via the API). Because currently there is no way to get a list of segments pre-processed. so when people have this problem and have many segments and sites, it will be hard for them to find which segments are problematic, and eg. they'd have to use a SQL query we provide to figure it out. so maybe also it would help to report somewhere, how many segments will be pre-processed. Maybe a diagnostics could tell super users the list of all segment names and which websites they are created on, and which are created across "all sites", and highlight those which are pre-processed. (or just show the pre-processed ones). |
Maybe we wouldn't list all pre-processed segments in the system check as there could be hundreds or thousands of sites and there could be hundreds of segments or more. Maybe for now just the number of preprocessed segments could be shown in the system summary There could be also a CLI command or so to get a list of all segments that are being pre-processed (archived in background). It's bit annoying re when people use the segment only for visits log. I wish there was a way to separate these or make it more clear for people to understand when to use which type of segments but it's not really trivial. Maybe filters would partially help maybe (not to work on this now just mentioning it in general): #15438 So just to confirm we would allow anyone with view access to create Segments that are being |
it would be useful to show the number of segments like you suggest eg. |
Yeah listing hundreds of them in worst case wouldn't help either though. Linking to export them might work. Whether people find this information, and know what to do with it etc is all a different question I suppose. And even them realising it is because of segments etc etc. |
@tsteur so the reqs for this issue can be summarized as:
anything else? |
I would keep the
Yes 👍
Yes. We'll also need to change permissions that users with view access will be able to create auto archive segments when browser archiving is disabled . Additionally:
|
Should both be visible in the system summary then? The last archiving time diagnostic was added so super users can see if something is wrong w/ their cli archiving because it hasn't completed in a while (IIRC).
Should this be controlled by a config or measurable setting? I could imagine some admins might not want this?
We could put a quick blurb w/ some instructions about setting up cron/link to the faq at the end of installation. Would be useful to at least let them know there's something that the installer can't do itself, but would be good for most setups. What do you think? |
I suppose you mean in the system check? yes both should still exist. It's just that we no longer need to check if browser archiving is disabled in the last archiving time diagnostic check as it's a duplicate and will be done in the
I'd say we could always add it later when it's requested or really needed. Be good to add the complexity only when really needed (that part is already quite complex enough).
Could do 👍 |
I mean here: the system check info appears to just be an informational result: I think maybe there should be a warning displayed? |
Yes it shouldn't be informational but a proper check and show a warning 👍 |
fyi, it looks like this is already the case: https://github.com/matomo-org/matomo/blob/4.x-dev/plugins/SegmentEditor/API.php#L109-L119 only requires admin access when browser archiving is enabled |
@diosmosis I think this one is when |
@tsteur that seems different, the code in that function seems to allow view access auto-archive if |
The goal of this issue is to help users who have poor experience of Piwik by detecting when it happens and displaying a help notification to guide them in right direction.
Problem: Piwik loads stats slowly for many users when Browser trigger archiving is enabled in settings. It affects mostly
Suggested tasks
Dashboard
andAll Websites
dashboard, detect when stats take more than a few seconds to load and display inline notification suggesting to solve the issue.Last 30 days
then we should pre-process it anyway, see Pre-process custom date ranges when users have selected them as their default date range #6672How often should the notification be displayed? if the user closes the notification, maybe it could be displayed again in one week. The notification should be displayed to all users but messaging should change for Super User to recommend them to change the settings or setup cron (for normal users it would recommend to contact the Piwik Administrator to let them know of the issue and suggest them to tweak Piwik).
Who is affected by issue?
Almost all users by default will experience this issue sooner rather than later. By default the automated cron is not setup and user has browser archiving enabled in piwik settings. by default we load Today, and All websites can also load the Last 30 days, so the user will request a lot of fresh data most of the time!
Maybe you have other ideas?
The text was updated successfully, but these errors were encountered: