New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Visits Over Time uses all available columns which causes the report generate slowly. #17941
Comments
@weixuezhang the solution would be to set up auto archiving see https://matomo.org/docs/setup-auto-archiving/ . When you then fetch a report it won't need to generate any reports. Fetching archived data is quite fast and there we aren't planning to change anything. For range dates there is an issue already: #7573 . We will look into this more as part of #7573 and therefore marking this as a duplicate @weixuezhang |
I have set up auto archiving with below settings in config.ini.php I add below code at head of method "public function query($sql, $bind = array())" inside file "/matomo-4.4.1/core/Db/Adapter/Pdo/Mysql.php" From matomo.log, I can see below infomation, looks metrics are fetched from several plugins, but actually only ["nb_visits","nb_uniq_visitors"] for key viewDataTableParameters_admin_VisitsSummary.getEvolutionGraph inside option table, should only VisitsSummary plugin be related. INFO VisitsSummary[2021-09-01 11:58:27 UTC] [cc6bb] sql: SELECT idsite, date1, date2, |
@weixuezhang fyi if you have disabled the archiving in the config then you also need to set either Can you maybe log more information like what the exact URL is for that request? and show how long it took to fetch the reports? Is the page slow for you? |
@tsteur , I set enable_general_settings_admin = 0, but log result is same as previous. URL of the request is like blow. It is the URL of report "Visits Over Time" After adding "columns=nb_visits" to the URL as below, I can see only VisitsSummary metrics is fetched, and the page shows faster several times than before. So, looks the problem is the report fetching all metrics (in stead of only that from option cache) when no columns set in URL parameters. |
Thanks @weixuezhang . I'll reopen the issue again as this is bit different as the other referenced issue. I'm not sure we will work on it too soon though as usually fetching these reports is quite fast. Do you have a performance problem there or how came you looked into it @weixuezhang ? be great to know to understand implications better. |
@tsteur , when I use comparison features on long period of date, I noticed the performance problem. I also found the time consumed by SQL query of obtaining archiveId for metrics on each plugin is the major portion (about 80%) of the overall query time, and also a major portion of the whole report time. So, if we can get rid of the unneeded metrics queries, the report will be several times more faster. |
BTW @weixuezhang you might want to test this change if it improves the situation maybe a little bit: https://github.com/matomo-org/matomo/pull/17943/files |
@tsteur , I tested your change, but matomo.log still shows the unneeded sql queries. I also tested the cases of URL parameter period equaling day, week and month, problem is the same. I tried a temp change as https://github.com/matomo-org/matomo/pull/17954/files, the unneeded queries can disappear, but I'm not sure it is the right fix, and if it causes other problems. |
Above suggested PR would not fix that one |
I would like to add that I also have problems with the display of the widget "Visits over time" in all of my clients' installations. Whenever a user has selected any segment, this widget takes a very long time (usually the web server timeout stops). Auto archiving is always active and browser archiving is inactive. I'm not sure that is important to this problem |
@utrautmann you might want to put this additional configuration into |
@utrautmann do you know if the segment is configured to be "pre-processed" or to be processed in "Real time"? You can find out when editing that segment. |
@tsteur All segments in the affected client installations are configured to be "pre-processed". But browser_archiving_disabled_enforce is actually set to 0. I will make a change. |
Thanks for this @utrautmann I'm expecting this will help. I've also created a PR in #18123 so that it won't be needed to set this config in that case. |
Thanks for your answer @tsteur . The "Visits Over Time" widget now renders faster. |
Thanks for contributing to this issue. As it has been a few months since the last activity and we believe this is likely not an issue anymore, we will now close this. If that's not the case, please do feel free to either reopen this issue or open a new one. We will gladly take a look again! |
When no columns specified in parameter, Visits Over Time uses all available columns which causes report generate slowly. In this situation, it should use columns from option table record option_name of viewDataTableParameters_***_VisitsSummary.getEvolutionGraph.
Expected Behavior
When no columns specified in parameter, Visits Over Time should use columns from option table record option_name of viewDataTableParameters_***_VisitsSummary.getEvolutionGraph.
Current Behavior
Visits Over Time uses all available columns so that generate many metrics not needed and lost time.
Possible Solution
Steps to Reproduce (for Bugs)
In global.ini.php, set "log_level = DEBUG" and "log_sql_queries = 1", we can see query for segment 'done90a5a511e1974bca37613b6daec137ba' which is in Frequency plugin, whose metrics are not shown in Visits Over Time.
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] Db::fetchAll() executing SQL: SELECT option_name, option_value FROM
matomo_option
WHERE option_name LIKE ?DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] Db::fetchAll() executing SQL: SELECT arc1.idarchive, arc1.value, arc1.name, arc1.ts_archived, arc1.date1 as startDate, arc2.value as nb_visits, arc3.value as nb_visits_converted
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] FROM matomo_archive_numeric_2021_08 arc1
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] LEFT JOIN matomo_archive_numeric_2021_08 arc2 on arc2.idarchive = arc1.idarchive and (arc2.name = 'nb_visits')
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] LEFT JOIN matomo_archive_numeric_2021_08 arc3 on arc3.idarchive = arc1.idarchive and (arc3.name = 'nb_visits_converted')
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] WHERE arc1.idsite = ?
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] AND arc1.date1 = ?
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] AND arc1.date2 = ?
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] AND arc1.period = ?
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] AND (((arc1.name IN ('done90a5a511e1974bca37613b6daec137ba','done90a5a511e1974bca37613b6daec137ba.VisitsSummary'))))
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2]
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] AND arc1.ts_archived IS NOT NULL
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] ORDER BY arc1.ts_archived DESC, arc1.idarchive DESC
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] Db::fetchOne() executing SQL: SELECT 1
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] FROM matomo_log_visit
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] WHERE idsite = ?
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] AND visit_last_action_time >= ?
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] AND visit_last_action_time <= ?
DEBUG CoreAdminHome[2021-08-31 03:04:55 UTC] [a58b2] LIMIT 1
Context
Your Environment
The text was updated successfully, but these errors were encountered: