Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot show monthly evolution for cities (seems too slow/inefficient) #10782

Open
cyberbeat opened this issue Oct 23, 2016 · 5 comments
Open
Labels
c: Performance For when we could improve the performance / speed of Matomo.

Comments

@cyberbeat
Copy link
Contributor

I have a medium traffic website, and I'd like to see the monthly evolution data for each city:

piwik/index.php?date=2016-10-23&apiMethod=UserCountry.getCity&label=%40MyCity%252C%2520MyRegion%252C%2520MyCountry&disableLink=1&module=CoreHome&action=getRowEvolutionPopover&colors=%7B%22backgroundColor%22%3A%22%23ffffff%22%2C%22lineColor%22%3A%22%23162c4a%22%2C%22minPointColor%22%3A%22%23ff7f7f%22%2C%22maxPointColor%22%3A%22%2375bf7c%22%2C%22lastPointColor%22%3A%22%2355aaff%22%7D&idSite=1&period=month

The data does not load. It seems, that it is poor indexed, or too much data is archived.
I tried to look through the piwik code, and noticed:

a) the archived blobs for the usercountry-plugin are not indexed by city-label
b) in my case, there are over 2000 cities in each monthly-blob, so trying to show the standard view evolution for the last ~12 months for one city, more than 24000 rows have to be examined, right?

Perhaps it would be a good idea to limit the rows in the user-country-archiver plugin? Like
$maximumRowsInDataTableLevelZero = 100
$columnToSortByBeforeTruncation = 'nb_visits',

Or are there other ways to optimize these api queries?

@cyberbeat cyberbeat changed the title Cannot show monthly evolution for cities (seems to slow/inefficient) Cannot show monthly evolution for cities (seems too slow/inefficient) Oct 23, 2016
@mattab
Copy link
Member

mattab commented Nov 11, 2016

Hello @cyberbeat - thanks for the report. do you also experience this problem on demo.piwik.org or is it working well there?

@mattab mattab added the c: Performance For when we could improve the performance / speed of Matomo. label Nov 11, 2016
@mattab mattab added this to the 3.0.0 milestone Nov 11, 2016
@cyberbeat
Copy link
Contributor Author

On demo.piwik.org there are only 23 cities. I have hundreds of cities alone for germany. So this is not comparable.

I would create own sql tables for this, piwik would be much faster:

type (user_city, user_region, ..), id (city-id/region-id/..), year, month, day, visits,..

and it could even be aggregated with day=0 / month = 0
Very fast with good indices. And if it is even possible to use integer-ids for each entity (city/..) it needs very low storage size.

As for now, for piwik to lookup a monthly history for a city, it has to read many blobs "UserCountry_city", loop all hundrets of city-entries of the array in each blob, find the right city, and do the same for the next blob.

@mattab
Copy link
Member

mattab commented Feb 21, 2017

@mattab mattab closed this as completed Feb 21, 2017
@mattab mattab added the worksforme The issue cannot be reproduced and things work as intended. label Feb 21, 2017
@cyberbeat
Copy link
Contributor Author

This took around 20 seconds for me each for two tries with different cities (period=day):

http://demo.piwik.org/index.php?module=CoreHome&action=index&idSite=7&period=day&date=yesterday&updated=2#?idSite=7&period=month&date=2017-02-18&category=General_Visitors&subcategory=UserCountry_SubmenuLocations&popover=

And you may have not a low budget vserver like me, and default-php-timeoutlimits will apply a few seconds later. I have switched to another solution in the meantime because of this.

@mattab
Copy link
Member

mattab commented Feb 21, 2017

It's true that now it also takes 15-20s here. I'll reopen then

@mattab mattab reopened this Feb 21, 2017
@mattab mattab removed the worksforme The issue cannot be reproduced and things work as intended. label Feb 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo.
Projects
None yet
Development

No branches or pull requests

2 participants