Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flattened "entry page" and "exit page" take a long time to be generated #11645

Closed
RMastop opened this issue Apr 24, 2017 · 7 comments · Fixed by #13496
Closed

flattened "entry page" and "exit page" take a long time to be generated #11645

RMastop opened this issue Apr 24, 2017 · 7 comments · Fixed by #13496
Assignees
Labels
c: Performance For when we could improve the performance / speed of Matomo.
Milestone

Comments

@RMastop
Copy link
Contributor

RMastop commented Apr 24, 2017

In an environment, having around 90k pageviews per month, the flattened entry page report and exit page report take some time to be generated:

selecting top 10, takes around 10 seconds,
selecting top 50, takes around 19 seconds,
selecting the default top 100, takes around 22 second

If one would compare this to the pages report top 100 plattened report, it loads in around 4 seconds. In the past the flattened pages report were slow but the speed of those report was improved at some point (I don't recall what commit caused the improvements).

Could the Entry page and Exit page report be a candidate for the same loadtime improvements?

@mattab mattab added the c: Performance For when we could improve the performance / speed of Matomo. label Jun 21, 2017
@tassoman
Copy link
Contributor

Ciao!
We happy updated to Matomo 3.2.0 in php 7.0 fpm environment by Centos7 with OpCache enabled.
Running a dualhead application installation on a single instance of MySql 5.7 DBMS.
Counting in total 6.2 millions actions last month.

Sometimes our web editors need to report theire website activity, flattened entry and exit pages actions included.

We ran in the same issue. more, our Matomo installation hangs giving back errors because php-fpm killed by SIGSEGV. It happens also asking for daily flattened entry/exit page reports.

So I've raised execution parameters to 1GB RAM amount and time to 120 sec but PHP it's still going on Fatal Error because of Allowed memory size exausted in Line 1333 of DataTable.php (links properly to v2.3.0 tag file's version)

I ask you help because SysOps feels we're running already an amount of resoruces.
Can be better? Others reports are speedy flawless ok.

Thank you for caring, bye!

@tassoman
Copy link
Contributor

tassoman commented Jun 1, 2018

Ciao @mattab , I'm wondering if there are any news on this topic.
Do you think this issue can gain some popularity in development team in the next future?

@mattab
Copy link
Member

mattab commented Jun 1, 2018

Hi @tassoman Hard to say, probably not, unless we experience the problem ourselves or one of our customers...

@tassoman
Copy link
Contributor

tassoman commented Jun 4, 2018

Thank you for fast response, I understand your point of view. Anyway I hope some company might be interested tho. 🔢

@tsteur
Copy link
Member

tsteur commented Jun 4, 2018

We can reproduce this for month as well. Takes quickly 20 seconds even when just viewing 50 rows as @RMastop mentioned.

I only had a quick look in the code but couldn't find anything obvious except maybe https://github.com/matomo-org/matomo/blob/3.5.1/plugins/Actions/DataTable/Filter/Actions.php#L65-L67 could you try to remove those 3 lines to see if it works better?

@mattab mattab added this to the 3.7.0 milestone Jul 3, 2018
@tassoman
Copy link
Contributor

tassoman commented Jul 9, 2018

Hello crew,
I did some testing after commenting lines of code @tsteur said.
We tried rendering monthly entry-page report with 50 rows limit and it ran without any memory issue.

Then we tried running the yearly entry-page report with 50 rows and it ran also until reaching of max-execution time.

To us is a quite good solution, looks like memory usage has decreased in an empiric way.
(I'm watching htop in the console and zabbix monitoring)

Thank you again for caring, best regards :)

@tsteur
Copy link
Member

tsteur commented Jul 9, 2018

I think we could update core and remove those 3 lines https://github.com/matomo-org/matomo/blame/3.5.1/plugins/Actions/DataTable/Filter/Actions.php#L65-L67 . They were added 3 years ago when we fixed a bug to track URLs correctly whereas some were before url encoded and some not. To make sure the same URLs were still aggregated correctly no matter if url encoded or not we added this group by. But I think it is not really needed anymore unless you made like aggregations over the last couple of years. Unless you updated only < 1 year from 2.12 to 3.X in which case URLs may be still mixed up in yearly archives for example...

So we could in the worst case only apply this group by only for yearly archives if this was a concern. However, users may see this problem especially in yearly archives when there are many different urls so it wouldn't help too much. For monthly and yearly archives for example we could run a simple check where we iterate over each URL, and check if there are any duplicates with urlencoded + urldecoded and only if needed run the group by. This should be fairly fast and improve speed quite a bit. Should be also quick to develop.

cc @mattab may be a good performance improvement as it currently consumes a lot of CPU

@tsteur tsteur self-assigned this Sep 28, 2018
tsteur added a commit that referenced this issue Sep 28, 2018
fix #11645

Debugged this on demo.matomo.org.... 

Wasn't as guessed https://github.com/matomo-org/matomo/blame/3.5.1/plugins/Actions/DataTable/Filter/Actions.php#L65-L67  but the flat parameter. After applying it the performance improved from minutes to less than a second.
tsteur added a commit that referenced this issue Nov 20, 2018
…ed (#13496)

fix #11645

Debugged this on demo.matomo.org.... 

Wasn't as guessed https://github.com/matomo-org/matomo/blame/3.5.1/plugins/Actions/DataTable/Filter/Actions.php#L65-L67  but the flat parameter. After applying it the performance improved from minutes to less than a second.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants