Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Matomo - Queue Processing slower with Exclude parameters list defined #15291

Open
siva538 opened this issue Dec 19, 2019 · 5 comments
Open

Matomo - Queue Processing slower with Exclude parameters list defined #15291

siva538 opened this issue Dec 19, 2019 · 5 comments
Labels
c: Performance For when we could improve the performance / speed of Matomo.

Comments

@siva538
Copy link
Contributor

siva538 commented Dec 19, 2019

Hello Matomo team,

We were able to process at higher speeds in Queue Processing Cron Jobs (16 #) where the rate touched 1.5K to 2K requests per second consolidated from 16 queues.

However once we defined the exclude parameters list (~1.6K individual entries in that), the rate of processing fallen down to < 100 from 16 queues in total.

Please let us know if there is a fix that can be done to address this performance problem or any workaround available.

Thanks in advance.

@tsteur
Copy link
Member

tsteur commented Dec 19, 2019

@siva538 looking at . the code removing excluded parameters is quite straight forward and should not take that long. I doubt we can do much there. Are you sure this is due to that? Do you maybe have extremely long URLs?

@siva538
Copy link
Contributor Author

siva538 commented Dec 20, 2019

Thanks @tsteur for the quick response.

We have intentionally put the "Exclude Parameters" to make sure those are not part of the actual page URLs that get tracked.

We have done another performance test w/o that list being present and the performance is good as before.

Yes we do have the long URLs with various query parameters in each.

Is it possible to optimize the matomo here?

Thanks a lot.

@tsteur
Copy link
Member

tsteur commented Dec 20, 2019

Matomo actually always excludes like 20 different parameters or so even if you haven't configured a single parameter. How many different parameters do you exclude and how long are your URLs?

@siva538
Copy link
Contributor Author

siva538 commented Dec 26, 2019

Hello @tsteur, sorry for the delay in response. We use < 1700 individual parameters to be excluded as mentioned above. The URLs are generally 1-2K characters long. Hope this clarifies.

@tsteur tsteur added the c: Performance For when we could improve the performance / speed of Matomo. label Dec 26, 2019
@tsteur
Copy link
Member

tsteur commented Dec 26, 2019

I see, that's quite long URLs and a lot of parameters. I would say it's not really on our roadmap or a big priority as it's quite edge case. Are you familiar with PHP? The code for excluding parameters is https://github.com/matomo-org/matomo/blob/3.13.0/core/UrlHelper.php#L65-L91 and maybe you could create a pull request that makes this faster?

By the looks you can use regexp in excluded parameter names (haven't tested this though). Maybe you can reduce the number of excluded parameters through regexp thus making it faster?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo.
Projects
None yet
Development

No branches or pull requests

3 participants