@siva538 opened this Issue on December 19th 2019 Contributor

Hello Matomo team,

We were able to process at higher speeds in Queue Processing Cron Jobs (16 #) where the rate touched 1.5K to 2K requests per second consolidated from 16 queues.

However once we defined the exclude parameters list (~1.6K individual entries in that), the rate of processing fallen down to < 100 from 16 queues in total.

Please let us know if there is a fix that can be done to address this performance problem or any workaround available.

Thanks in advance.

@tsteur commented on December 19th 2019 Member

@siva538 looking at . the code removing excluded parameters is quite straight forward and should not take that long. I doubt we can do much there. Are you sure this is due to that? Do you maybe have extremely long URLs?

@siva538 commented on December 20th 2019 Contributor

Thanks @tsteur for the quick response.

We have intentionally put the "Exclude Parameters" to make sure those are not part of the actual page URLs that get tracked.

We have done another performance test w/o that list being present and the performance is good as before.

Yes we do have the long URLs with various query parameters in each.

Is it possible to optimize the matomo here?

Thanks a lot.

@tsteur commented on December 20th 2019 Member

Matomo actually always excludes like 20 different parameters or so even if you haven't configured a single parameter. How many different parameters do you exclude and how long are your URLs?

@siva538 commented on December 26th 2019 Contributor

Hello @tsteur, sorry for the delay in response. We use < 1700 individual parameters to be excluded as mentioned above. The URLs are generally 1-2K characters long. Hope this clarifies.

@tsteur commented on December 26th 2019 Member

I see, that's quite long URLs and a lot of parameters. I would say it's not really on our roadmap or a big priority as it's quite edge case. Are you familiar with PHP? The code for excluding parameters is https://github.com/matomo-org/matomo/blob/3.13.0/core/UrlHelper.php#L65-L91 and maybe you could create a pull request that makes this faster?

By the looks you can use regexp in excluded parameter names (haven't tested this though). Maybe you can reduce the number of excluded parameters through regexp thus making it faster?

Powered by GitHub Issue Mirror