Hello Matomo team,
We were able to process at higher speeds in Queue Processing Cron Jobs (16 #) where the rate touched 1.5K to 2K requests per second consolidated from 16 queues.
However once we defined the exclude parameters list (~1.6K individual entries in that), the rate of processing fallen down to < 100 from 16 queues in total.
Please let us know if there is a fix that can be done to address this performance problem or any workaround available.
Thanks in advance.
@siva538 looking at . the code removing excluded parameters is quite straight forward and should not take that long. I doubt we can do much there. Are you sure this is due to that? Do you maybe have extremely long URLs?
Thanks @tsteur for the quick response.
We have intentionally put the "Exclude Parameters" to make sure those are not part of the actual page URLs that get tracked.
We have done another performance test w/o that list being present and the performance is good as before.
Yes we do have the long URLs with various query parameters in each.
Is it possible to optimize the matomo here?
Thanks a lot.
Matomo actually always excludes like 20 different parameters or so even if you haven't configured a single parameter. How many different parameters do you exclude and how long are your URLs?
Hello @tsteur, sorry for the delay in response. We use < 1700 individual parameters to be excluded as mentioned above. The URLs are generally 1-2K characters long. Hope this clarifies.
I see, that's quite long URLs and a lot of parameters. I would say it's not really on our roadmap or a big priority as it's quite edge case. Are you familiar with PHP? The code for excluding parameters is https://github.com/matomo-org/matomo/blob/3.13.0/core/UrlHelper.php#L65-L91 and maybe you could create a pull request that makes this faster?
By the looks you can use regexp in excluded parameter names (haven't tested this though). Maybe you can reduce the number of excluded parameters through regexp thus making it faster?