The goal of this issue is to build a tool that will make it easy for Piwik developers to detect when Piwik master branch becomes slower, to ensure that Piwik is fast and usable even under extreme conditions.
we're doing a good job at not often regressing core APIs and key features, and we tend/aim to write tests when fixing regressions. But for Piwik to shine, it must not only be stable and reliable, but also damn fast even when pushed to the limits. Speed is really a key feature of a great software product.
We are proud to say that there is no data limit. To make it a reality we want to create an environment where it's easy for our team members (and the community) to be confident that a given Piwik build is fast even when "limits" are pushed.
we could measure performance of all main Piwik modules (UIs, Archiving, API, Tracker...) under several performance and load test cases (Tracker requests bombing, creating 50k websites and requesting dashboard, archiving 1k segments, etc.).
Our first automated performance use case could be: All websites dashboard should load fast when there are 20k+ websites added in Piwik. This was regressed a few times in the past eg. #7877 and @tsteur points out in that it's very easy to regress and introduce slowness in particular in the
MultiSites.getAll API. triggered codepath is very wide and this source code modified & refactored often. We know it's a matter of time before it becomes slower again.
We could define performance requirements and create acceptance speed tests eg. "All websites dashboard should load in less than 5 seconds when 30k websites are requested and pre-archived". Maybe we add a new CI job at every commit to run those performance tests, and the CI job is Green when the speed requirements are met (how to deal with issues with CI server speed un-reliability). Maybe we need to run such process on a dedicated server we control or rent in a cloud. Let's discuss.
Your thoughts are most welcome!
I'm sure there are solutions for this already (frameworks/libs etc). In the end we only need to call urls (that works even for archiving tests) and measure the time etc. If there's a good, affordable paid service this could be an interesting thing as well.
Not a direct solution to the problem but could help: having the platform where the cloud traffic is reproduced and monitoring with New Relic. New Relic supports for example annotating deployments, so you can compare performances before/after deploying a new beta. Of course this is not a catch all, but still better than nothing and probably easier to setup at first?