It is not currently possible to test Piwik under high load. The current approach is to use xhprof while tracking one or many requests and while archiving. Since archiving happens once per hour and there is never more than once 'archiving process' running at once (other core:archive processes should continue the existing run), using xhprof alone is ok. Tracker performance, on the other hand is not so easily gauged.

Sure, using xhprof on a single request or multiple requests will provide some useful information and allow you to optimize, but it says nothing about how the tracker will behave under high levels of concurrency. Simulating high load requires many concurrent tracking requests and special monitoring.

Enter Locust

Locust (http://locust.io/) is a load testing framework that can simulate millions of concurrent users. It also looks very easy to use. Though it requires writing python code. (I'm ok w/ that, not sure if everyone else is.)

We can create a system that uses EC2 & locust to automate load testing the tracker. Then after the tracker is tested, we can automatically launch archiving w/ xhprof using the data that was just tracked.

This can provide us with performance regression testing under high load, but more importantly, we can test the effect plugins have on tracker & archiving performance automatically. By comparing the results of a test when a plugin is enabled to the test for core, we can see just how much overhead a plugin adds, and maybe even find ways to boost performance.

The System

• Plugins should be able to create a custom locust profile describing test visits and test setup. For example, Goals would make sure goals are defined for the test websites being tracked, and define visits that triggered conversions. An example locust file w/ Piwik visit data might look like:

  class VisitWithConversionTasks(TaskSet):
      @task
      def shortVisitWithConversion(self):
          self.client.get("?idsite=1&..."); # action 1
          self.client.get("..."); # action 2
          # ...
  
      @task(5) # occurs 5 times less than above
      def manualGoalConversion(self):
          self.client.get("?idsite=1&idgoal=1&...");
          # ...
  
  class VisitWithConversion(Locust):
      weight = 1
      task_set = VisitWithConversionTasks
  
  class VisitWithMultipleConversion(TaskSet):
      # ...
  
  class VisitWithMultipleConversion(Locust):
      weight = 3 # happens 3 times less frequently than visit with conversion
      task_set = VisitWithMultipleConversionTasks
  

• A new command tests:load [--plugin=...] should be created that will start up an EC2 instance w/ Piwik installed, then a EC2 instance w/ locust installed. Later, if it's needed to run tests against millions of concurrent visits, perhaps clusters can be created. [Note: Perhaps puppet files or something else can be used to specify different Piwik instance configurations. For example, normal w/ no redis cache, normal w/ redis cache, w/ queued tracking installed, etc.]

• The command will run the load test and save metric data to the filesystem, then use xhprof on a core:archive run and save the profile.

• If possible, the command should get database performance metrics. Something that can be used to pinpoint problematic queries or tell if caches aren't being effective.