The archiving phase took about 2 hours.

In case the "Provider" plugin is enabled, try to disable it. This can make it much faster

In case the "Provider" plugin is enabled, try to disable it. This can make it much faster

fyi @tsteur the Provider plugin is automatically disabled by the log importer, unless --enable-reverse-dns is specified

Requests imported per second: 261.23 requests per second

It's decent number, but to improve performance of the log import process, one can try to increase the imported requests per second. We have before got 600 or even 800 requests per second using log analytics. It will help performance if you use the QueuedTracking plugin: http://plugins.piwik.org/QueuedTracking

Since this import I set up Piwik to do an hourly import/analyse run as per the docs. What I'm seeing is spectacularly bad performance that's interfering with normal server operation. It's clear from my activity monitoring that Piwik restarts its analysis from the begining of the log file every time it's run, and it hits the database for every single record, every time - so as the current log file gets bigger, Piwik's processing time gets longer and longer, until the log file is rotated and it drops back down again.

You can see this happening in this graph: the little peaks get bigger through the day:

Over a week it looks like this:

This doest look like much, but it's actually saturating disk access. Here's the database POV:

This is all due to Piwik activity because the app on this server only uses MyISAM, and this graph only shows InnoDB, which Piwik uses. You can see here the InnoDB activity since installing Piwik: the initial import (that I originally opened this ticket for) followed by 2 weeks of hourly imports:

What I gather from this is that Piwik does not know how to skip records that have been seen before - so rather than looking up the latest timestamp in the DB and then scanning the log file and ignoring all lines prior to that, or storing a log file inode and byte ofset so that it can jump straight to where it left off, it starts from the beginning every time and absolutely hammers the database and iowait.

I'm not using reverse DNS lookups, I don't need them for this case.

Is this normal?

@Synchro Thanks for the feedback. it is "normal" that Piwik works in this way, but we are aware this is not good enough and we hope to do better in the future 👍 I've created a new issue to discuss this particular problem: Log Analytics could detect log lines that were already imported and skip them automatically #144 I hope the description helps. for now you may have to workaround this issue by importing each log file only once, eg. once a day.