Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

import of apache log file takes to long (over 10h) #3081

Closed
anonymous-matomo-user opened this issue Apr 4, 2012 · 1 comment
Closed

import of apache log file takes to long (over 10h) #3081

anonymous-matomo-user opened this issue Apr 4, 2012 · 1 comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. worksforme The issue cannot be reproduced and things work as intended.

Comments

@anonymous-matomo-user
Copy link

Hi Piwik-Team

I'm new to the apache and web-analytics world and i received the task to migrate from urchin 6.0 to piwik.

We use a vm machine as a piwik server with 4 cpus and 4gb ram.
After installing piwik 1.7.2 RC6 i've tried to import a 200MB apache logfile ( over 1 Millions of lines ).

The records/sec is between 20 - 40... and take over 10h to import! i've tried to change the amount of recorders to the suggested value ( recorders = cpu's -1 ) but this changed nothing :-(

i don't use any special options:
python import_logs.py --url=http://piwiksrv --show-progress --idsite=6 --config=/var/www/piwik/config/config.ini.php /root/wwwlogs/sito_test_log/sitobps_access_log --recorders=3

cut out from import process:
1011393 lines parsed, 915958 lines recorded, 26 records/sec
1011393 lines parsed, 915983 lines recorded, 25 records/sec
1011393 lines parsed, 916007 lines recorded, 24 records/sec
1011393 lines parsed, 916032 lines recorded, 25 records/sec
1011393 lines parsed, 916058 lines recorded, 26 records/sec
1011393 lines parsed, 916084 lines recorded, 26 records/sec
1011393 lines parsed, 916110 lines recorded, 26 records/sec
1011393 lines parsed, 916138 lines recorded, 28 records/sec
1011393 lines parsed, 916164 lines recorded, 26 records/sec
1011393 lines parsed, 916190 lines recorded, 26 records/sec
1011393 lines parsed, 916215 lines recorded, 25 records/sec
1011393 lines parsed, 916238 lines recorded, 23 records/sec
1011393 lines parsed, 916263 lines recorded, 25 records/sec
1011393 lines parsed, 916286 lines recorded, 23 records/sec
1011393 lines parsed, 916309 lines recorded, 23 records/sec

also during the import process the top output doesn't show any high io waits, load ...

any tips you can give me?

Thx and regards

Ivan Torretti

Keywords: log import
@mattab
Copy link
Member

mattab commented Apr 4, 2012

Thanks for the feedback. If you know Python it would be great to profile and find out why it is so slow on your system. You should achieve 100 requests per second with standard server like yours.

PLease post your feedback on the official ticket for this script: #703 and we appreciate any kind of debugging you can do!

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug For errors / faults / flaws / inconsistencies etc. worksforme The issue cannot be reproduced and things work as intended.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mattab @anonymous-matomo-user