Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[2.4.1] apache logs import #5881

Closed
d-bo opened this issue Jul 24, 2014 · 2 comments
Closed

[2.4.1] apache logs import #5881

d-bo opened this issue Jul 24, 2014 · 2 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.

Comments

@d-bo
Copy link

d-bo commented Jul 24, 2014

Apache logs batch imported using command:

python import_logs.py --url=http://xxxxx/piwik/index.php $f --idsite=2 --enable-http-errors --enable-http-redirects --enable-static --enable-bots

Response cut:

Processing '/home/vathsven/www-logs/apache-access.100.gz' file...
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /home/vathsven/www-logs/apache-access.100.gz...
1200 lines parsed, 400 lines recorded, 392 records/sec (avg), 400 records/sec (current)
1723 lines parsed, 1000 lines recorded, 492 records/sec (avg), 600 records/sec (current)
2400 lines parsed, 1600 lines recorded, 524 records/sec (avg), 600 records/sec (current)
2937 lines parsed, 2200 lines recorded, 541 records/sec (avg), 600 records/sec (current)
3600 lines parsed, 2800 lines recorded, 552 records/sec (avg), 600 records/sec (current)
4200 lines parsed, 3400 lines recorded, 559 records/sec (avg), 600 records/sec (current)
4600 lines parsed, 3800 lines recorded, 536 records/sec (avg), 400 records/sec (current)
5400 lines parsed, 4600 lines recorded, 567 records/sec (avg), 800 records/sec (current)
5800 lines parsed, 5000 lines recorded, 548 records/sec (avg), 400 records/sec (current)
6505 lines parsed, 5800 lines recorded, 572 records/sec (avg), 800 records/sec (current)
7106 lines parsed, 6400 lines recorded, 574 records/sec (avg), 600 records/sec (current)
7600 lines parsed, 6800 lines recorded, 559 records/sec (avg), 400 records/sec (current)
8200 lines parsed, 7400 lines recorded, 562 records/sec (avg), 600 records/sec (current)
8800 lines parsed, 8000 lines recorded, 564 records/sec (avg), 600 records/sec (current)
9600 lines parsed, 8800 lines recorded, 579 records/sec (avg), 800 records/sec (current)
10400 lines parsed, 9600 lines recorded, 592 records/sec (avg), 800 records/sec (current)
11000 lines parsed, 10200 lines recorded, 592 records/sec (avg), 600 records/sec (current)
11800 lines parsed, 11000 lines recorded, 603 records/sec (avg), 800 records/sec (current)
11828 lines parsed, 11600 lines recorded, 602 records/sec (avg), 600 records/sec (current)

Purging Piwik archives for dates: 2014-04-14

To re-process these reports with your newly imported data, execute the following command: 
$ /path/to/piwik/console core:archive --url=http://example/piwik/

Reference: http://piwik.org/docs/setup-auto-archiving/ 

Logs import summary
-------------------

    11828 requests imported successfully
    171 requests were downloads
    0 requests ignored:
        0 invalid log lines
        0 requests done by bots, search engines, ...
        0 HTTP errors
        0 HTTP redirects
        0 requests to static resources (css, js, ...)
        0 requests did not match any known site
        0 requests did not match any requested hostname

Website import summary
----------------------

    11828 requests imported to 1 sites
        1 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 19 seconds
    Requests imported per second: 595.88 requests per second

Cannot see results on the dashboard. Nothing really imported ?
@d-bo d-bo changed the title apache logs import [2.4.1] apache logs import Jul 24, 2014
@mattab
Copy link
Member

mattab commented Jul 24, 2014

To re-process these reports with your newly imported data, execute the following command:
$ /path/to/piwik/console core:archive --url=http://example/piwik/

@mattab mattab closed this as completed Jul 24, 2014
@mattab mattab added the invalid label Jul 24, 2014
@d-bo
Copy link
Author

d-bo commented Jul 24, 2014

seems to be not re-processing:

[5dbf0] ---------------------------
[5dbf0] INIT
[5dbf0] Piwik is installed at: http://xxxxxxx/piwik/index.php
[5dbf0] Running Piwik 2.4.1 as Super User
[5dbf0] ---------------------------
[5dbf0] NOTES
[5dbf0] - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Piwik UI > Settings > General Settings. 
[5dbf0]   See the doc at: http://piwik.org/docs/setup-auto-archiving/
[5dbf0] - Reports for today will be processed at most every 10 seconds. You can change this value in Piwik UI > Settings > General Settings.
[5dbf0] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
[5dbf0] - Will process 0 websites with new visits since 7 days 0 hours 
[5dbf0] - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script) , IDs: 2
[5dbf0] ---------------------------
[5dbf0] START
[5dbf0] Starting Piwik reports archiving...
[5dbf0] Archived website id = 2, period = day, 0 visits in last last52 days, 0 visits today, Time elapsed: 1.831s
[5dbf0] Archived website id = 2, period = week, 0 visits in last last104 weeks, 0 visits this week, Time elapsed: 4.966s
[5dbf0] Archived website id = 2, period = month, 0 visits in last last52 months, 0 visits this month, Time elapsed: 1.898s
[5dbf0] Archived website id = 2, period = year, 0 visits in last last7 years, 0 visits this year, Time elapsed: 0.837s
[5dbf0] Archived website id = 2, 4 API requests, Time elapsed: 9.538s [1/1 done]
[5dbf0] Done archiving!
[5dbf0] ---------------------------
[5dbf0] SUMMARY
[5dbf0] Total visits for today across archived websites: 0
[5dbf0] Archived today's reports for 1 websites
[5dbf0] Archived week/month/year for 1 websites
[5dbf0] Skipped 1 websites: no new visit since the last script execution
[5dbf0] Skipped 0 websites day archiving: existing daily reports are less than 10 seconds old
[5dbf0] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
[5dbf0] Total API requests: 4
[5dbf0] done: 1/1 100%, 0 vtoday, 1 wtoday, 1 wperiods, 4 req, 9652 ms, no error
[5dbf0] Time elapsed: 9.652s
[5dbf0] ---------------------------
[5dbf0] SCHEDULED TASKS
[5dbf0] Starting Scheduled tasks... 
[5dbf0]  No task to run
[5dbf0] done
INFO CoreConsole[2014-07-24 10:10:06] [5dbf0] ---------------------------

nothing at dashboard.
apache logs are 'default' formatted and > 100Mb.

@mattab mattab added the wontfix label Aug 3, 2014
@mattab mattab added the Bug For errors / faults / flaws / inconsistencies etc. label Sep 20, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug For errors / faults / flaws / inconsistencies etc. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mattab @d-bo