Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fatal Error: '' from import_logs.py #6451

Closed
orlitzky opened this issue Oct 15, 2014 · 6 comments
Closed

Fatal Error: '' from import_logs.py #6451

orlitzky opened this issue Oct 15, 2014 · 6 comments
Labels
duplicate For issues that already existed in our issue tracker and were reported previously. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone
Long term

Comments

@orlitzky
Copy link

This started about a week ago, I don't think anything changed here. Every night we run,

import_logs.py --enable-http-errors --enable-http-redirects --enable-bots --enable-static --recorders=6 --url=https://analytics.example.com/ <all logs from yesterday>

Now it regularly gets stuck:

Parsing log /var/log/apache2/example.com/www/access/access-2014-10-14.log...
6886 lines parsed, 6284 lines recorded, 79 records/sec (avg), 148 records/sec (current)
6886 lines parsed, 6461 lines recorded, 80 records/sec (avg), 177 records/sec (current)
6886 lines parsed, 6676 lines recorded, 82 records/sec (avg), 215 records/sec (current)
...
6886 lines parsed, 6676 lines recorded, 25 records/sec (avg), 0 records/sec (current)

and eventually fails with the following:

6886 lines parsed, 6676 lines recorded, 25 records/sec (avg), 0 records/sec (current)
Fatal error: ''
You can restart the import of "/var/log/apache2/example.com/www/access/access-2014-10-14.log" from the point it failed by specifying --skip=5 on the command line.

Is that the sixth line of the log file? If so, there's nothing weird in it.

The problem began on 2.7.0, but persists after an upgrade to 2.8.0.
@tsteur
Copy link
Member

tsteur commented Oct 15, 2014

Is there a chance that you can send us the log with like the first 10 lines? In case it is reproducible with only 10 lines...
If so, please send to hello at piwik.org.

@orlitzky
Copy link
Author

It's on its way, but some more testing suggests that the log file isn't (entirely) to blame. With six recorders,

202 lines parsed, 190 lines recorded, 94 records/sec (avg), 77 records/sec (current)
...
202 lines parsed, 190 lines recorded, 5 records/sec (avg), 0 records/sec (current)
...

(and eventual death). But if I set it to 1,2,3,4, or 5 recorders, it succeeds almost immediately.

@tsteur
Copy link
Member

tsteur commented Oct 16, 2014

Thx for sending the log. Unfortunately, I couldn't reproduce no matter using 1,2,3,4,5,6 or 10 recorders. Tried it several times. Even added a website to my Piwik to make sure it is tracked. Also had a look at the code and could not find anything. I am using Python 2.7.3 by the way.

That's my output http://pastebin.com/U6x8atf4

Also tried to import other logs with more entries from VisitorGenerator without any issues.

@diosmosis @mattab can you guys try to reproduce it as well?

@orlitzky
Copy link
Author

It certainly is perplexing. It fails with 6 and 7 recorders, but works with 8.9, and 10. Python here is 2.7.7. Strace was... not enlightening.

@mattab mattab added Bug For errors / faults / flaws / inconsistencies etc. c: Log Analytics (import_logs.py) labels Oct 17, 2014
@mattab mattab added this to the Short term milestone Oct 17, 2014
@mattab
Copy link
Member

mattab commented Dec 1, 2014

Decreasing priority, feel free to comment if you also experience this issue with high number of recorders.

@mattab mattab added Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. and removed Bug For errors / faults / flaws / inconsistencies etc. labels Dec 1, 2014
@mattab mattab modified the milestones: Mid term, Short term Dec 1, 2014
@orlitzky orlitzky mentioned this issue Dec 1, 2014
Closed
@mattab
Copy link
Member

mattab commented Mar 12, 2015

Issue was moved to the new repository for Piwik Log Analytics: https://github.com/piwik/piwik-log-analytics/issues

refs #7163

@mattab mattab closed this as completed Mar 12, 2015
@mattab mattab added the duplicate For issues that already existed in our issue tracker and were reported previously. label Mar 12, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate For issues that already existed in our issue tracker and were reported previously. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Milestone
Long term
Development

No branches or pull requests
3 participants
@tsteur @mattab @orlitzky