Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log analytics piping in Apache #6405

Closed
donjan opened this issue Oct 9, 2014 · 4 comments
Closed

Log analytics piping in Apache #6405

donjan opened this issue Oct 9, 2014 · 4 comments
Labels
duplicate For issues that already existed in our issue tracker and were reported previously. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@donjan
Copy link

donjan commented Oct 9, 2014

Trying to get direct piping of log analytics to work on an Apache 2.4.7 / Ubuntu 14.04 system.
From looking at issues #3757, #3163 and #6200 something like this in the Apache conf file should work:

CustomLog "|/usr/bin/python -u /var/www/piwik/misc/log-analytics/import_logs.py --url=http://testsrv/piwik/ - --idsite=1 --log-format-name=common_vhost" vhost_combined

But this doesn't add any data to Piwik (checked in PMA, and accordingly nothing in the web UI). Adding --output=/var/log/piwik/test.log or checking Apache's error.log doesn't provide more clues (I can provoke errors in both files by using incorrect syntax or unknown option names).

The access log is in the vhost_combined format and the following as root does work:

tail -n1 access.log | /usr/bin/python -u /var/www/piwik/misc/log-analytics/import_logs.py --url=http://testsrv/piwik/ - --idsite=1 --log-format-name=common_vhost

It correctly adds the last page hit ("1 requests imported to 1 sites"), but when omitting the last option (--log-format-name=common_vhost) it gives:

0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log (stdin)...
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Fatal error: cannot automatically determine the log format using the first 100000 lines of the log file. 
Maybe try specifying the format with the --log-format-name command line argument.

I've tried various permutations, verbosely copying from misc/log-analytics/README.md and the threads linked above (especially since #3757 concludes that something somehow does work), but can't get a clear handle on what is going wrong. It looks as if it should be working, so any hints or an up to date how-to are appreciated.

@mattab mattab added this to the Long term milestone Oct 11, 2014
@donjan donjan changed the title Log analytics piping into Apache Log analytics piping in Apache Oct 12, 2014
@donjan
Copy link
Author

donjan commented Oct 12, 2014

@mattab
What would be an estimate for "long term"?
Log analytics is the main option for me since PHP tagging on various 3rd party web apps is cumbersome and JS tracking isn't acceptable. Log piping instead of rotation would make me switch to Piwik.

@gaumondp
Copy link

@donjan , long term in Piwik's universe means 3 to 5 years.

Source: http://piwik.org/roadmap/

@mattab
Copy link
Member

mattab commented Oct 17, 2014

@donjan if you need something faster you are welcome to sponsor developer time on it. Please get in touch at: http://piwik.org/consulting/ if you're interested

@mattab mattab added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: Log Analytics (import_logs.py) labels Nov 3, 2014
@mattab mattab modified the milestones: Mid term, Long term Nov 3, 2014
@mattab
Copy link
Member

mattab commented Mar 12, 2015

Issue was moved to the new repository for Piwik Log Analytics: https://github.com/piwik/piwik-log-analytics/issues

refs #7163

@mattab mattab closed this as completed Mar 12, 2015
@mattab mattab added the duplicate For issues that already existed in our issue tracker and were reported previously. label Mar 12, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate For issues that already existed in our issue tracker and were reported previously. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

3 participants