Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IIS Advanced Logging Module log files support for Log Analytics #6795

Closed
mattab opened this issue Dec 3, 2014 · 2 comments
Closed

IIS Advanced Logging Module log files support for Log Analytics #6795

mattab opened this issue Dec 3, 2014 · 2 comments
Assignees
@diosmosis
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Milestone
2.10.0

Comments

@mattab
Copy link
Member

mattab commented Dec 3, 2014

The goal of this issue is to add support in the Log Analytics tool for automatically importing the access logs generated by IIS web server called IIS Advanced Logging Module.

This is a common log format for users of IIS server. Here is an example:

#Software: IIS Advanced Logging Module
#Version: 1.0
#Start-Date: 2014-11-18 00:00:00.128
#Fields:  date-local time-local s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) cs(Host) sc-status sc-substatus sc-win32-status TimeTakenMS
2014-11-17 17:00:00.363 10.10.28.140 GET /Products/X/_Images/ico_print.gif - 80 - "70.95.93.8" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G900V Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.59 Mobile Safari/537.36" "http://example.com/Search/SearchResults.pg?informationRecipient.languageCode.c=en" "xzy.example.com" 200 0 0 109
2014-11-17 17:00:00.660 10.10.28.140 GET /Topic/hw43061 - 80 - "157.55.39.72" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" - "example.hello.com" 302 0 0 0
2014-11-17 17:00:00.675 10.10.28.140 GET /hello/world/6,681965 - 80 - "173.5.186.174" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com" - "hello.example.com" 404 0 0 359

This was also requested in #261 with a similar yet different format:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port
cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
time-taken
2014-04-10 12:40:24.190 - GET /Common/shop/shoppingcartAJAX.asp - 80 -
192.168.1.1 "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36" 200 0 0 2036

Notes:

  • User should be able to import such logs without having to use the complicated "Custom regular expression feature" (example).
  • Maybe all IIS advanced log files contain the schema at the start of the file eg. #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken which maybe we could use to parse the file?
  • Add automated test in eg. ImportLogsTest or so

Refs #4707
@mattab mattab added Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: Log Analytics (import_logs.py) labels Dec 3, 2014
@mattab mattab added this to the Piwik 2.11.0 milestone Dec 3, 2014
@mattab mattab mentioned this issue Dec 3, 2014
Closed
@mattab mattab modified the milestones: Piwik 2.10.0 , Piwik 2.11.0 Dec 4, 2014
@mattab
Copy link
Member Author

mattab commented Dec 5, 2014

While we're at it it would be nice to map cs-username to uid see #5980

@diosmosis diosmosis self-assigned this Dec 5, 2014
diosmosis pushed a commit that referenced this issue Dec 8, 2014
Refs #6795, add --iis-map-field option to allow users of log importer…
fa367aa 
… script to parse log files created using the IIS Advanced Logging module. Includes tests for such a case (including python unit test + PHP System test).
@diosmosis diosmosis mentioned this issue Dec 8, 2014
Closed
@diosmosis
Copy link
Member

Fixed by #6825.

diosmosis pushed a commit that referenced this issue Dec 18, 2014
Refs #6795, remove quotes around referrer & host if found when parsin…
f960e41 
…g hits in log importer.
@mattab mattab mentioned this issue Dec 9, 2014
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@diosmosis diosmosis

Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Projects
None yet
Milestone
2.10.0
Development

No branches or pull requests
2 participants
@diosmosis @mattab