Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Import cloudfront logs #5894

Closed
espringe opened this issue Jul 29, 2014 · 1 comment
Closed

Support Import cloudfront logs #5894

espringe opened this issue Jul 29, 2014 · 1 comment
Labels
duplicate For issues that already existed in our issue tracker and were reported previously. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone
Long term

Comments

@espringe
Copy link

Log format documentation:
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html

Here's an example of it:

#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken
2014-07-21      01:58:45        DFW3    563     180.76.5.149    GET     d116n0k3gjrs63.cloudfront.net   /robots.txt     301     -       Mozilla/5.0%2520(Windows%2520NT%25205.1;%2520rv:6.0.2)%2520Gecko/20100101%2520Firefox/6.0.2     -       -       Redirect        NLmAqKRfyqQreOK6jMmjVhh8vaUV-CbEM7m_Kta_eoZIxl0VWTEmcQ==        www.moneypot.com        http    188     0.000
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken
2014-07-21      01:44:46        AMS50   3920    5.134.58.69     GET     d116n0k3gjrs63.cloudfront.net   /img/icons/chart.png    200     https://www.moneypot.com/       Mozilla/5.0%2520(Windows%2520NT%25205.1;%2520rv:30.0)%2520Gecko/20100101%2520Firefox/30.0       -       -       Hit     j2Rusy95IefqGwjXxxfdEp1r53CLEXf7KlcHBHqyHKHN6GnQSrN8-A==        www.moneypot.com        https   329     0.002
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken
2014-07-21      01:59:06        FRA50   1096    41.227.234.26   GET     d116n0k3gjrs63.cloudfront.net   /css/header.css 200     https://www.moneypot.com/       Mozilla/5.0%2520(Windows%2520NT%25206.2;%2520WOW64;%2520rv:30.0)%2520Gecko/20100101%2520Firefox/30.0    -       -       Miss    0_y6dyIn9nsl1lI1kewczEf8BCktKcQS1hOiuAjiGunQpHnOzpgBfQ==        www.moneypot.com        https   316     0.417
2014-07-21      01:59:06        FRA50   7129    41.227.234.26   GET     d116n0k3gjrs63.cloudfront.net   /img/icons/bitcoin-ic.png       200     https://www.moneypot.com/       Mozilla/5.0%2520(Windows%2520NT%25206.2;%2520WOW64;%2520rv:30.0)%2520Gecko/20100101%2520Firefox/30.0    -       -       Miss    YSRgbarSQU48CjLSnxey1dGb7El85f1z_Ez1MM8fcdEzzBaX_GZS9Q==        www.moneypot.com        https   341     0.415
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken
2014-07-21      01:44:46        AMS50   2878    5.134.58.69     GET     d116n0k3gjrs63.cloudfront.net   /img/icons/secure.png   200     https://www.moneypot.com/       Mozilla/5.0%2520(Windows%2520NT%25205.1;%2520rv:30.0)%2520Gecko/20100101%2520Firefox/30.0       -       -       Hit     kDqn-NGq9YfCnUhK_6tYPVyMScysHrnFfySMMkWvkV43PHpZMX2Xaw==        www.moneypot.com        https   330     0.002

Since it uses w3c extended format, it is related to #5418

Cloudfront stores the logs in a directory, full of gzipped log files, e.g.

E2S2NV7MT2UOQA.2014-07-29-14.yEaJrFBy.gz
E2S2NV7MT2UOQA.2014-07-29-14.ZkCwYK8H.gz
E2S2NV7MT2UOQA.2014-07-29-14.ZpGqzm7o.gz

So it would be extra nice if one could just specify the containing directory.
@mattab mattab added this to the Mid term milestone Aug 3, 2014
@diosmosis
Copy link
Member

Fixed in #6851. Specifying a directory w/ log files is out of scope for the log importer.

@mattab mattab added the duplicate For issues that already existed in our issue tracker and were reported previously. label Dec 15, 2014
@mattab mattab mentioned this issue Dec 28, 2014
Closed
@espringe espringe mentioned this issue Dec 15, 2014
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate For issues that already existed in our issue tracker and were reported previously. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Milestone
Long term
Development

No branches or pull requests
3 participants
@diosmosis @espringe @mattab