Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom log format for log analytics #5947

Closed
sudeshg opened this issue Aug 7, 2014 · 5 comments
Closed

Custom log format for log analytics #5947

sudeshg opened this issue Aug 7, 2014 · 5 comments
Labels
answered For when a question was asked and we referred to forum or answered it. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone
Backlog (Help wan...

Comments

@sudeshg
Copy link

sudeshg commented Aug 7, 2014

Hi,

How can i create custom log format in piwik. Actually i am unable to load log file through log_analytics.py script. I think i have to create new log format.

Sample log line:
xxx.xxx.xxx.x - - [21/Dec/2013:04:11:59 +0000] "GET /cds/phQoANRmI3SvEcHzQurM3vMGqI5u6iAYcEGjbwLrPZOLVbcOeys-qSavG9Fz03sZ1PAZkys48rhfHPrG4qhwZGB31Q5b4IdAgDaMYJO8inpJntM.?id=b2YxxoncjD7B-jzZLa75&expirationTime=1387858318631 HTTP/1.1" 200 6218 "-" "-" 15

Error Message:
Fatal error: cannot automatically determine the log format using the first 100000 lines of the log file.
Maybe try specifying the format with the --log-format-name command line argument.

Could you please send me documentation for creating custom log formats?

thanks,
Sudesh
@sgiehl
Copy link
Member

sgiehl commented Aug 7, 2014

Is that any standard log format or something you defined?
If it is an standard we might consider to add it to core. Otherwise please have a look at the script itself: https://github.com/piwik/piwik/blob/master/misc/log-analytics/import_logs.py#L256
You may need to add your custom format on your local copy

@ocontant
Copy link

ocontant commented Aug 7, 2014

I have the same issue with a what seems to be a legit common log format. Only mine had an extra space between the - - after the IP. I had to sed them out. I'm also looking to fix it in import script instead of using sed for 3To of gz file.

Do you think you could revise the regex of the common log format and take care to ignore extra join space? *like 

 "  -  -  "
instead of 
 " - - "
or if you don't want to put it in a review of your core, can you just give me the _COMMON_LOG_FORMAT regex or let me know how to add a new one if it is easy.



Thank you very much!  And thank you for coding this very good apps :)

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
      



  

  @mattab





  


    

      

  

    

        
    
    

  


      
          
    
            Copy link

      
    

  


  

      

      

  
  Member


      

  


  

    

      

      
          mattab
  

      

      

      commented


        Aug 9, 2014


      
    


  





      


        


  
    

      
          
@ocontan




Do you think you could revise the regex of the common log format and take care to ignore extra join space?




t 👍  if you would be able to provide pull request we would definitely merge it

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
          

  

    
  

  


    

        @mattab
mattab




          added
      

  Bug


    and removed
      

  Bug


  labels


    Aug 9, 2014

  





  

  

    
  

  


    

        @mattab
mattab




        added this to the 
  Short term milestone


    Aug 9, 2014

  










      

  
      



  

  @ocontant





  


    

      

  

    

        
    
    

  


      
          
    
            Copy link

      
    

  


  

      

      



      

  


  

    

  





      


        


  
    

      
          
Hi,


I tried, but my regex-fu is not enough strong.  I think some of the syntax you used I’m not familiar with.  My feeling is that the regex I would use to change it would not be as optimized as the one you are currently using.


If you could do it, that would be better in my opinion.  Wish my regex-fu was as strong as the guys who made it in the current version. J


From: Matthieu Aubry [mailto:notifications@github.com]

Sent: 9 août 2014 13:50

To: piwik/piwik

Cc: Olivier Contant

Subject: Re: [piwik] Custom log format for log analytics (#5947)


@ocontan


Do you think you could revise the regex of the common log format and take care to ignore extra join space?


t  👍 https://assets-cdn.github.com/images/icons/emoji/unicode/1f44d.png if you would be able to provide pull request we would definitely merge it


—

Reply to this email directly or view it on GitHub #5947 (comment) .  https://github.com/notifications/beacon/2682054__eyJzY29wZSI6Ik5ld3NpZXM6QmVhY29uIiwiZXhwaXJlcyI6MTcyMzIyNTgxMSwiZGF0YSI6eyJpZCI6MzkwNzA4MDZ9fQ==--15f2753e903f963483c498a81206817fe4ddf7e7.gif

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
          







      

  
      



  

  @mattab





  


    

      

  

    

        
    
    

  


      
          
    
            Copy link

      
    

  


  

      

      

  
  Member


      

  


  

    

      

      
          mattab
  

      

      

      commented


        Dec 18, 2014


      
    


  





      


        


  
    

      
          
Hi @ocontant do you mind creating a new issue specifically for the "extra space" thing? we will definitely fix it. Thanks

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
          

  

    
  

  


    

        @mattab
mattab




          closed this as completed


    Dec 18, 2014

  




  


  

  

    
  

  


    

        @mattab
mattab




        added
  the 

  answered

  For when a question was asked and we referred to forum or answered it.
 label


    Dec 18, 2014

  





  

  

    
  


  

      

    @sudeshg
sudeshg



    mentioned this issue
    
      Dec 18, 2014
    





  


  

      
  
  Closed

  







  













  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    answered

  For when a question was asked and we referred to forum or answered it.
  

    Enhancement

  For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.








      

  

    

        

    Projects
  


        




    None yet




  



      

  

    
  

    Milestone
  


      
    
  
    Backlog (Help wanted)
  





    
      
          


  

    
      

        
    

    Development
  




              
No branches or pull requests







    
  




        
      

    

  
      

  

    

      4 participants
    

    

        
          @mattab 
        
          @sgiehl 
        
          @ocontant 
        
          @sudeshg