Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug during import when QueuedTracking enabled #12340

Closed
HubertNodzak opened this issue Dec 7, 2017 · 9 comments
Closed

Bug during import when QueuedTracking enabled #12340

HubertNodzak opened this issue Dec 7, 2017 · 9 comments
Labels
answered For when a question was asked and we referred to forum or answered it. Bug For errors / faults / flaws / inconsistencies etc.
Milestone

Comments

@HubertNodzak
Copy link

HubertNodzak commented Dec 7, 2017

Hello,
PIWIK: 3.2.1
QueuesTracking: 3.1.0

During import json files from Apache (when above plugins is enabled), there is an error:
2017-12-07 08:13:09,741: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:09,741: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:09,872: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:09,872: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:09,956: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:09,956: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,093: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,093: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,180: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,180: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,263: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,263: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,403: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,403: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,537: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,538: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,668: [INFO] bulk tracking returned invalid JSON 2017-12-07 08:13:10,668: [INFO] tracker response: GIF89a�!�,D; 2017-12-07 08:13:10,753: [INFO] bulk tracking returned invalid JSON

OS Centos 7 (on both systems).

@sgiehl
Copy link
Member

sgiehl commented Mar 9, 2018

Do you still have this issue? If so would you mind to share the command you used to import the log files?
We would aim to fix this if possible to reproduce

@sgiehl sgiehl added the Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. label Mar 9, 2018
@jsandner
Copy link

jsandner commented Jan 17, 2019

I experienced this error too on some Log-Lines. I am using format "common_complete" (Apache vhost_combined). The error occurs when:

@nomandera
Copy link

nomandera commented Jan 31, 2019

I am consistently seeing this error.

There are several tickets about this issue but none clarify if this is a warning or an error. Specifically are the logs still bulk imported correctly?

I have tested a number of things so far:

  • Local import using Debian and official apt package
  • Remote import using windows and official python
  • Remote import using windows and cygwin
  • Local and remote single file and wildcard multiple file import
  • Logs from four differernt IIS 7 cluster nodes
  • Logs from three different sites
  • With --recorders=1 and --recorders=2
  • With and without --token-auth=
  • With and without --enable-http-errors
  • With and without --enable-http-redirects
  • With and without --enable-static
  • With and without --enable-bots
  • With and without --accept-invalid-ssl-certificate

None of the above makes any difference at all.

This setup is four clustered front end boxes, one back end box for private access, one redis server and one sql server

  • Matomo version: 3.7.0 (official Deb)
  • MySQL version: 10.1.37-MariaDB-0+deb9u1
  • PHP version: 7.0.33-0+deb9u1

All the standard addons plus

  • QueuedTracking
  • HeatmapSessionRecording
  • MediaAnalytics
  • BulkTracking (default but just listing for clarity)

@tsteur
Copy link
Member

tsteur commented Jan 31, 2019

I wonder if it would be helpful to have a flag in log importer like --ignore-queued-tracking which appends a queuedtracking=0 to the tracking request and tells Matomo to avoid the queued tracking but instead execute them directly. This queuedtracking=0 would be also helpful when directly wanting to test some tracker issues etc.

When using log importer, there's usually not much benefit (or none, it actually adds some overhead) of using queued tracking so that can be useful. This way the output would be correctly.

Queued tracking can currently not really return a valid response because it wouldn't know whether any tracking requests were imported correctly or not. Maybe we would simply always need to append &queuedtracking=0 when importing data.

@tsteur
Copy link
Member

tsteur commented Jan 31, 2019

@mattab mattab added this to the 3.10.0 milestone Feb 12, 2019
@mattab mattab added Bug For errors / faults / flaws / inconsistencies etc. and removed Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. labels Feb 12, 2019
@nomandera
Copy link

Just to follow up on this. I cannot test on the specific problem environment this now as the official Matomo apt package is still at 3.7.0 which sticks QueuedTracking at 3.3.1

Please update Piwik 3.7.0 to a newer version, Piwik >=3.8.0-rc2,<4.0.0-b1 is required.

Since this is a production environment I am limited by policy to manually bypass/update.

When the updates appear I will retest.

@tsteur
Copy link
Member

tsteur commented Apr 22, 2019

I think this issue can be closed? What is left todo is matomo-org/matomo-log-analytics#240 and some docs

@nomandera
Copy link

Coincidentally I was working on this in the last few days.

I have tried everything I can, using all the latest advice and code, to stop the gif being returned rather than valid JSON. With all the recent changes the import actually works correctly albeit the weird gif error still persists.

I dont know if it is something specific to my setup but at least now it works.

@tsteur
Copy link
Member

tsteur commented Jul 18, 2019

@mattab see #12340 (comment) I suppose we can close the issue as only matomo-org/matomo-log-analytics#240 is left?

@mattab mattab closed this as completed Jul 18, 2019
@mattab mattab added the answered For when a question was asked and we referred to forum or answered it. label Jul 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
answered For when a question was asked and we referred to forum or answered it. Bug For errors / faults / flaws / inconsistencies etc.
Projects
None yet
Development

No branches or pull requests

6 participants