Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking API: provide a new JSON Tracker API #4625

Open
mattab opened this issue Feb 4, 2014 · 6 comments
Open

Tracking API: provide a new JSON Tracker API #4625

mattab opened this issue Feb 4, 2014 · 6 comments
Labels
c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.

Comments

@mattab
Copy link
Member

mattab commented Feb 4, 2014

Our tracking API is fully documented at: Tracking API - tracking users analytics

It is also fully featured, and yet still a bit unstructured. Because we have added parameters one after the other when they were needed for useful features, now we realize that they could be named better, and be more structured.

Now that JSON has gone mainstream, it is natural to provide a pure JSON proxy to our good old named parameters. In the making, we could introduce structure in elements like events or ecommerce interactions (define each as a JSON object).

New API names proposals (to be discussed)

  • rec -> record
  • idsite -> idSite
  • url -> actionUrl
  • action_name -> actionName
  • gt_ms -> actionGenerationTimeMs
  • _id -> visitId
  • _cvar -> pageCustomVariables
  • urlref -> urlReferrer
  • _idvc -> visitCount
  • _viewts -> tsPreviousVisit
  • _idts -> tsFirstVisit
  • _rcn -> campaignName
  • _rck -> campaignKeyword
  • res -> resolution
  • h+m+s -> merge into userLocalTime
  • ua -> userAgent
  • lang -> userLanguage?
  • cvar -> pageCustomVariables
  • search -> searchKeyword
  • search_cat -> searchCategory
  • search_count -> searchResultsCount
  • idgoal -> idGoal
  • group all ec_* into "ecommerce" object
  • group all location info (country, region, city, latitude/longitude) into "location" object
  • group all events e_* into an "event" object
@mattab mattab added this to the 2.x - The Great Piwik 2.x Backlog milestone Jul 8, 2014
@mattab
Copy link
Member Author

mattab commented Sep 9, 2014

See also #6000 Build a RESTful API for Piwik

@mattab mattab changed the title Tracking API: provide a new JSON piwik.php tracking API Tracking API: provide a new JSON Tracker API Sep 9, 2014
@mattab mattab added the c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. label Sep 9, 2014
@mattab mattab added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. and removed Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. labels Oct 11, 2014
@Spone
Copy link

Spone commented Oct 14, 2014

Hi @mattab ,

I've landed there from matomo-org/developer-documentation#28

I think you should consider using POST request for the Tracking API. Maybe you can also allow GET requests in case the client cannot do POST requests (Google Analytics does that), but the better HTTP verb to use in this case in POST (see here).

Some other thoughts:

apiv will now be passed in the URL (for instance http://mypiwik.com/api/v2/) if I understand correctly.

lang -> userLanguage?

I would name it locale or userLocale, and allow just ISO 639 (language) code (optionally with an ISO 3166-1 alpha-2 (2-letter country) code).

Also, I don't know if you're familiar with JSON Schema, but it may be a good way to validate such user-submitted data.

Best,
Hans

@fulstadev
Copy link

fulstadev commented Jan 31, 2024

Hi @mattab,

Is this feature planned anytime soon? It is an issue when trying to integrate Matomo as a 'destination' in customer data platforms, particularly when it comes to authentication, which is required to forward the correct timestamp and IP of an event in such cases. For safety reasons, it's better to forward the token_auth only within the POST request body, as your docs also say. Yet this is unfortunately not possible with tools that only allow to forward data with the application/json content-type, as long as Matomo's Tracking API only understands x-www-form-urlencoded POST requests.

It would also be more user-friendly to build variable request payloads, instead of needing to worry about concatenating all query parameters followed by URL-Encoding. A great plus for people who cannot code and construct their requests within a CDP, filling the according key-value pairs into text fields. That part of this comment is a detail, of course, but making it easier for no-devs to use your tracking API, why not (just a thought).

But the problem with the authentication described above technically forces us to make a roundtrip to our own server just because of the Content-Type and the token_auth, except we're missing something? The only other solution we see is exposing to token_auth as query parameter in the HTTP requests that forward events received in our CDP to Matomo?

@mattab
Copy link
Member Author

mattab commented Jan 31, 2024

@fulstadev Thanks for your feedback! I was wondering about this: people who cannot code and construct their requests within a CDP, filling the according key-value pairs into text fields. It's a detail, of course, but it would it easier for no-devs to use your tracking API, too. -> Could you maybe share some screenshots showing, what the experience is like for you, when using your CPD and trying to construct the request? What tool do you use?

@fulstadev
Copy link

fulstadev commented Jan 31, 2024

@mattab Sure, no problem. If the Matomo API could understand JSON payloads, you could use the default masks these tools often offer, in a simple way (the 'webhook URL' would be the URL representing a matomo tracking API call, and what you see in the image with the '$'-prefix are (JSON) variables from the event incoming into the CDP.

Bildschirmfoto 2024-01-31 um 21 13 34

But because Matomo only understands the parameters via URL, you instead have to prepare the URL using sth like:

Bildschirmfoto 2024-01-31 um 21 19 26

This is fine as long as you don't have any spaces or whatever, which is when URL-encoding is needed. The tool we use actually does not offer a URL-encoder for the generation of the webhook URL to be used as data pipeline destination. Consequently, the only solution we see is to generate the URL (url-encoded) to be used for the matomo tracking API call entirely within our backend, and send that URL along as attribute of any event we send to our CDP. Then we can use the ready URL within the CDP, and create an automated pipeline. But note that this still does not solve the issue of the token_auth, as the HTTP request the tool sends is an application/json request (which by the way also eliminates any URL-encoding issues anyway). So either we still send the token_auth as GET parameter, or we cannot set the timestamp and the IP of a tracked event, which means that the timestamp and IP of every event that is forwarded from our CDP into Matomo are actually incorrect (for example categorizing the incoming event to originate from the country of the infrastructure of the CDP, we could already confirm that when weird sequences of country-A <-> country-B showed up in our events history).

Another thing is that Matomo Tracking API URLs quickly become long. If no-devs have to use sth like 20 placeholders within it, join them with ampersands, and then watch out that everything is url-encoded (if the code permits it), it can be tedious to build this with pure placeholders, again for non-devs. It also happened to our devs already that a pretty long Matomo tracking URL failed to be executed because an ampersand was missing in front of one of the placeholders, and the resulting matomo request was treated as a 400 and broke the flow. Stupid stuff like that may occur more easily, that's the experience we can share.

So we're not sure how you weigh these issues, but the token issue is of greater importance for us..? Thanks anyways for your quick reaction!

And please don't misunderstand this post, we really appreciate matomo and its features, a lot.

A last note regarding the UX with CDPs though is that we've been working with more than three of the rather big players, and they were not offering an out-of-the-box integration with matomo, as with other analytics tools. Hence the only workaround for a data pipeline through to matomo is an actual webhook endpoint as data pipeline target (probably already at the dev-level here). We just think that a JSON-version of your API would make that integration much easier.

@mattab
Copy link
Member Author

mattab commented Feb 2, 2024

Thanks for the feedback @fulstadev, that's appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Projects
None yet
Development

No branches or pull requests

3 participants