@tsteur opened this Issue on January 13th 2022 Member

from https://forum.matomo.org/t/referral-exclusions/33582/25

The resolution suggested by Matomo in this article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ does not really address the underlying issue - it merely recategorizes the referrer from “website” to “direct entry”, which in reality isn’t the true referrer for the user’s visit.

Here is an example of what is happening.

  1. User clicks on an email link to visit the website.
  2. User browses website and adds items to basket (at this stage referrer is “Email campaign”)
  3. User completes the order but and website redirects to 3rd party payment provider (paypal/amazon etc)
  4. User returns to website (but now the referrer for the visit is reset from “Email campaign” to “website” or if you have set the referrer exclusions it will be set to “direct entry”. What we need is for it to remain as “Email Campaign”.

As a result we are over attributing the channel “direct entry” greatly (40% vs 15% on GA).

I'm not sure if we actually should call this feature "Referrer exclusion" as it's more like an "ignore refrerrer" or "keep the original referrer"? We wouldn't exclude this tracking request. We would still track that referrer but re-use the original referrer from the same visit. If it's from a different visit then we would use again a direct visit?

@tsteur commented on January 13th 2022 Member

@mattab Any idea what this feature could be named and how we would configure this? Not sure if we want to have this per site and globally or whether it be enough only one way etc. It could get confusing maybe for people that don't need this feature. Maybe it could be even a separate plugin?

@heurteph-ei commented on January 13th 2022

In my point of view, if a user :

  1. Comes from one referrer,
  2. Then browse the measured site (tracked by Matomo),
  3. Then browse out the measured site (eg. payment gateway or really leave the measured site)
  4. Then come back to the measured site during the 30 minutes delay (for any reason: payment or anything else)

Then the referrer should at least be the 1st one! (would it be possible to accumulate referrers?) And if the other referrer is considered as "direct entry” (what is done in case of excluded referrer) it means the only real referrer is still the 1st one...

Note also that in case of payment gateway, the returning back delay should not exceed short delay as in other ways, the payment token would probably expire... Then the scenario where the 1st referrer is a payment gateway should be very rare case...

@adsham commented on January 13th 2022

Thanks @tsteur for creating the issue here on my behalf.

I largely agree with @heurteph-ei - the main reason for referral data is to monitor how traffic is being driven to the website, especially if we attribute value from sales to a referrer. I can't really see any benefit of having "direct entry" as the referrer as it is misleading. However I do see a legitmate scenario that could be obscured by always using the first referrer.

Here are 2 examples

1)

  • Comes from Email referrer,
  • Then browse the measured site (tracked by Matomo),
  • Then browse out the measured site (eg. payment gateway or really leave the measured site)
  • Then come back to the measured site during the lifetime of a visit (e.g. 30mins)

In the above scenario - Paypal (for example) should be set up as a referral exclusion and therefore the 1st referrer should be used (Email).

2)

  • Comes from Email referrer,
  • Then browse the measured site (tracked by Matomo),
  • Then browse out the measured site (to search for a coupon code or referral link)
  • Then come back to the measured site during the lifetime of a visit (e.g. 30mins)

In the above scenario we may want to know both the former or the latter referrer.

With this in mind I would suggest either...

  • Making it a configuration option ("use first referrer" or "use latest referrer") at the measurable level ( in combination with the referral exclusions).

or

  • Adding a secondary field(s) to the underlying data so the user can choose to report on either first or last referrer - (and also make this new data available via the api)
@mattab commented on January 17th 2022 Member

What i don't understand yet, is that the scenarios above should already work meaning:
as long as the subsequent page views (after purchase) occur within 30min of the last pageview on the website, then the pageviews should be stored in the existing visit (with the correct original campaign). The fact that it doesn't work for some people is not clear.

@adsham @heurteph-ei Could you maybe confirm if the subsequent visits (the one with referrer "Paypal" or the direct entries) ones occured within 30 min of their last pageview?

(the only other reason i can think of is that people use a different browser to pay and that Matomo cannot recognise them as the same user, but that should be very edge case)

@adsham commented on January 17th 2022

Hi @mattab

Thanks for picking this up. You will see that several users have picked up on this issue here https://forum.matomo.org/t/referral-exclusions/33582/22 and possibly here too https://forum.matomo.org/t/what-can-explain-the-mysterious-too-many-direct-entries-phenomenon/31721/6

I have also just looked through our data and can confirm that there are no "delays" as you are suggesting. However I'm not really in a position to fully test this out at the moment (we do not have a test payment provider currently configured) perhaps @heurteph-ei or someone from the forum could do this? I will ask there too.

Could the resolution in the article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ be causing the original referrer to be overwritten with "direct entry"?

Many thanks once again, hopefully we can find a resolution to this soon!

@CryptoEU commented on January 17th 2022

Hello,

for me its very simple and there are different ways, how you can create the "correct" goal*1.

*1Goal: as already written. We have to know, from where the customer is coming from.
If Matomo delete now all referers (change to direct entry, using the payment gateway as referrer) thats just washing statistic with 0 sense. Nobody can track now, from where the customer really came then - ofc - he didnt came from PP, Klarna, CC etc.

I am not a technician, so I have no idea, why some 30 mins. timeframe is here on the table.
Sometimes its happend, that someone needs - because of distraction - longer with PP then 30 mins (PP=PayPal). The usual session for PP is running much longer also "all" shop sessions.
To me - 30 mins - are fine. Ofc some would be maybe out of this window, but thats 0.x% and ok if we dont have them tracked.

To make it lesser complicated:
Why not just making a "just track the first referrer" option in a XX min. timeframe?
or
Just track (and show!) all referrers.*2

*2 this would have the advantage, that you can directly see a fast overview of the daily payment gatesways too, without needing your ERP.

Exlucde paymentgateways and putting all referrers as direct entry or use the gateway as referrer is for nobody really helpful.

@frimipiso commented on January 17th 2022

Hey there,

it is great that this issue is picket up now. I have been experiencing this problem for a long time now.

Here is our setup:

  • Shopware 5.6 Shopsystem with paypal as payment provider
  • When the user goes to paypal for payment and returns to our Shopware shop, a new session is started with referrer paypal.

This completely messes up all conversion attribution, as most customers use paypal.

All customers return from paypal within the 30 minute timespan.

I would opt for an "attribute to first/last referrer" configuration in matomo.

Thanks for taking care of this and let me know if I can be of further help.

Cheers

Jens

P.S.: If I would use the ignore referrer option, the new session would have a direct access referrer which even more messes up attribution, since I cannot identify anymore that paypal messed up the referrer.

@SimonNL commented on January 21st 2022

Coming from GA and having struggled with the same issue there years ago, I quickly ran into this again with Matomo. Specifically the 'payment service provider' scenario. With GA we started with exclusion list but could finally solve this for good by adding the 'ignore_referrer' parameter to the PSP return/error/pending page (e.g. /thank-you?ignore_referrer=true) so regardless of the payment method, the site in between is always ignored. This works better than referral exclusion lists because the payment urls tend to change over time and new ones are easily overlooked. A lot of e-commerce sites will already be using this parameter so would supporting this not solve the issue for many cases?

Powered by GitHub Issue Mirror