@tsteur opened this Issue on January 13th 2022 Member

from https://forum.matomo.org/t/referral-exclusions/33582/25

The resolution suggested by Matomo in this article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ does not really address the underlying issue - it merely recategorizes the referrer from “website” to “direct entry”, which in reality isn’t the true referrer for the user’s visit.

Here is an example of what is happening.

  1. User clicks on an email link to visit the website.
  2. User browses website and adds items to basket (at this stage referrer is “Email campaign”)
  3. User completes the order but and website redirects to 3rd party payment provider (paypal/amazon etc)
  4. User returns to website (but now the referrer for the visit is reset from “Email campaign” to “website” or if you have set the referrer exclusions it will be set to “direct entry”. What we need is for it to remain as “Email Campaign”.

As a result we are over attributing the channel “direct entry” greatly (40% vs 15% on GA).

I'm not sure if we actually should call this feature "Referrer exclusion" as it's more like an "ignore refrerrer" or "keep the original referrer"? We wouldn't exclude this tracking request. We would still track that referrer but re-use the original referrer from the same visit. If it's from a different visit then we would use again a direct visit?

@tsteur commented on January 13th 2022 Member

@mattab Any idea what this feature could be named and how we would configure this? Not sure if we want to have this per site and globally or whether it be enough only one way etc. It could get confusing maybe for people that don't need this feature. Maybe it could be even a separate plugin?

@heurteph-ei commented on January 13th 2022

In my point of view, if a user :

  1. Comes from one referrer,
  2. Then browse the measured site (tracked by Matomo),
  3. Then browse out the measured site (eg. payment gateway or really leave the measured site)
  4. Then come back to the measured site during the 30 minutes delay (for any reason: payment or anything else)

Then the referrer should at least be the 1st one! (would it be possible to accumulate referrers?) And if the other referrer is considered as "direct entry” (what is done in case of excluded referrer) it means the only real referrer is still the 1st one...

Note also that in case of payment gateway, the returning back delay should not exceed short delay as in other ways, the payment token would probably expire... Then the scenario where the 1st referrer is a payment gateway should be very rare case...

@adsham commented on January 13th 2022

Thanks @tsteur for creating the issue here on my behalf.

I largely agree with @heurteph-ei - the main reason for referral data is to monitor how traffic is being driven to the website, especially if we attribute value from sales to a referrer. I can't really see any benefit of having "direct entry" as the referrer as it is misleading. However I do see a legitmate scenario that could be obscured by always using the first referrer.

Here are 2 examples

1)

  • Comes from Email referrer,
  • Then browse the measured site (tracked by Matomo),
  • Then browse out the measured site (eg. payment gateway or really leave the measured site)
  • Then come back to the measured site during the lifetime of a visit (e.g. 30mins)

In the above scenario - Paypal (for example) should be set up as a referral exclusion and therefore the 1st referrer should be used (Email).

2)

  • Comes from Email referrer,
  • Then browse the measured site (tracked by Matomo),
  • Then browse out the measured site (to search for a coupon code or referral link)
  • Then come back to the measured site during the lifetime of a visit (e.g. 30mins)

In the above scenario we may want to know both the former or the latter referrer.

With this in mind I would suggest either...

  • Making it a configuration option ("use first referrer" or "use latest referrer") at the measurable level ( in combination with the referral exclusions).

or

  • Adding a secondary field(s) to the underlying data so the user can choose to report on either first or last referrer - (and also make this new data available via the api)
@mattab commented on January 17th 2022 Member

What i don't understand yet, is that the scenarios above should already work meaning:
as long as the subsequent page views (after purchase) occur within 30min of the last pageview on the website, then the pageviews should be stored in the existing visit (with the correct original campaign). The fact that it doesn't work for some people is not clear.

@adsham @heurteph-ei Could you maybe confirm if the subsequent visits (the one with referrer "Paypal" or the direct entries) ones occured within 30 min of their last pageview?

(the only other reason i can think of is that people use a different browser to pay and that Matomo cannot recognise them as the same user, but that should be very edge case)

@adsham commented on January 17th 2022

Hi @mattab

Thanks for picking this up. You will see that several users have picked up on this issue here https://forum.matomo.org/t/referral-exclusions/33582/22 and possibly here too https://forum.matomo.org/t/what-can-explain-the-mysterious-too-many-direct-entries-phenomenon/31721/6

I have also just looked through our data and can confirm that there are no "delays" as you are suggesting. However I'm not really in a position to fully test this out at the moment (we do not have a test payment provider currently configured) perhaps @heurteph-ei or someone from the forum could do this? I will ask there too.

Could the resolution in the article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ be causing the original referrer to be overwritten with "direct entry"?

Many thanks once again, hopefully we can find a resolution to this soon!

@CryptoEU commented on January 17th 2022

Hello,

for me its very simple and there are different ways, how you can create the "correct" goal*1.

*1Goal: as already written. We have to know, from where the customer is coming from.
If Matomo delete now all referers (change to direct entry, using the payment gateway as referrer) thats just washing statistic with 0 sense. Nobody can track now, from where the customer really came then - ofc - he didnt came from PP, Klarna, CC etc.

I am not a technician, so I have no idea, why some 30 mins. timeframe is here on the table.
Sometimes its happend, that someone needs - because of distraction - longer with PP then 30 mins (PP=PayPal). The usual session for PP is running much longer also "all" shop sessions.
To me - 30 mins - are fine. Ofc some would be maybe out of this window, but thats 0.x% and ok if we dont have them tracked.

To make it lesser complicated:
Why not just making a "just track the first referrer" option in a XX min. timeframe?
or
Just track (and show!) all referrers.*2

*2 this would have the advantage, that you can directly see a fast overview of the daily payment gatesways too, without needing your ERP.

Exlucde paymentgateways and putting all referrers as direct entry or use the gateway as referrer is for nobody really helpful.

@frimipiso commented on January 17th 2022

Hey there,

it is great that this issue is picket up now. I have been experiencing this problem for a long time now.

Here is our setup:

  • Shopware 5.6 Shopsystem with paypal as payment provider
  • When the user goes to paypal for payment and returns to our Shopware shop, a new session is started with referrer paypal.

This completely messes up all conversion attribution, as most customers use paypal.

All customers return from paypal within the 30 minute timespan.

I would opt for an "attribute to first/last referrer" configuration in matomo.

Thanks for taking care of this and let me know if I can be of further help.

Cheers

Jens

P.S.: If I would use the ignore referrer option, the new session would have a direct access referrer which even more messes up attribution, since I cannot identify anymore that paypal messed up the referrer.

@SimonNL commented on January 21st 2022

Coming from GA and having struggled with the same issue there years ago, I quickly ran into this again with Matomo. Specifically the 'payment service provider' scenario. With GA we started with exclusion list but could finally solve this for good by adding the 'ignore_referrer' parameter to the PSP return/error/pending page (e.g. /thank-you?ignore_referrer=true) so regardless of the payment method, the site in between is always ignored. This works better than referral exclusion lists because the payment urls tend to change over time and new ones are easily overlooked. A lot of e-commerce sites will already be using this parameter so would supporting this not solve the issue for many cases?

@adsham commented on January 28th 2022

I'd like to provide some examples to demonstrate the overreporting of direct entries in one of our clients, I can confidently say the “direct entry” attribution is far too high using all of the Matomo attribution techniques.

Here is a monthly view
image

Here is a yearly view
image

Here is a segmented yearly view for new visitors (I wouldn’t expect new users to be showing such high direct entries)
image

Therefore I’m fairly certain something is not attributing correctly, whether it’s us the users (it seems there are many users as above having similar issues) who have not understood so something or whether Matomo tracking is not attributing as it should in some cases I’m not sure - any suggestions are welcome.

@ts1985 commented on February 10th 2022

We have the same problem. I think most have the same problem and it's important to find a solution for this. The conversions are thus virtually useless, since it distorts everything.
Without configuration you have conversions to the payment providers, but you want to know where the visitor originally came from and not which payment provider he used.
If you put the payment providers on the referral exclusion list, then you have the conversions on "Direct Entry" which doesn't make sense because the customer might have come from search engines, websites etc..

Hope somebody will find a solution soon.

@adsham commented on February 10th 2022

mattab , tsteur

What i don't understand yet, is that the scenarios above should already work meaning: as long as the subsequent page views (after purchase) occur within 30min of the last pageview on the website, then the pageviews should be stored in the existing visit (with the correct original campaign). The fact that it doesn't work for some people is not clear.

@adsham @heurteph-ei Could you maybe confirm if the subsequent visits (the one with referrer "Paypal" or the direct entries) ones occured within 30 min of their last pageview?

(the only other reason i can think of is that people use a different browser to pay and that Matomo cannot recognise them as the same user, but that should be very edge case)

Perhaps then this is a bug rather than an enhancement? - it seems that many of us are experiencing this issue.

@ts1985 commented on February 10th 2022

I can give you these details:

  • No they don't use another browser to pay
  • It's not just PayPal. We have several Payment Providers, all with the same problem.
@adsham commented on March 10th 2022

Any update on this? It seems to be pushed back again into a later milestone.

The referral data is an important part of our service and we have recently seen a client leave to a competitor because of the inaccurate data in our reports (from Matomo).

The initial issue was raised almost 3 years ago in the forum https://forum.matomo.org/t/referral-exclusions/33582/25 and this Github issue was raised 2 months ago and pushed back into milstone 4.10 (4.8 released today).

Please could we have some committment as to when this will be addressed as it's getting harder to explain to clients that the issue is being looked at but cannot tell them when it will be looked into/fixed.

Many thanks in advance!
Adham

@Demichev commented on April 20th 2022

Hello.
We have the same issue. I temporarily modified Matomo code (return false immediately in onExistingVisit() for Referrers Columns classes). We are testing it now. I think it will solve the problem for us but it is good to have an official option.
What do you think?

@adsham commented on April 20th 2022

Hello. We have the same issue. I temporarily modified Matomo code (return false immediately in onExistingVisit() for Referrers Columns classes). We are testing it now. I think it will solve the problem for us but it is good to have an official option. What do you think?

Hi @Demichev - this is interesting, we are still waiting on an update. We have lost business over this and have built some of our services around the referrer functionality so it is important that a solution is found. It would be great if you could share your experience and solution if you find one, perhaps it will the developers with an official solution?

Also @justinvelluppillai What is the "backlog" milestone? Will these be actioned? Do we have an idea of timings? Many thanks

@Demichev commented on April 21st 2022

Hi @Demichev - this is interesting, we are still waiting on an update. We have lost business over this and have built some of our services around the referrer functionality so it is important that a solution is found. It would be great if you could share your experience and solution if you find one, perhaps it will the developers with an official solution?

Hello @adsham. I should clarify my approach. I did not try to exclude payment site referrers, I disabled all referrers from late actions. Only referrer from the first action of each visit is recorded if any. Payment site still can be set as referrer if visit is expired. We are experimenting with visit length up to one day.
I can share my changes but they should be restored each time Matomo is updated. We have some other changes and it is real pain to update Matomo now. So I decided to try to write here some feature requests.

@adsham commented on April 21st 2022

Hi @Demichev - this is interesting, we are still waiting on an update. We have lost business over this and have built some of our services around the referrer functionality so it is important that a solution is found. It would be great if you could share your experience and solution if you find one, perhaps it will the developers with an official solution?

Hello @adsham. I should clarify my approach. I did not try to exclude payment site referrers, I disabled all referrers from late actions. Only referrer from the first action of each visit is recorded if any. Payment site still can be set as referrer if visit is expired. We are experimenting with visit length up to one day. I can share my changes but they should be restored each time Matomo is updated. We have some other changes and it is real pain to update Matomo now. So I decided to try to write here some feature requests.

Thanks @Demichev - your approach would satisfy my requirements as it's the initial referrer which is most important in our case. Unfortunately we use the hosted service so I cannot make these changes to our tracking.

@adsham commented on April 21st 2022

Here is another example of this issue (or related issue) - a new project we are doing a "proof of concept" - unfortunately it's recording intra-website campaigns which isn't very useful for what we are trying to demonstrate to a potential client. This may be the intended behaviour, but I would suggest that it is the first referrer to the site that is important in most cases. Perhaps a FirstReferrer & LastReferrer field would take care of this?

image

(Tiles & carousel campaign mediums here are links clicked within the website)

@adsham commented on April 22nd 2022

Hi @mattab

Could you provide an update on this, I'm receiving a lot of pressure both internally and externally and I'm still none the wiser as to when this will be investigated/resolved. I appreciate you have other priorities and issues, but we are currently paying $300-$500 a month for a service that isn't providing us with the data that it should.

This issue was raised on the other forum in 2019 and this github issue in January - we are also a small team so we appreciate that things need to be prioritized and scheduled, but I can't even tell our clients or my manager that this will be resolved by the end of the month, the end of next month, next year or never.

I would also say this doesn't seem to be an isolated issue, many users have reported the same issue.

@justinvelluppillai commented on April 22nd 2022 Contributor

Hi @adsham I agree this issue is important and I can see how moving it around the milestones has been unhelpful in terms of providing clarity about when it will be done. I'll see if we can provide a timeframe for this being addressed soon.

@adsham commented on April 25th 2022

Quick way to recreate this issue. Rather than a later referrer “overwriting” the first referrer – it is generating a new visit when a new referrer/campaign is encountered during the visit – and this subsequent visit will get associated with the conversion/goal.

1) On a Matomo tracked website (standard javascript tracking). Click a link or visit the website with a testable referrer. In this example I visit…
https://XXXXX.co.uk/?utm_campaign=testA
Looking in the real time tracking I can see the visit (Visit 4 – April 25th 2022 13.40:46)

image

2) In the same browser session, I edit the final character of the URL, so that the campaign now becomes testB https://XXXXX.co.uk/?utm_campaign=testB and press return. This now generates a new visit – in this example you can see visit 5 (April 25 2022 13.41:46).

image

In a real-world situation what this means is that it is the latter referrer that gets associated to the visit that goes on to make an ecommerce order or goal. When the “real” referrer in the test above was testA and should be associated with the subsequent ecommerce order/goal (unfortunately testB will get the credit in this example). My understanding from the documentation and response above is this should be tracked as the same visit as it’s in the same browser tab and within 30 mins (with no other activity between each action). I have tested my test above without the utm_campaign tag and all page views are correctly allocated to the 1 visit.

@Demichev commented on April 25th 2022

@adsham
There are two settings in Matomo config. I don't know if they are available for change in hosted version. I think you need create_new_visit_when_campaign_changes = 0.

; if set to 1, actions that contain different campaign information from the visitor's ongoing visit will
; be treated as the start of a new visit. This will include situations when campaign information was absent before,
; but is present now.
create_new_visit_when_campaign_changes = 1

; if set to 1, actions that contain different website referrer information from the visitor's ongoing visit
; will be treated as the start of a new visit. This will include situations when website referrer information was
; absent before, but is present now.
create_new_visit_when_website_referrer_changes = 0

I want the option to use referrer from the first action only. I think this option has sense.

@justinvelluppillai commented on April 25th 2022 Contributor

The create_new_visit_when_campaign_changes setting seems relevant here. There is also the option to use first referrer attribution for conversions with _paq.push(['setConversionAttributionFirstReferrer', true]); (as described here: https://matomo.org/faq/general/faq_106/).

@heurteph-ei commented on April 26th 2022

@justinvelluppillai reading the FAQ, I think the setConversionAttributionFirstReferrer should work on several visits. Not in the case of a single visit. The problem in this thread is during a single visit (within the 30 min timeframe):

  • Referrer A
  • Some Pages (at this time the referrer is A)
  • Outlink to the payment provider
  • Thankyou page _(now the referrer is the payment provider)

The rule (for a visit) should be: if the page is the first one of the visit, then the referrer (or campaign or etc.) must be registered. In other case, the referrer shouldn't be changed...

@justinvelluppillai commented on April 26th 2022 Contributor

That's correct re setConversionAttributionFirstReferrer. We will look into the other problem when we review this issue next milestone - also the first comment's issue of many of them maybe being attributed to Direct Entry seems unexpected.

@adsham commented on April 26th 2022

Hi @justinvelluppillai

It would also be useful to know how those of us with the hosted Matomo service (Innocraft) can change these config settings. As far as I am aware there is no option to do this, but can these settings be changed for us server-side? If so what is the process for doing this?

Thanks

@justinvelluppillai commented on April 26th 2022 Contributor

Hi @adsham, Matomo Cloud customers can get in touch with our support team about changing config settings that can't be changed in the UI.

@adsham commented on May 4th 2022

HI @justinvelluppillai

How do Matomo Cloud users contact support? I sent an email 7 days ago with no response? It's really frustrating with clients waiting. There is no ticket number or acknowledgement.

May I suggest a support ticket system, where we can track progress? On the website it states that Matomo Cloud offers "In-depth" support, but currently we are feeling under supported, despite paying $300-$600 a month.

For instance - If we require a config change as above.

1) What is the request process?
2) How long should we anticipate before an acknowledgement?
3) Can we expect an estimated timeframe for completion?
4) Are their any SLA commitments from Matomo?

Many thanks
Adham

@frimipiso commented on May 5th 2022

It's too bad that there is still no solution to this problem.

We want to start using GoogleAds now, but we cannot attribute conversions to Ad campaigns if the customers pay via Paypal.

So we are forced to move back to Google Analytics which is a shame because we heavily adapted our shops for Matomo.

But we will essentially be wasting money if we cannot attribute sales to campaigns ...

Cheers

Jens

@mattab commented on May 9th 2022 Member

Sorry for the delay it's taking to address this issue, we agree it is impactful. it is now high priority and will be investigated by the team this month. Stay tuned for some updates & thank you for all your reports :+1:

@sgiehl commented on May 10th 2022 Member

Ok. First of all I will summarize the current behavior that is implemented in Matomo regarding referrer handling:

Whenever the referrer details changes within a visit Matomo currently forces a new visit in this cases:

  • A website referrer was detected and config create_new_visit_when_website_referrer_changes = 1
  • A campaign change was detected, the current referrer wasn't a direct entry and config create_new_visit_when_campaign_changes = 1

If the details changed (to something other than direct entry), but no new visit was forced (due to config) the referrer information will be updated in the current visit if the visit wasn't a direct entry initially.


So looking through all the issue comments my understanding would be, that we need some kind of exclusion list. If a user is coming from a url/host on that list, this action will be treated as Direct Entry, which will then prevent updating the referrer details or forcing a new visit.

I would add that list the way we are currently handling query parameter exclusion. So that means there is a global configuration option, but also the possibility to add urls/hosts on a per site level.

I will now start working on that. If there is any additional input or requirements that I should take into account when implementing that. Let me know.

@ts1985 commented on May 10th 2022

I don't think that this is the solution. To explain:

  1. User visits example.com (doesn't matter if it's direct entry or a referrer. Let's say it was direct entry for this example)
  2. User buys something and is forwarded to paypal.com
  3. User comes back after he paid to example.com and reached the goal

Current issue:
If you check now the reached goal, you see, that this visitor reached the goal from paypal.com and not from direct entry. This information is completely wrong. Your goal is now full of visits from paypal.com and other payment providers. But you want to know where the visitor was from and not what payment provider he used during his visit.

If you now integrate an exclusion list and add paypal.com on this list and "this action will be treated as Direct Entry" the problem is, that all visitors using paypal.com as payment method are as "direct entry" in the goal information. But maybe it was not a direct entry, it was a visitor from google.com?

It's really important to make sure that this issue is solved and no other issues are build with this solution. Otherwise more wrong data is in the system.

@Demichev commented on May 10th 2022

Why should be referrer updated in the first place? What if I want one visit with the original (first) referrer?

@heurteph-ei commented on May 10th 2022

Why not just 2 options (at site level):

  • use_last_referrer_as_visit_referrer (default = 0)
    • case 0: same behavior as now (the last referrer overwrites any other referrer)
    • case 1: the first referrer stays the only referrer for the whole visit (even if it was direct entry?)
  • use_last_campaign_as_visit_referrer` (default = 0)
    • same idea as the other option...

(you have to decide which ones between use_last_*_as_visit_referrer and create_new_visit_when_*_changes has the most precedence...)

@sgiehl commented on May 10th 2022 Member

@heurteph-ei I'm not sure if that would allow all required use cases. It might be possible someone wants the referrer to be updated or a new visit to be forced when the referrer changes, but still wants payment proxies (or similar) not to trigger this behavior.
Having a list of urls that are kind of treated as direct entry, should give more flexibility I guess.

But for sure we could also discuss about a configuration value to disallow overwriting the referrer at all (if no new visits are forced).

@ts1985 commented on May 10th 2022

Why not just 2 options (at site level):

  • use_last_referrer_as_visit_referrer (default = 0)

    • case 0: same behavior as now (the last referrer overwrites any other referrer)

Makes no sense. As we here already discussed, the current behavior results into wrong information. Leaving this bug as default would therefore not be helpful.

@sgiehl commented on May 10th 2022 Member

@ts1985 My explanation might not have been detailed enough. A tracked action that is handled as direct entry will never update the referrer stored in database. So when the user comes back from the payment provider, within the visit time frame, the action would then no longer trigger a new visit, nor would it update the referrer that was initially set with the visit (if the referrer is ignored, as described above).

@heurteph-ei commented on May 10th 2022

@ts1985 this was proposed for 2 reasons:

  1. This has been implemented like this, so I assume there was a good reason... Also as I am not in the head of everybody, maybe some business need to keep only the last referrer...
  2. By default, we must not change the behavior of existing Matomo instances (in case of case 1). But for new instances, you're right, the default value should be 1 (as I think that it is the behavior most user would expect)

@sgiehl This is for the case where some "in-process provider" (like payment providers) can also be a referrer... That can't be excluded...

@mattab commented on May 11th 2022 Member

Reading the recent comments, and as suggested by you, the problem is that the ecommerce/checkout subdomain (paypal.com) appears as a "website" referrer. And because Matomo by default credits ecommerce conversion to the last non direct referrer, then all ecommerce attribution reports show the paypal.com domain (making all attribution reports useless because then it won't show you the actual acquisition channel used by the user...). then the ecommerce attribution works fine. But the "Multi Channel" reports will show the paypal.com ( https://matomo.org/faq/reports/set-up-and-analyse-multi-channel-conversion-attribution-reports/ )

So Instead if Matomo had a list of domains used by the most common ecommerce/checkout platforms (including the paypal.com), then Matomo could ignore this domain and instead make the visit a "direct entry".

We could keep the list with these domains in the global.ini.php so people could also override the list if they want (or empty the list, if for some reasons they want to see the paypal.com referrer). We could add new domains to the list on request or via pull requests. And it would make sense to have this become the default behavior by default.

Maybe that could work nicely, but even when it's completed, i'm not yet 100% sure that if fixes everyone's issues here - to be confirmed?

@adsham commented on May 11th 2022

Reading the recent comments, and as suggested by you, the problem is that the ecommerce/checkout subdomain (paypal.com) appears as a "website" referrer. And because Matomo by default credits ecommerce conversion to the last non direct referrer, then all ecommerce attribution reports show the paypal.com domain (making all attribution reports useless because then it won't show you the actual acquisition channel used by the user...). BTW one solution is to use Multi Channel attribution to get this information: https://matomo.org/faq/reports/set-up-and-analyse-multi-channel-conversion-attribution-reports/

But obviously it's a big issue, and it could make a lot of sense to have a "special" handling of websites acting as checkout/ecommerce platforms (such as paypal.com).

So Instead if Matomo had a list of domains used by the most common ecommerce/checkout platforms (including the paypal.com), then Matomo could ignore this domain and instead make the visit a "direct entry".

We could keep the list with these domains in the global.ini.php so people could also override the list if they want (or empty the list, if for some reasons they want to see the paypal.com referrer). We could add new domains to the list on request or via pull requests. And it would make sense to have this become the default behavior by default.

Maybe that could work nicely, but even when it's completed, i'm not yet 100% sure that if fixes everyone's issues here - to be confirmed?

Thanks for giving this some thought. This has already been provided as a solution here https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ however this doesn't provide a real solution and still hides the first real referrer.

I'd like to provide some examples to demonstrate the overreporting of direct entries in one of our clients, I can confidently say the “direct entry” attribution is far too high using all of the Matomo attribution techniques.

Here is a monthly view image

Here is a yearly view image

Here is a segmented yearly view for new visitors (I wouldn’t expect new users to be showing such high direct entries) image

Therefore I’m fairly certain something is not attributing correctly, whether it’s us the users (it seems there are many users as above having similar issues) who have not understood so something or whether Matomo tracking is not attributing as it should in some cases I’m not sure - any suggestions are welcome.

In my response above https://forum.matomo.org/t/referral-exclusions/33582/25
You can see that using the marketing attribution report we have an extremely high proportion of direct entries as a result.

@sgiehl commented on May 11th 2022 Member

Maybe there is another issue with calculating the numbers in MultiChannelConversionAttribution plugin.
For now I would like to focus on how Matomo itself should track and attribute the referrers. Once this was sorted we can revisit the multi channel topic and check that.

So the current visit referrer attribution behavior when using some sort of payment provider is this:

  • Visitor visits site from certain source X
  • Visitor is forwarded to payment provider Y
  • Visitor comes back from payment provider Y (within the visit time frame)

Now the behavior depends on the configuration and set up

  • If the payment provider url was added as a site url
    --> The action will be attached to visit and the attributed visit referrer will remain X
  • If config create_new_visit_when_website_referrer_changes = 0 (which is default)
    --> The action will be attached to the visit. the attributed referrer will only be updated to Y, if it was a direct entry before)
  • If config create_new_visit_when_website_referrer_changes = 1 is set
    --> A new visit will be started having Y as attributed visit referrer

Note: If the visitor comes back from the payment provider after the visit time frame, a new visit will always be created. If the payment provider url was added as site url, that new visit will be attributed as direct entry. Otherwise this visit will have the payment provider as referrer.

So far the referrer attribution of visits. Conversion attribution has a special handling, that additionally depends on the javascript tracking:

When a visitor comes to the site from a certain source X (other than direct) and cookies are enabled, Matomo will create a attribution cookie that contains the referrer details. This cookie is sent with the tracking requests and will be used to attribute the conversions. If the cookie isn't present Matomo will automatically use the current visit referrer for attributing the conversion.

Note: If the url stored in the cookie can be resolved as a site url while tracking, it will be ignored and the visit referrer is used as well

If a visitor opens the site again, coming from any source other than the urls configured for the javascript tracker (e.g. with setDomains), the referrer cookie will be updated to this new referrer. And following conversions will be attributed to it.

This behavior can be changed by using the javascript tracker method setConversionAttributionFirstReferrer. In that case the cookie won't be overwritten and the first referrer will be used for attributing the conversions as long as the cookie exists.
By default this cookie has a lifetime of 6 months, which can be adjusted using setReferralCookieTimeout


So thinking about the currently suggested workaround to add the payment provider to the site urls, this might be only the half way, as the conversion attribution cookie would still be updated (and later ignored) when coming back from payment provider. So ongoing visits will no longer be attributed to the first visits referrer. To solve that adding the payment providers url to setDomains might be required.

The tracking code we are suggesting to add, by default doesn't contain a call to setDomains, unless the option In the "Outlinks" report, hide clicks to known alias URLs of matomo.org is checked. But if a site for example is running and tracked on more than one domain, the referrer cookie might also get updated each time when switching between the urls, which might be unexpected behavior as well.

So overall if the payment provider is added as a site url and also added with setDomain in the javascript tracker, I actually think the attribution behavior should be correct. Or am I seeing something wrong?

@adsham commented on May 11th 2022

If the payment provider url was added as a site url
--> The action will be attached to visit and the attributed visit referrer will remain X

This behavior is what I think most of us expect and require - currently though, this will change X to Z (direct entry) and we lose the first referrer, X.

See this document https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/

@sgiehl commented on May 11th 2022 Member

@adsham Looking at the code I would disagree. It looks like the referrer should only be updated if the original referrer of the visit was a direct entry and if the new referrer is not direct entry. So it should never be updated from a website, campaign or search engine to direct entry. See https://github.com/matomo-org/matomo/blob/0f89a53f82d1d0d50cd1e22773be9321256bbe93/plugins/Referrers/Columns/ReferrerType.php#L58-L65

That's at least how it should work for attributing the visit.
As mentioned above this is handled slightly different for the attribution of a conversion.

Are you sure that the visit's referrer is changed? If that is the case you would also see smaller number in the referrers reports for campaigns, websites and search engines. If "only" the campaign would be incorrectly attributed, that would most likely only change the ecommerce reports.

For now I will stop implementing any possible improvement like a referrer exclusion. It still seems not to be clear what the real problem is here. I will start writing some additional tests to cover as many test cases as possible around referrer attribution. Maybe that helps to discover some unexpected behavior.

@adsham commented on May 11th 2022

From the documentation (https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/)...

"Then whenever Matomo will track visits in the future, any website referrer that matches the same hostnames as the “URLs” of your website, will appear as “Direct entry” instead of “Website” referrer channel."

And here is my initial forum post of what is happening in my case https://forum.matomo.org/t/referral-exclusions/33582/25.

For one of our clients (who have since left because of this) we were reporting 44% Direct Entry vs 14% in Google Analytics. We were measuring "new" customers so we knew direct entry should not be anything like 44% - comparing with GA was showing that a lot of campaign referrers were being assigned direct entry because of the 3rd party payment referral issue referenced in the above documentation.

A bit more of an explanation of the data I am using. We use the hosted/cloud version so cannot access the database and instead download visit data from the api (could this be picking up the last referrer?) I use the endpoint Live.getLastVisitsDetails - we use this data to combine with offline data (TV, radio, direct mail etc to provide an overall attribution report of online & offline activity). This is why this data must be accurate for us.

Here are fields from the returned api data we are using.

image

Thanks

@mattab commented on May 11th 2022 Member

@adsham

  1. in your screenshot you use attribution model "Last interaction" but instead it would be more accurate/useful to use the "Last non direct" attribution model - would this maybe give better results already?
  2. also you may want to use setConversionAttributionFirstReferrer if the first referrer was more relevant to you? https://matomo.org/faq/general/faq_106/

download visit data from the api (could this be picking up the last referrer?) I use the endpoint Live.getLastVisitsDetails - we use this data to combine with offline data (TV, radio, direct mail etc to provide an overall attribution report of online & offline activity). This is why this data must be accurate for us.

Yes, the data from API will return the "Current referrer" of the visit, and not the original referrer (which is only stored in the cookie). The "original referrer" stored in the cookie will be however copied into the conversion row (not the visit).

I had a quick look and i'm not actually sure if it's possible to get the RAW data for the channel attributed to the conversions. in the "action > goal" entries eg in this request, it seems the action rows with <type>goal</type> don't have the referrer information

@sgiehl

  • Does the Live.getLastVisitsDetails Raw data API already able to return the referrer attributed to each goal conversion?
@mattab commented on May 12th 2022 Member

@sgiehl Thanks for the comment above, that's a great summary of how it works :+1:

So thinking about the currently suggested workaround to add the payment provider to the site urls, this might be only the half way, as the conversion attribution cookie would still be updated (and later ignored) when coming back from payment provider. So ongoing visits will no longer be attributed to the first visits referrer. To solve that adding the payment providers url to setDomains might be required.

Feedback:

  1. instead of asking people to use setDomains (which they could of course if required) but maybe we could try to make it work for many people by default by putting the "checkout domains" like paypal.com etc. in the JS tracker code also by default? (from say a INI config setting) so it just works by default at least for paypal (and the most popular ones).

  2. additionally, as @SimonNL suggested above, a great additional solution to catch even more cases by default would be to support this "ignore_referer" / ignore_referrer and when set, the cookie wouldn't be updated + visit referrer in Matomo wouldn't be set, it would solve the issue for even more ecommerce shops.

Proposed executive summary of the issue:

  • The first goal conversion triggered by a user purchasing on Paypal is attributed correctly in all Goal/Ecommerce reports in Matomo (this works as expected)
  • BUG Because matomo uses last click by default, then once the user comes back from Paypal the first time, then all future conversions will be attributed to Paypal (the issue we're all having here).

Expected instead:

  • Paypal does not show up in Attribution reports, Website Referrers, Multi channel attribution reports
  • Cookie does not get filled in with Paypal.
  • this should work easily for other ecommerce platforms (paypal and maybe others people use a lot?), and whenever the URL contains ignore_refer[r]er=true|1
@ts1985 commented on May 12th 2022

This is a general but and just adding some providers like PayPal to a list, will not fix the bug. It's a workaround and the bug will still exist.
There are hundred of payment processors and payment providers. Sometimes you are not forwarded to the payment provider, you are forwarded to a payment page like Adyen, Stripe and also here hundred of services.

And payment is just a topic we discuss here. There could be other cases. Maybe if you are forwarded to Google, Facebook whatever to login.

So the solution is to fix the bug and not to build workarounds. Or you have to maintain thousands of URLs and provide a easy solution for all Matomo admins to add own URLs to the list.
And on the other side, Google Analytics for example is just working. Without this bug.

And the other problem in this discussion is that here are mentioned other bugs. Can you all please create own bug tickets and don't add them here? And yes, I can also confirm, that such bugs exists like that there are much!! more direct entries in Matomo than in Google Analytics.

@adsham commented on May 12th 2022

@mattab first non-direct does not help it is not reporting nearly half of the conversions.

image

I agree with @ts1985 and I apologize if I have confused the matter - I was hoping showing the marketing attribution report would demonstrate the issue in a simple way but I suspect its muddied the water.

If we come back to the issue many of us are facing:

  1. User clicks on an email link to visit the website. (Matomo tracks utm data e.g. utm_medium=email & utm_source=abandonedCartReminder and sets referrerType=campain )
  2. User browses website and adds items to basket (at this stage referrer is “Email campaign”)
  3. User completes the order and website redirects to 3rd party payment provider (paypal/amazon etc) or as @ts1985 this could be any 3rd party service
  4. User returns to website (but now referrerType for the visit is reset from “campaign” to “website” or if you have set the referrer exclusions (https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/) it will be set to “direct entry”. What we need is for it to remain as “Email Campaign”.

It is possible that in point 4 that it is generating a new visit and splitting the 1 visit into 2, but in any case the ecommerce conversion gets credited to website or direct entry, when in the real world we need the option to have the above steps maintained as 1 visit (if inside 30 min) and also be able to report the first referrer of this visit (not first referrer of first visit or most recent referrer of this visit) was the email abandoned cart campaign. in this example

Additionally for us cloud users we need to be able to access this from the API, so @sgiehl perhaps you are correct that the code is working as expected (and is stored somewhere in the database but could you confirm how I can access this "initial referrer for each visit" from the API so that I can use this information in our reporting?

Ideally at the visit level we would have firstReferrerOfThisVisit & lastReferrerOfThisVisit data but if not possible firstReferrerOfThisVisit would be more appropriate (with a configuration option at the site level in case some users want last referrer).

Hope this clarifies things!

@sgiehl commented on May 12th 2022 Member

Thanks for all the feedback and updates. I am actually still not sure what exactly the expected behavior in each use case would be.
I'm pretty sure there might be some unexpected behaviors, that we need to fix. But to be sure we don't do some adjustments on the one end and start breaking things on another end, I have started writing tests, trying to cover as many use cases as possible. It might take till early next week until I have finished writing them.
Once finished, the test cases will kind of document the current behavior (unless I find something obvious I can directly fix). Afterwards I would kindly ask you all to look through the defined test scenarios to identify those, that would need to be changed/fixed. And maybe note possible test scenarios I might have missed so we can add them.
I will give an update here, once I'm done with the tests.

@MatomoForumNotifications commented on May 16th 2022

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/exclude-referrers-from-conversion-attribution/45949/2

@sgiehl commented on May 18th 2022 Member

After having a closer look at the code and building a test matrix with over 400 potential combinations, I have written tests to cover all cases I could think of. This includes the configuration options for creating new visits when the website or campaign changes, as well as having a referrer's url in the site urls and using attribution cookies that might be placed and updated when the referrer changes. As having a look at the tests won't make much sense for non-techies I now tried to build some flow charts to visualize how the attribution in Matomo works based on the test results.


Attribution of visits and conversions

The following chart will describe how the attribution in Matomo it self takes place:
Referrer Attribution


Handling of Attribution Cookie

In the part above there is a check if a valid attribution cookie is being sent. This is handled by the javascript tracker. The chart below describes when a cookie will be set and/or updated.
Referrer Attribution Cookie


I hope this helps everyone to understand how the attribution process works. It anything is unclear or there are any related questions, let me know.

Regarding the original issue that conversions might be attributed to the wrong channel, when returning from a payment provider. In case the payment provider is added to the site urls and also added in javascript using setDomains, the payment provider should not occur anywhere in the referrers and the visits and conversions should be attributed correctly.
Guess the setDomains part is missing in our FAQ and might need to be added.


Additionally for us cloud users we need to be able to access this from the API, so @sgiehl perhaps you are correct that the code is working as expected (and is stored somewhere in the database but could you confirm how I can access this "initial referrer for each visit" from the API so that I can use this information in our reporting?

Currently the visit details return by the API only contain the referrer attributed to the visit. The conversions might though be attributed to another referrer. I will tomorrow start setting up a PR to add the referrer to the conversion details returned in a visit. So that information will become visible. Maybe we could even show that in the visitor log (at least if the referrer differs from the visit)

@sgiehl commented on May 20th 2022 Member

I might have found a small issue, which might affect the visit attribution while writing more tests for cases where attribution cookies are used. The flow chart above also was incorrect for that case, as it doesn't took the cookie into account for visit attribution, which is actually done, but only if the cookie attribution is for a campaign. I've updated the chart. Hope it now reflects the current behaviour.

@sgiehl commented on May 24th 2022 Member

Does anyone have some feedback on the flow charts above? Those charts should document the current attribution behavior. If there is anything unexpected, let us know, so we can discuss possible changes.

@ts1985 commented on May 24th 2022

I had presented here several times the problem, as it was also named by others here and in other forums, etc..
As a result, we now have a float chat and the statement that everything works as expected.
Accordingly, it is not acknowledged that there is a bug or just a design error here. The result is that Matomo is "useless" in this sense and people then have to switch to Google Analytics or other software after all.

The float chart is very technical. That you can probably set / maintain something in various places / must or whatever, is not a solution from my point of view. Apart from the fact that there is no interface for this in the admin.
Who uses Matomo expects that it works. But it does not.

Again:

  • Visitor comes to website from page Y, Google whatever.
  • Visitor makes a purchase and thus achieves a goal
  • Now you expect to see that the purchase originally came from page Y, Google whatever
  • Instead, 90% of the time you only see that the purchase came from PayPal, credit card provider or one of thousands of other possible payment methods. This info is therefore false and useless.

If this state is still ok from your point of view, then this is either a serious design error or a bug.

The fact that you generally have a problem assigning visitors correctly can be seen in the problem mentioned here (and in other places) that an extremely high number of visitors are assigned to "direct entry". If you compare this with alternatives like Google Analytics, significantly more visitors are assigned to correct sources.

So there are several serious problems here that may be related. This should be recognized and fixed.
So I don't need a flow chart to understand that everything works. That it does not work correctly has been described several times in various places by different people.

Sorry if that may come across as a bit rude. But when you recognize a problem, then see that others report the same and as a result you are told that everything works as expected, that is more than frustrating.

@mattab commented on May 24th 2022 Member

Thanks @sgiehl for the flow chart, that's very helpful indeed.

2 notes on this:

  • Could we put it on the developer guides after this issue is closed?
  • In the first chart, on the bottom right, it says "Conversion is attributed to Referer Y" -> But I would have expected instead to read "Conversion is attributed to the current Visit Attribution" (which may be either X or Y depending on what the flow was). Can you confirm this is the case and if so, update the chart above?

Now I think the problem we're all concerned about is this flow:
new

This behavior is maybe the one causing all the trouble here.

I see maybe 2 possible solutions

Either solution 1

We remove this behavior and instead the "Visit attribution" stays untouched (either Direct or Referer X).
This is maybe what @ts1985 suggests which makes sense.

OR solution 2

it has 2 parts:

a) We implement a new behavior (in Chart 1) such as this: "Visit attribution updated to Referer Y" only if:

  • it's not a know payment provider domain (paypal, etc. etc <- we should try find a list of such URLs to include by default so we cover most cases)
  • OR the URL of the website contains &ignore_refer[r]er=true|1 url parameter (in the request where the Referer Y is suddenly set"

b) Additionally we implement new behavior in Chart 2 ("handling of attribution cookie") So that any visitor who enters from "a known payment provider domain" then doesn't trigger the whole flow "Visitor comes to the website from referer X" because the payment provider would not be counted as a "referer" and be ignored. And the same would happen if the url contains ignore_refer[r]er.
(so maybe this logic needs to be implmented in both Tracking API and also that JS code handling attribution cookie)

==

Analysis on solutions

If we go with solution 1) It will actually regress for some users. Indeed, some people want that the Referer Y which suddenly appears in a visit, should be set in the visit and overwrite the direct entry or existing website referer. (I actually don't know how big this use case is, but certainly it improves data quality for some people - Not people in this conversation though)

If we go with solution 2), in general it feels like it would fix the issue completely (maybe?!) and also not regress behavior for anyone (where ignoring a new referer might be undesired for some people). So maybe solution 2) only improves behavior & data accuracy while also fixing the issue we have here?
The only downside of solution 2) is having to maintain a list of payment provider URLs etc. But maybe someone else already created this list? And maybe we could manage it in a new open source repository to encourage contributions?

@adsham commented on May 25th 2022

Hi @sgiehl

Thanks for your thorough investigation so far, it looks like we are starting to get somewhere. It shows the complexity in the process.

I do agree with @mattab with regards to the fact there may be genuine use cases where we might want to maintain the latter referrer (even though I would say the majority would be interested in the referrer that initiated the visit). Rather than maintaining a list of exclusions which lead to issues (would the list be a global one? Would each site need an exclusion list? Would different websites have different requirements for the exclusion list?) - Could we instead have a config option that just always respects Referrer X (and overrides any update to Y)?

Also please do pardon my ignorance, I confess I do not fully appreciate the inner workings of the process despite the detailed data flow above. I have highlighted the paths that cause an update to referrer Y (from X). Could you clarify what "Prev. visit" is defined as? Is it the first step in the data flow (before being redirected back to the site, or is it a visit from the past?) Looking at it with my simple eyes and my current experiences, I am not concerned whether or not the previous step/visit was direct or campaign or website or something else (red highlighted paths). From a business point of view I want to know what got the customer to the website (was it my email or ppc campaign or was it a 3rd party referral, was it a search engine referral). If another campaign supersedes this during the lifetime of this visit - from my personal viewpoint I am not concerned, I want to maintain the initial visit referrer.

image

As a side note, I am not familiar with some of the flags such as "create_new_visit_when_website_referrer_changes" and "create_new_visit_when_campaign_changes" - Could you confirm the defaults of these for hosted Matomo users.

Many thanks

@adsham commented on May 25th 2022

I would also agree that the data flow should go into the documentation - many users may not understand the complexity behind the attribution process.

@sgiehl commented on May 25th 2022 Member

@ts1985
Simply trying to implement some new stuff that solves a specific issue won't help anyone, unless we are able to guarantee that nothing else would break with it. And that is the reason why I started writing tests and creating the flow charts. To me it was important to kind of document the current behavior. That does not mean that this behavior needs to be correct, or does not need to be adjusted or could be improved.
But before implementing new stuff we need to ensure that this behavior is either correct or we adjusted/fixed it so it does what it should be. Once that is clarified I'm happy to discuss how we can handle the issue regarding referring payment providers.
As mentioned before currently this can be achieved by adding the payment provider to the site urls as well as using setDomains in javascript. This might for sure not be the best solution, but at least a workaround until we have something more useful.


@mattab

Could we put it on the developer guides after this issue is closed?

Sure. But unless we have a common tool to create/edit such charts, no one else might be able to update the charts later. So might be good to clarify that internally first.

In the first chart, on the bottom right, it says "Conversion is attributed to Referer Y" -> But I would have expected instead to read "Conversion is attributed to the current Visit Attribution" (which may be either X or Y depending on what the flow was). Can you confirm this is the case and if so, update the chart above?

The right side actually is for visits with referrer Y, the left side is with referrer X (the lines from the step above are continued). Guess we could unify the last part to either using cookie referrer or visit referrer and don't differ between referrer X and Y from before.

Now I think the problem we're all concerned about is this flow:

I'm not sure if that part is the problematic one. It actually only says that the referrer would be updated from DIRECT to WEBSITE in that case, which seems correct.

Regarding the payment provider referrer issue I actually see this possibility:

  • Implement new "Referrers to ignore" feature in site settings as well as in javascript tracker
    • There will be a global config for referrers to ignore
    • Each site will be able to add additional referrers to ignore
    • The JavaScript tracker will have a new method setIgnoredReferrers
    • The Code generator will automatically include the new method call to ignore the configured hosts

Additionally we could also use a new url parameter that triggers the javascript tracker to ignore the referrer.
So as soon as the url parameter &ignore_refer[r]er=true|1 is provided, the javascript tracker will neither send the referrer with the tracking request, nor update the attribution cookie to the current referrer.

I'm not sure if starting to automatically hide known payment providers would be much of a help for everyone.


@adsham

Thanks for your notes. With prev. visit referrer I actually meant the referrer previously set for the current visit, so in the flow chart this would the previous step. Would it be more clear to write Referrer X instead?
The default for the configuration options are: create_new_visit_when_website_referrer_changes = 0 and create_new_visit_when_campaign_changes = 1


The part in the flow charts I personally would not have expected is, that the attribution cookie might set a visit referrer if it is a campaign. This actually also means that if I visit a page from a campaign and an attribution cookie is placed, that all direct visits that happen afterwards would be attributed to the campaign. My expectation would be direct entries where the conversions might still be attributed to the campaign. But maybe that is something that is expected from a marketing/business side of view?


Anyway, I guess it would be good to maybe gather a list of changes everyone might require and afterwards create issues for each requirement/feature request separately, so the product team will be able to prioritize them.

@justinvelluppillai @mattab could you maybe take over from here and discuss and decide what we want/need to change or implement. I'm happy to do that once that was clarified.

@ts1985 commented on May 25th 2022

A list of payment providers to ignore is not a solution. Reasons:

  • it's a bug that the referrer changes. Ignoring a list of payment providers means also that if they mention a website in their blog or link it on their website is not visible anymore. And of course I also want to track the visitors coming from a payment provider. But if someone comes to our website and uses just a payment provider I want to know where this visitor was from and not that he used the payment provider x.
  • also I'm talking here about payment provider, as written before, same issue could be for other third party services like login via Google/Facebook/whatever and maybe some other stuff.

So a list of such providers is not the solution. The problem is the bug that the referrer of visitors is changed if they leave the website to do some third party stuff and then come back when finished.

@Demichev commented on May 26th 2022

@mattab

I see maybe 2 possible solutions

Either solution 1

We remove this behavior and instead the "Visit attribution" stays untouched (either Direct or Referer X). This is maybe what @ts1985 suggests which makes sense.

OR solution 2

it has 2 parts:
a) We implement a new behavior (in Chart 1) such as this: "Visit attribution updated to Referer Y" only if:

  • it's not a know payment provider domain (paypal, etc. etc <- we should try find a list of such URLs to include by default so we cover most cases)
  • OR the URL of the website contains &ignore_refer[r]er=true|1 url parameter (in the request where the Referer Y is suddenly set"

Can we have a mix of two solutions?

  1. You don't need to fully remove this behavior, just add new option to config as @heurteph-ei said earlier. So there is no regression here.
  2. I found that some users keep opened page from our site with referrer to payment site after purchase. So it starts new visit with payment site referrer weeks and weeks later after actual purchase. Of course we can add some JS to refresh page and reset referrer but it is good to have a list with ignored sites. And I don't like any url parameters. It can not be done in some cases.
@mattab commented on May 31st 2022 Member

Maybe we could do what @Demichev you suggested:

  1. introduce new INI settng update_referrer_in_visit_when_referrer_changes set to 1 by default (that people can set to 0 to ignore all referrer updates)
  2. also introduce list of payment providers including paypal.com etc to always ignore (new INI setting + setting in JS traker) so they are always ignored in the cookie and in the visit's referrer
  3. also introduce similar ignore logic when the &ignore_refer[r]er url parameter is found

@sgiehl @justinvelluppillai could we do these 3 maybe?

@sgiehl commented on June 1st 2022 Member
  1. introduce new INI setting update_referrer_in_visit_when_referrer_changes set to 1 by default (that people can set to 0 to ignore all referrer updates)

This one could actually make problems and I'm not sure which problems it should solve. The referrer is currently only updated when it was previously set to DIRECT. The reason for that is, that we assume it might not have been set correctly with the first hit. In some cases it might maybe happen that other tracking request(s) might be sent or processed before the initial page view is processed. If the first request does not contain the referrer, it wouldn't be updated with the pageview afterwards, which then might cause even more unexpected DIRECT hits.

From my technical point of view I would actually expect to have create_new_visit_when_website_referrer_changes = 1 by default. Having it disabled, which we currently have, actually hides all website referrers that might come in between. I don't know how other analytics solutions are handling that, but I would actually expect that every time the referrer changes (to something other than DIRECT), a new visit should be started. Technically someone needs to leave the page in order to come back with a new referrer, so why don't we count that as a new visit?
Maybe we should think about cleaning that up with Matomo 5 and only have one config option like create_new_visit_when_referrer_changes = 1

  1. also introduce list of payment providers including paypal.com etc to always ignore (new INI setting + setting in JS traker) so they are always ignored in the cookie and in the visit's referrer

already started working on that here: https://github.com/matomo-org/matomo/tree/referrerexclusion
can continue to work on that.

  1. also introduce similar ignore logic when the &ignore_refer[r]er url parameter is found

So the parameter will trigger the javascript not to store the referrer in the cookie and maybe even prevent sending the referrer with the tracking request. On PHP side we can also discard the referrer in that case and drop the parameter from the tracked url. But should that also affect tracking campaigns? What should happen if the tracked url contains some campaign parameters but also ignore_refer[r]er? Should we ignore the campaign in that case as well?

@justinvelluppillai once clarified: shall we create new issues for each of the points, so we can plan them into the next milestones separately?

@justinvelluppillai commented on June 1st 2022 Contributor

@sgiehl that sounds good to create separate issues. Perhaps 2. can make it into this milestone, do you think?

@sgiehl commented on June 1st 2022 Member

@justinvelluppillai I will go on working on it later, but not sure if it will pass the review process till the RC release.

@mattab commented on June 8th 2022 Member

I would actually expect that every time the referrer changes (to something other than DIRECT), a new visit should be started. Technically someone needs to leave the page in order to come back with a new referrer, so why don't we count that as a new visit?

The problem with creating new visits is that if in case of an edge case situation, then you end up creating many new visits, and that has a lot of impact on data accuracy and trust. Here is an example such edge case: if someone has tabs opened, and one of the tabs has the Old referrer, then whenever their browser reloads and tabs reload, it would create possibly 2 visits each time. It quickly becomes a bug that many people can spot and I think there are more such edge case(s).

should that also affect tracking campaigns? What should happen if the tracked url contains some campaign parameters but also ignore_refer[r]er? Should we ignore the campaign in that case as well?

I'd say either is fine. I don't have strong preference. (But if i had to choose, then maybe it's simpler to also ignore tracking campaign parameters when ignore_referer is set, so as to make sure it won't create a new visit. )

@sgiehl commented on June 28th 2022 Member

@mattab I have now created pull requests to cover some parts of your suggestions.

https://github.com/matomo-org/matomo/pull/19302 will introduce the possibility to set a list of referrers that should be ignored. This needs be done in the site settings (and/or globally), which causes the PHP tracker to ignore such referrers. In addition there will be a new method for the javascript tracker to ignore certain referrers. This is needed to prevent the attribution cookie to be updated, which might otherwise cause a change in conversion attribution.

https://github.com/matomo-org/matomo/pull/19420 will introduce the url parameters ignore_refer[r]er, which lets the javascript and php tracker ignore the current referrer. The url parameter is automatically removed. As discussed, this also includes campaign detection.

Imho those new features should give everyone the possibility to ignore their payment provider (or other services) as referrer. I'm actually not sure if it's worth to implement a static list of known providers. Maybe adding/updating some FAQs might be good enough.

@ts1985 commented on June 28th 2022

A list of payment providers to ignore is not a solution. Reasons:

  • it's a bug that the referrer changes. Ignoring a list of payment providers means also that if they mention a website in their blog or link it on their website is not visible anymore. And of course I also want to track the visitors coming from a payment provider. But if someone comes to our website and uses just a payment provider I want to know where this visitor was from and not that he used the payment provider x.
  • also I'm talking here about payment provider, as written before, same issue could be for other third party services like login via Google/Facebook/whatever and maybe some other stuff.

So a list of such providers is not the solution. The problem is the bug that the referrer of visitors is changed if they leave the website to do some third party stuff and then come back when finished.

Please read this again and understand, that your solution is not a solution. This will not fix the bug. It's just a workaround which will result in more issues without fixing the main issue.

@sgiehl commented on June 29th 2022 Member

@ts1985 That topic is not as easy as you might think. Updating the referrer is only done in some specific cases and only if the previously stored referrer was direct. So that can't overwrite any other referrers. But that actually isn't the problem the issue topic was about. The conversion attribution is not only defined by the visit referrer. If there is an attribution cookie, that one will overwrite the conversion attribution. As this one is handled in javascript, we cant really do anything on the server side.

If you don't want a conversion being attributed to a certain service provider, you would need to unset the referrer for the javascript tracking or use setConversionAttributionFirstReferrer. Or with the new feature you can define domains/hosts that should be ignored for that. I'm aware that there might be cases where someone still wants to track the referrer when the user is initially coming from that service provider. But at least on javascript side we can't know that, as we don't have any details on the current visit there. As the new feature allows to define only subdomains or domains including a path, that should be enough to only exclude certain urls of the service provider, while still tracking others.

@ts1985 commented on June 29th 2022

I think it is that easy. Just don't update the referrer. It's a bug that the referrer is updated for the same visit.

And another big problem, also described here by different people, is, that there are so many "direct" visits. Some compared to Google Analytics for example and that there are much less "direct" visits.
So it seems that Matomo often doesn't recognize the Referrer of a visit.

@sgiehl commented on June 29th 2022 Member

Well, not updating the referrer anymore will bring you even more direct visits 🤷
Anyway, I can't compare where GA is getting it's referrer data from. We are tracking a referrer as soon as one is provided by the browser. This might e.g. not be the case if the referring website restricts the referrer from being sent.

@ts1985 commented on June 29th 2022

There is a complete thread about this "too many direct entries" topic: https://forum.matomo.org/t/what-can-explain-the-mysterious-too-many-direct-entries-phenomenon/31721

It's a real problem. I know it's another problem but it seems the Referrer topic in general is problematic for Matomo.

Powered by GitHub Issue Mirror