@mattab Any idea what this feature could be named and how we would configure this? Not sure if we want to have this per site and globally or whether it be enough only one way etc. It could get confusing maybe for people that don't need this feature. Maybe it could be even a separate plugin?

In my point of view, if a user :

1. Comes from one referrer,
2. Then browse the measured site (tracked by Matomo),
3. Then browse out the measured site (eg. payment gateway or really leave the measured site)
4. Then come back to the measured site during the 30 minutes delay (for any reason: payment or anything else)

Then the referrer should at least be the 1st one! (would it be possible to accumulate referrers?) And if the other referrer is considered as "direct entry” (what is done in case of excluded referrer) it means the only real referrer is still the 1st one...

Note also that in case of payment gateway, the returning back delay should not exceed short delay as in other ways, the payment token would probably expire... Then the scenario where the 1st referrer is a payment gateway should be very rare case...

Thanks @tsteur for creating the issue here on my behalf.

I largely agree with @heurteph-ei - the main reason for referral data is to monitor how traffic is being driven to the website, especially if we attribute value from sales to a referrer. I can't really see any benefit of having "direct entry" as the referrer as it is misleading. However I do see a legitmate scenario that could be obscured by always using the first referrer.

Here are 2 examples

• Comes from Email referrer,
  

• Then browse the measured site (tracked by Matomo),
  

• Then browse out the measured site (eg. payment gateway or really leave the measured site)
  

• Then come back to the measured site during the lifetime of a visit (e.g. 30mins)
  

In the above scenario - Paypal (for example) should be set up as a referral exclusion and therefore the 1st referrer should be used (Email).

• Comes from Email referrer,
  

• Then browse the measured site (tracked by Matomo),
  

• Then browse out the measured site (to search for a coupon code or referral link)
  

• Then come back to the measured site during the lifetime of a visit (e.g. 30mins)
  

In the above scenario we may want to know both the former or the latter referrer.

With this in mind I would suggest either...

• Making it a configuration option ("use first referrer" or "use latest referrer") at the measurable level ( in combination with the referral exclusions).

or

• Adding a secondary field(s) to the underlying data so the user can choose to report on either first or last referrer - (and also make this new data available via the api)

What i don't understand yet, is that the scenarios above should already work meaning:
as long as the subsequent page views (after purchase) occur within 30min of the last pageview on the website, then the pageviews should be stored in the existing visit (with the correct original campaign). The fact that it doesn't work for some people is not clear.

@adsham @heurteph-ei Could you maybe confirm if the subsequent visits (the one with referrer "Paypal" or the direct entries) ones occured within 30 min of their last pageview?

(the only other reason i can think of is that people use a different browser to pay and that Matomo cannot recognise them as the same user, but that should be very edge case)

Hi @mattab

Thanks for picking this up. You will see that several users have picked up on this issue here https://forum.matomo.org/t/referral-exclusions/33582/22 and possibly here too https://forum.matomo.org/t/what-can-explain-the-mysterious-too-many-direct-entries-phenomenon/31721/6

I have also just looked through our data and can confirm that there are no "delays" as you are suggesting. However I'm not really in a position to fully test this out at the moment (we do not have a test payment provider currently configured) perhaps @heurteph-ei or someone from the forum could do this? I will ask there too.

Could the resolution in the article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ be causing the original referrer to be overwritten with "direct entry"?

Many thanks once again, hopefully we can find a resolution to this soon!

Hello,

for me its very simple and there are different ways, how you can create the "correct" goal*1.

*1Goal: as already written. We have to know, from where the customer is coming from.
If Matomo delete now all referers (change to direct entry, using the payment gateway as referrer) thats just washing statistic with 0 sense. Nobody can track now, from where the customer really came then - ofc - he didnt came from PP, Klarna, CC etc.

I am not a technician, so I have no idea, why some 30 mins. timeframe is here on the table.
Sometimes its happend, that someone needs - because of distraction - longer with PP then 30 mins (PP=PayPal). The usual session for PP is running much longer also "all" shop sessions.
To me - 30 mins - are fine. Ofc some would be maybe out of this window, but thats 0.x% and ok if we dont have them tracked.

To make it lesser complicated:
Why not just making a "just track the first referrer" option in a XX min. timeframe?
or
Just track (and show!) all referrers.*2

*2 this would have the advantage, that you can directly see a fast overview of the daily payment gatesways too, without needing your ERP.

Exlucde paymentgateways and putting all referrers as direct entry or use the gateway as referrer is for nobody really helpful.

Hey there,

it is great that this issue is picket up now. I have been experiencing this problem for a long time now.

Here is our setup:

• Shopware 5.6 Shopsystem with paypal as payment provider
• When the user goes to paypal for payment and returns to our Shopware shop, a new session is started with referrer paypal.

This completely messes up all conversion attribution, as most customers use paypal.

All customers return from paypal within the 30 minute timespan.

I would opt for an "attribute to first/last referrer" configuration in matomo.

Thanks for taking care of this and let me know if I can be of further help.

Cheers

Jens

P.S.: If I would use the ignore referrer option, the new session would have a direct access referrer which even more messes up attribution, since I cannot identify anymore that paypal messed up the referrer.

Coming from GA and having struggled with the same issue there years ago, I quickly ran into this again with Matomo. Specifically the 'payment service provider' scenario. With GA we started with exclusion list but could finally solve this for good by adding the 'ignore_referrer' parameter to the PSP return/error/pending page (e.g. /thank-you?ignore_referrer=true) so regardless of the payment method, the site in between is always ignored. This works better than referral exclusion lists because the payment urls tend to change over time and new ones are easily overlooked. A lot of e-commerce sites will already be using this parameter so would supporting this not solve the issue for many cases?

I'd like to provide some examples to demonstrate the overreporting of direct entries in one of our clients, I can confidently say the “direct entry” attribution is far too high using all of the Matomo attribution techniques.

Here is a monthly view

Here is a yearly view

Here is a segmented yearly view for new visitors (I wouldn’t expect new users to be showing such high direct entries)

Therefore I’m fairly certain something is not attributing correctly, whether it’s us the users (it seems there are many users as above having similar issues) who have not understood so something or whether Matomo tracking is not attributing as it should in some cases I’m not sure - any suggestions are welcome.

We have the same problem. I think most have the same problem and it's important to find a solution for this. The conversions are thus virtually useless, since it distorts everything.
Without configuration you have conversions to the payment providers, but you want to know where the visitor originally came from and not which payment provider he used.
If you put the payment providers on the referral exclusion list, then you have the conversions on "Direct Entry" which doesn't make sense because the customer might have come from search engines, websites etc..

Hope somebody will find a solution soon.

mattab , tsteur

What i don't understand yet, is that the scenarios above should already work meaning: as long as the subsequent page views (after purchase) occur within 30min of the last pageview on the website, then the pageviews should be stored in the existing visit (with the correct original campaign). The fact that it doesn't work for some people is not clear.

@adsham @heurteph-ei Could you maybe confirm if the subsequent visits (the one with referrer "Paypal" or the direct entries) ones occured within 30 min of their last pageview?

(the only other reason i can think of is that people use a different browser to pay and that Matomo cannot recognise them as the same user, but that should be very edge case)

Perhaps then this is a bug rather than an enhancement? - it seems that many of us are experiencing this issue.

I can give you these details:

• No they don't use another browser to pay
• It's not just PayPal. We have several Payment Providers, all with the same problem.

Any update on this? It seems to be pushed back again into a later milestone.

The referral data is an important part of our service and we have recently seen a client leave to a competitor because of the inaccurate data in our reports (from Matomo).

The initial issue was raised almost 3 years ago in the forum https://forum.matomo.org/t/referral-exclusions/33582/25 and this Github issue was raised 2 months ago and pushed back into milstone 4.10 (4.8 released today).

Please could we have some committment as to when this will be addressed as it's getting harder to explain to clients that the issue is being looked at but cannot tell them when it will be looked into/fixed.

Many thanks in advance!
Adham

Hello.
We have the same issue. I temporarily modified Matomo code (return false immediately in onExistingVisit() for Referrers Columns classes). We are testing it now. I think it will solve the problem for us but it is good to have an official option.
What do you think?

A list of payment providers to ignore is not a solution. Reasons:

• it's a bug that the referrer changes. Ignoring a list of payment providers means also that if they mention a website in their blog or link it on their website is not visible anymore. And of course I also want to track the visitors coming from a payment provider. But if someone comes to our website and uses just a payment provider I want to know where this visitor was from and not that he used the payment provider x.
• also I'm talking here about payment provider, as written before, same issue could be for other third party services like login via Google/Facebook/whatever and maybe some other stuff.

So a list of such providers is not the solution. The problem is the bug that the referrer of visitors is changed if they leave the website to do some third party stuff and then come back when finished.

Please read this again and understand, that your solution is not a solution. This will not fix the bug. It's just a workaround which will result in more issues without fixing the main issue.

@ts1985 That topic is not as easy as you might think. Updating the referrer is only done in some specific cases and only if the previously stored referrer was direct. So that can't overwrite any other referrers. But that actually isn't the problem the issue topic was about. The conversion attribution is not only defined by the visit referrer. If there is an attribution cookie, that one will overwrite the conversion attribution. As this one is handled in javascript, we cant really do anything on the server side.

If you don't want a conversion being attributed to a certain service provider, you would need to unset the referrer for the javascript tracking or use setConversionAttributionFirstReferrer. Or with the new feature you can define domains/hosts that should be ignored for that. I'm aware that there might be cases where someone still wants to track the referrer when the user is initially coming from that service provider. But at least on javascript side we can't know that, as we don't have any details on the current visit there. As the new feature allows to define only subdomains or domains including a path, that should be enough to only exclude certain urls of the service provider, while still tracking others.

I think it is that easy. Just don't update the referrer. It's a bug that the referrer is updated for the same visit.

And another big problem, also described here by different people, is, that there are so many "direct" visits. Some compared to Google Analytics for example and that there are much less "direct" visits.
So it seems that Matomo often doesn't recognize the Referrer of a visit.

Well, not updating the referrer anymore will bring you even more direct visits 🤷
Anyway, I can't compare where GA is getting it's referrer data from. We are tracking a referrer as soon as one is provided by the browser. This might e.g. not be the case if the referring website restricts the referrer from being sent.

There is a complete thread about this "too many direct entries" topic: https://forum.matomo.org/t/what-can-explain-the-mysterious-too-many-direct-entries-phenomenon/31721

It's a real problem. I know it's another problem but it seems the Referrer topic in general is problematic for Matomo.

We're experiencing the same problems that conversions aren't attributed correctly to the campaigns, but to visits from Paypal, or to "deref-gmx.net". The latter is an example that hasn't been mentioned yet; it's from an email campaign that includes campaign parameters in the email, but the visit (and subsequent ecommerce purchase) is still attributed to a website, with no trace of the campaign to be found in the visitor's profile. (I've disable cookies, don't know if that has anything to do with it.) What we're seeing is that according to Matomo's statistics, none of our campaigns in the recent months have led to any conversions, even though this is very unlikely in general, we see a spike in purchases after campaigns, and at least for some conversions we have evidence that they definitely came from a campaign.

@sgiehl From what you described, you intend to add a way to ignore certain referers, but that requires everyone to find out which URLs they should add, and also will exclude some legitimate referers. The example with deref-gmx.net shows how fragile and open ended that is, as you probably wouldn't have thought of this domain. (The other idea about the URL parameter wouldn't work at all, as we cannot influence it, obviously.) Could you describe how this would get us correct attributions both in the absence of a full knowledge of the referrers we have to exclude, and without requiring every single user to adjust their configuration?

From what I understand, the underlying problem here really is that the source of a visit is ever changed after the fact at all. This seems fundamentally wrong: The source of a visit is fully determined by the referrer (or parameter) of the first visited page. If, as you described above, the events are sometimes not submitted in the right order, then that can be worked around e.g. by only assigning a source after a few seconds. But importantly, once a source has been decided on, it should never be changed anymore, because it is by then a historic fact that shouldn't be affected by any later events.

It also seems wrong that direct visits are treated differently. A direct (= unknown) visit is a source like any other and shouldn't be changed after the fact either. In particular, if we don't know where a user came from at first, the fact that at a later time they came from an identifiable source still doesn't give us any knowledge about their first visit. Instead, there probably needs to be a distinct state for the source, namely "uninitialized", which would be different from a "direct visit"; this state would only exist as long as we don't have enough information to make a decision yet.

Now, this doesn't take the more complicated cases into consideration, e.g. the campaign id changing in the middle of a visit. IMO these should probably be counted as a new visit, as such a change is evidence that the user left and then returned through a different channel. But whatever is decided here, the complicated cases should not affect the bevaviour of the simple cases.

tl;dr: May I suggest to implement a mode where it only determines the source from the first page visit (or the first few visits in the first few seconds, if technically necessary), never changes it afterwards, and treats direct visits like any other? Then we can experiment with that, and if it works out, make it the default mode, leaving the previous mode for backwards compatibility or even removing it. I'm pretty sure this is what many of the participants (and lurkers) in this discussion want, even if it later might have to be adjusted to handle some edge cases.

Not updating the referrer at all can be easily achieved by removing a couple of methods:

Removing those would fully stop updating any referrer information after the first tracking request of a visit.

Only setting/updating it for the first page view or within a certain time frame sound fine for me, but that's not a decision I can make. ping @mattab

But: Conversions might still be attributed to another referrer. Those attributions are handled using an attribution cookie, which might be updated when returning from any external service (as by default it uses the last referrer).

Not updating the referrer at all can be easily achieved by removing a couple of methods:

matomo/plugins/Referrers/Columns/ReferrerType.php

Lines 58 to 68 in 1155273

	public function onExistingVisit(Request $request, Visitor $visitor, $action)
	{
	$information = $this->getReferrerInformationFromRequest($request, $visitor);
	if ($this->isCurrentReferrerDirectEntry($visitor)
	&& $information['referer_type'] != Common::REFERRER_TYPE_DIRECT_ENTRY
	) {
	return $information['referer_type'];
	}
	return false;
	}

matomo/plugins/Referrers/Columns/ReferrerName.php

Lines 40 to 50 in 1155273

	public function onExistingVisit(Request $request, Visitor $visitor, $action)
	{
	$information = $this->getReferrerInformationFromRequest($request, $visitor);
	if ($this->isCurrentReferrerDirectEntry($visitor)
	&& $information['referer_type'] != Common::REFERRER_TYPE_DIRECT_ENTRY
	) {
	return $information['referer_name'];
	}
	return false;
	}

matomo/plugins/Referrers/Columns/ReferrerUrl.php

Lines 53 to 63 in c973567

	public function onExistingVisit(Request $request, Visitor $visitor, $action)
	{
	$information = $this->getReferrerInformationFromRequest($request, $visitor);
	if ($this->isCurrentReferrerDirectEntry($visitor)
	&& $information['referer_type'] != Common::REFERRER_TYPE_DIRECT_ENTRY
	) {
	return $this->trimUrl($information['referer_url']);
	}
	return false;
	}

Removing those would fully stop updating any referrer information after the first tracking request of a visit.

Only setting/updating it for the first page view or within a certain time frame sound fine for me, but that's not a decision I can make. ping @mattab

But: Conversions might still be attributed to another referrer. Those attributions are handled using an attribution cookie, which might be updated when returning from any external service (as by default it uses the last referrer).

Can these be removed in hosted instances of Matomo?

@adsham this can be only removed if you have access to the source code. For Matomo Cloud this can't be changed.

it's from an email campaign that includes campaign parameters in the email, but the visit (and subsequent ecommerce purchase) is still attributed to a website, with no trace of the campaign to be found in the visitor's profile. (I've disable cookies, don't know if that has anything to do with it.)

@schuetzm if you disable cookies, then the "same-visit" conversions should be still attributed, but indeed any visit from a newsletter from days ago or weeks ago will not be attributed to the newsletter. Only visits generated from the newsletter and directly converting will be attributed. (ref = https://matomo.org/faq/general/faq_156/)

Could you describe how this would get us correct attributions both in the absence of a full knowledge of the referrers we have to exclude, and without requiring every single user to adjust their configuration?

the knowledge of the problematic referrers is clear to people who have the problem, because most of the goal conversions / ecommerce conversions are attributed to these "problematic referrers" so they appear in many places in the reports and it looks buggy. So if people realise this is buggy, then hopefully they will find the feature (although I can see how that won't be easy for many people).

The example with deref-gmx.net shows how fragile and open ended that is, as you probably wouldn't have thought of this domain.

That's why the feature lets you enter the domain within the UI of Matomo so you can enter that domain name there and it will apply to all websites automatically (or you can only set it for one website if you want)

If we get complaints from a few people like we did for paypal.com then we can also add it to the Matomo list of excluded referrers so it will be applied to all Matomo users. But so far it was only very obvious for paypal.com

e.g. the campaign id changing in the middle of a visit. IMO these should probably be counted as a new visit,

Fyi that's already the case, see: https://matomo.org/faq/how-to/faq_19616/

@adsham I believe your attribution issues will be fixed on Matomo Cloud once the PRs are merged and released and deployed on the Cloud. Not sure yet when this will be all done but should be by end of July.

@adsham I believe your attribution issues will be fixed on Matomo Cloud once the PRs are merged and released and deployed on the Cloud. Not sure yet when this will be all done but should be by end of July.

Thankyou @mattab for the update and looking forward to having this resolved finally.

Not updating the referrer at all can be easily achieved by removing a couple of methods:

--- snip ---

Removing those would fully stop updating any referrer information after the first tracking request of a visit.

Thanks! I commented them out and invalidated the historical data, but there are still many conversions that are attributed to Paypal. Would this change only apply to future visits?

@schuetzm Yes. That only applies to future visits.

I disabled referrer overwrites in May in our instance of matomo. And I have analyzed 4 last months in our matomo db (direct sql queries). Referrer overwrites gave a large wrong statistics. 12 conversions in April for one referrer site but only 3 of them are correct. A common picture is that user bought a product (with direct entry) and then wanders back and forth to the blog or forum and then returns to the main site. I am not going to argue with matomo team anymore, I am used to make a series of changes in the matomo code after each applied update.
But I found one strange thing: If visit starts from the page of our own site (old opened tab or something) then matomo_log_visit.referer_url field is filled with this referrer but matomo_log_visit.referer_name = NULL. Then user goes to facebook and then returns to our site and referer_name becomes facebook (when referrer overwrites are not disabled) but referer_url = <our_site>. Do you think this is correct behaviour, @sgiehl ?

@Demichev Do you know what the referrer type was set to? Was the initial value 1 and then updated to 7? The referrer overwrites are only looking at the type afaik.

I can also confirm that with the above mentioned functions disabled we're now seeing more plausible statistics: many direct visits, many from particular search engines, and a few from campaigns. (Although there are still a few from Paypal, where it probably lost the connection because we're not using cookies.)

So, can we please have a switch for this behaviour in the configuration? Do you want me to open a separate issue for this feature request?

@Demichev Do you know what the referrer type was set to? Was the initial value 1 and then updated to 7? The referrer overwrites are only looking at the type afaik.

As I can see all refs to our site with null name have type = 1 and with filled name have types 2 (search engines I guess), 3 (other sites), 7 (YouTube). I don't know what it means.

@Demichev See

Not sure if storing the referrer url, if it is flagged as a direct entry, is correct. An url from the same site imho should never be stored as referrer url or at least I can't see any good reason for that.

A common picture is that user bought a product (with direct entry) and then wanders back and forth to the blog or forum and then returns to the main site.

when this is happening and causing data issues, then it may be worth also tracking all sub-domains/sites in one new Matomo website using cross domain tracking https://matomo.org/faq/how-to/faq_23654/

(Alternatively another workaround (with downside) is you can also define your blog and forum as "alias URLs" in the main site in Matomo: "Alias URLs: domains and subdomains that are tracked in this website. This will ensure that tracked domains don’t appear in the Referrer report." see https://matomo.org/faq/how-to/create-and-manage-websites/ -- But downside is that then the blog and forum won't be attributed as a source directly anymore... (unless you cross link with the URLs campaign parameters)

when this is happening and causing data issues, then it may be worth also tracking all sub-domains/sites in one new Matomo website using cross domain tracking https://matomo.org/faq/how-to/faq_23654/

I know about multi domain visitors, we plan to test it in the future.

Not sure if storing the referrer url, if it is flagged as a direct entry, is correct. An url from the same site imho should never be stored as referrer url or at least I can't see any good reason for that.

Don't know a good solution but current inconsistency when different refs are stored for url and name looks wrong for me.

Thanks again for all the valuable inputs.

I will close this issue here now. Based on the discussion in here, we have implemented some new features that allow excluding referrers:

• Provide possibility to configure referrer exclusion list #19302 introduced the possibility to set a list of referrers that should be ignored. This needs be done in the site settings (and/or globally), which causes the PHP tracker to ignore such referrers. In addition there is a new method for the javascript tracker to ignore certain referrers.

• Implement url parameter to ignore referrer #19420 introduced the url parameters ignore_refer[r]er, which lets the javascript and php tracker ignore the current referrer. The url parameter is automatically removed.

I'm aware that besides those features there is still a discussion about updating referrer attributions in general.
As this issue meanwhile contains too many different topics it's hard to see which problems were actually solved and which still remain.
I have therefor created #19657, which is meant to only handle the topic "Updating referrer attribution".
If anyone has some further input on that specific topic, feel free to comment there.
If there are any other issue regarding referrer attribution, please consider creating new issues, with specific topics, so we are able to handle them easier.

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/multi-channel-conversion-attribution-models-comparison/45580/12

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/does-matomo-attribute-more-traffic-to-the-direct-channel/47334/2

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/referral-exclusions/33582/34