Campaigns referers are not attributed to Goal Conversions #2168

Since 1.2 the Goalmanager uses the refering URL and the current URL, not the entry URL to determine the referer. But only the entry URL has campaign infos, so this info is lost.

The Goalmanager should use the existing data from the current Visitor and not analyze the referer a second time.

Attachment:
patch.diff

Attachment:
fix-referrer-attribution.patch

From piwik.org/docs/javascript-tracking/
setConversionAttributionFirstReferrer( bool ) Set to true to attribute a conversion to the first referrer. By default, conversion is attributed to the most recent referrer.

If that doesn't give you the desired effect, please post samples showing observed and expected results.

I have tested setConversionAttributionFirstReferrer( true ), but this changes nothing.

I made some tests:
On www.example.org i have a link to www.mysite.com/site1.html?utm_campaign=testcampaign&utm_source=testsource, piwik is configured to use utm_campaign and utm_source as keyword and referer.

If i hit this link a new Visitor is tracked as an vistor from testsource with keyword testcampaign. Everything is ok. In the database the log_visit table shows me:
referer_type = 6
referer_name = testsource
referer_keyword = testcampaign

If i now go to a second site where i track manualy a goal, a conversion is tracked, but in the log_conversion table it shows me:
referer_type = 3
referer_name = example.org
referer_keyword = NULL

I take a look on the core/Tracker/GoalManager.php an this happens becaus the GoalManager analyses the referer again.

$referrer = $referrer->getRefererInformation($refererUrl, $currentUrl = '', $idSite);

Because the currentUrl is empty, the referer Class cannot detect the campaign.

I have made a patch that fixes this for me.

Thanks for the patch lcb! I will commit soon

Actually, the patch is not enough. Technically, we should detect the campaign parameters in the piwik.js and store the "campaign name & keyword" in the cookie.

Imagine the use case

• visitor comes from campaign name=Email keyword=June2011
• visitor comes back 2 days later, with direct entry, and buys on the site
  To attribute the purchase to the original campaign, we must have the _urlref cookie store the landing URL containing the piwik campaign parameters.

We could maybe just store the actual parameters, pre-parsed, in the first party cookie.

Currently these parameters are piwik_campaign and piwik_kwd

Maybe the best solution would be to move the Campaign detection in the piwik.js all together, similarly to what GA and others are doing?

Let's do the server-side patch first.

Then create a new ticket to move the logic/storage into piwik.js since this impacts a number of tickets (#79, #517, #604, #855, #1042).

(In [4229]) Refs #2168: thanks lcb for report & patch

This will attribute the campaign correctly, if the Goal happens on the first Visit.

The campaign will not be attributed when Goals are triggered on subsequent visits and when a Campaign was the Referer used to reach the website, in a previous visit.

Proposal for fix: piwik.js will detect the campaign name & keyword parameters in the landing URL, when the visit is new. It will then store in a separate new cookie (same lifetime as referer cookie).

The values to store are the name and keyword. When a campaign is stored in the cookie, the _ref cookie is NOT deleted.

Later, we might decide to align with GA campaign tracking model ( 4 dimensions).
Also we might decide to attribute conversions differently (both to websites & campaigns) that is why we must store the URL ref and the campaign name/keywords at the same time.

The piwik.php tracker will then read these values and pick which one is preferred (the campaign is by default, higher priority than another URL referer)

thoughts?

Sounds good. (Other than the first party cookie overhead on each request and GET request length limits...)

As an update to my previous implementation suggestion, I propose that campaign name/keyword are stored in the _ref cookie along with ref url & ref timestamp, as to simplify cookie space (if there is no campaign, or no referer, the first (or last) fields are empty as they are fixed size)

I have to stop for the day, here is patch of current status

What is missing:

• Unit test the JS?
• Modify PiwikTracker to allow setting campaign name/URL / referer URL (refs #2222)
• Notify PiwikTracker.java for new specification and few new functions
• I tested the JS part I think works fine, but didn't yet test the php bit
• because utm_campaign and utm_term are now in the JS we might as well do #855 at same time

I noticed the JS fails JSLint, Anthon do you run the piwik.js via the online test tool when the test fails? (just checking)

feedback welcome... i'll finish tomorrow

I haven't looked at the patch, but yes, ci has a job to test against the latest online jslint.

re: piwik.js:

• the limit parameter in string.split(delimeter, limit) has inconsistent behaviour between browsers/versions; this came up before and is why there's a series of slice() calls when parsing the referral cookie value
• string.substring() isn't standard (and thus unsupported in some browsers/versions); use slice() instead
• API: I was planning to deprecate getVisitorId() since adding getVisitorInfo(), so a single getter for the attribution/referral array should be sufficient. Also, the campaign parameter names should probably be configureable
• unit tests: as mentioned in email, this is lacking and requires some refactoring.

(In [4370]) refs #2168 - fix bug per matt's email

OK thanks for the JS review I'll change it

• getVisitorId() is important to have, I think better have multiple getters to clarify the feature set and also, it will be easier to later add other campaign dimensions (when we add the 2 remaining to match other analytics packages)
• lack of JS tests: yes it's a bit frightening especially if we have to test the code in IE and opera... in FF and Chrome I use the dev console which is OK, only time consuming, I'll try and do it with IE as well

I'm adding the JS tests, this stuff is cool man!!

The test framework can use mock functions (replacing setCookie/getCookie) for testing. This will make it easier to test the code that loads the visitor and referrer cookies, and that handles a new visit.

(In [4378]) Fixes #2168

• Now crediting the right referrer for any goal conversion
• Referrer URLs, timestamp, and Campaign name & keyword (parsed from the landing page URL) are now stored in a first party cookie (JSON encoded)
• Added getters to the piwik.js to allow passing the values from the client, to store for later use in the PHP/Java Tracking API (paypal use case)
• Added integration tests and basic JS tests

Refs #2172

• Added a new API function: setAttributionInfo( $jsonEncoded ) which accepts JSON encoded array of 4 values (see implementation for details what to do with these values)
• 2 new parameters are: _rcn and _rck to pass to piwik the campaign name and campaign keyword that will be creditted for the Goal conversion (if a goal is converted)
• Also renamed setUrlReferer to setUrlReferrer -- Important that all public facing APIs use the proper writing

Refs #2222 Accurate Paypal tracking (or any other third party "after the fact" Goal conversion): implementation is now done, we must write some kind of guide and test

(In [4382]) fixes #2285, refs #2168