@djmetzle opened this Issue on January 10th 2023

Opening an issue on advice from the forums:
https://forum.matomo.org/t/mediaanalytics-error-message/48642/2

Expected Behavior

No errors process tracking requests with array form GET parameters.

Current Behavior

We're seeing type errors.

Possible Solution

We are currently using a patch to work around the type error from utm get param arrays:

diff --git core/UrlHelper.php core/UrlHelper.php
index 1e51828977..1faf819c63 100644
--- core/UrlHelper.php
+++ core/UrlHelper.php
@@ -279,7 +279,7 @@ class UrlHelper
     {
         $nameToValue = self::getArrayFromQueryString($urlQuery);

-        if (isset($nameToValue[$parameter])) {
+        if (isset($nameToValue[$parameter]) && is_string($nameToValue[$parameter])) {
             return $nameToValue[$parameter];
         }
         return null;

Note that this probably breaks the tests around UrlHelper::getParameterFromQueryString:
https://github.com/matomo-org/matomo/blob/8d05108aec86f5c6779b17d18c2e9a7ee7f2f827/tests/PHPUnit/Unit/UrlHelperTest.php#L117-L122

But those tests enforce contradiction with the docblock return type!
https://github.com/matomo-org/matomo/blob/8d05108aec86f5c6779b17d18c2e9a7ee7f2f827/core/UrlHelper.php#L275

This method should always return a string (not an array) in order to meet the documented interface.

Steps to Reproduce (for Bugs)

Tracking hits that include `utm` GET params in array format (which are not correct, but error out the tracker), such as ``` [2022-12-21 20:11:45] piwik.ERROR: Uncaught exception: TypeError: urldecode(): Argument #1 ($string) must be of type string, array given in /var/www/html/plugins/Referrers/Columns/Base.php:379 Stack trace: #0 /var/www/html/plugins/Referrers/Columns/Base.php(379): urldecode(Array) #1 /var/www/html/plugins/Referrers/Columns/Base.php(417): Piwik\Plugins\Referrers\Columns\Base->detectCampaignFromString('utm_campaign[]=...') #2 /var/www/html/plugins/Referrers/Columns/Base.php(560): Piwik\Plugins\Referrers\Columns\Base->detectReferrerCampaignFromLandingUrl() #3 /var/w ww/html/plugins/Referrers/Columns/Base.php(121): Piwik\Plugins\Referrers\Columns\Base->detectReferrerCampaign(Object(Piwik\Tracker\Request), Object(Piwik\Tracker\Visitor)) #4 /var/www/html/plugins/Referrers/Columns/Base.php(267): Piwik\Plugins\Referrers\Columns\Base->getReferrerInfor mation('', 'https://www.ifi...', 1, Object(Piwik\Tracker\Request), Object(Piwik\Tracker\Visitor)) #5 /var/www/html/plugins/Referrers/Columns/Keyword.php(35): Piwik\Plugins\Referrers\Columns\Base->getReferrerInformationFromRequest(Object(Piwik\Tracker\Request), Object(Piwik\Tracker\Vi sitor)) #6 /var/www/html/core/Tracker/Visit.php(514): Piwik\Plugins\Referrers\Columns\Keyword->onNewVisit(Object(Piwik\Tracker\Request), Object(Piwik\Tracker\Visitor), Object(Piwik\Tracker\ActionPageview)) #7 /var/www/html/core/Tracker/Visit.php(317): Piwik\Tracker\Visit->triggerHook OnDimensions(Array, 'onNewVisit') #8 /var/www/html/core/Tracker/Visit.php(210): Piwik\Tracker\Visit->handleNewVisit(NULL) #9 /var/www/html/core/Tracker.php(172): Piwik\Tracker\Visit->handle() #10 /var/www/html/plugins/QueuedTracking/Queue/Processor/Handler.php(46): Piwik\Tracker->tra ckRequest(Object(Piwik\Tracker\Request)) #11 /var/www/html/plugins/QueuedTracking/Queue/Processor.php(194): Piwik\Plugins\QueuedTracking\Queue\Processor\Handler->process(Object(Piwik\Tracker), Object(Piwik\Tracker\RequestSet)) #12 /var/www/html/plugins/QueuedTracking/Queue/Processor. php(143): Piwik\Plugins\QueuedTracking\Queue\Processor->processRequestSets(Object(Piwik\Tracker), Array) #13 /var/www/html/plugins/QueuedTracking/Commands/Process.php(88): Piwik\Plugins\QueuedTracking\Queue\Processor->process() #14 /var/www/html/vendor/symfony/console/Symfony/Compone nt/Console/Command/Command.php(257): Piwik\Plugins\QueuedTracking\Commands\Process->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #15 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(87 4): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #16 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application ->doRunCommand(Object(Piwik\Plugins\QueuedTracking\Commands\Process), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #17 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Inpu t\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #18 /var/www/html/core/Console.php(135): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #19 /var/www/html/core/Access.php(670): Piwi k\Console->Piwik\{closure}() #20 /var/www/html/core/Console.php(136): Piwik\Access::doAsSuperUser(Object(Closure)) #21 /var/www/html/core/Console.php(87): Piwik\Console->doRunImpl(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput )) #22 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #23 /var/www/html/console(32): Symfony\Component\Console\Applica tion->run() #24 {main} {"exception":"[object] (TypeError(code: 0): urldecode(): Argument #1 ($string) must be of type string, array given at /var/www/html/plugins/Referrers/Columns/Base.php:379)","ignoreInScreenWriter":true} {"class":"Piwik\\ExceptionHandler","request_id":138} Uncaught exception in /var/www/html/plugins/Referrers/Columns/Base.php line 379: urldecode(): Argument #1 ($string) must be of type string, array given #0 /var/www/html/plugins/Referrers/Columns/Base.php(379): urldecode(Array) #1 /var/www/html/plugins/Referrers/Columns/Base.php(417): Piwik\Plugins\Referrers\Columns\Base->detectCampaignFromString('utm_campaign[]=...') #2 /var/www/html/plugins/Referrers/Columns/Base.php(560): Piwik\Plugins\Referrers\Columns\Base->detectReferrerCampaignFromLandingUrl() #3 /var/www/html/plugins/Referrers/Columns/Base.php(121): Piwik\Plugins\Referrers\Columns\Base->detectReferrerCampaign(Object(Piwik\Tracker\Request), Object(Piwik\Tracker\Visitor)) #4 /var/www/html/plugins/Referrers/Columns/Base.php(267): Piwik\Plugins\Referrers\Columns\Base->getReferrerInformation('', 'https://www.ifi...', 1, Object(Piwik\Tracker\Request), Object(Piwik\Tracker\Visitor)) #5 /var/www/html/plugins/Referrers/Columns/Keyword.php(35): Piwik\Plugins\Referrers\Columns\Base->getReferrerInformationFromRequest(Object(Piwik\Tracker\Request), Object(Piwik\Tracker\Visitor)) #6 /var/www/html/core/Tracker/Visit.php(514): Piwik\Plugins\Referrers\Columns\Keyword->onNewVisit(Object(Piwik\Tracker\Request), Object(Piwik\Tracker\Visitor), Object(Piwik\Tracker\ActionPageview)) #7 /var/www/html/core/Tracker/Visit.php(317): Piwik\Tracker\Visit->triggerHookOnDimensions(Array, 'onNewVisit') #8 /var/www/html/core/Tracker/Visit.php(210): Piwik\Tracker\Visit->handleNewVisit(NULL) #9 /var/www/html/core/Tracker.php(172): Piwik\Tracker\Visit->handle() #10 /var/www/html/plugins/QueuedTracking/Queue/Processor/Handler.php(46): Piwik\Tracker->trackRequest(Object(Piwik\Tracker\Request)) #11 /var/www/html/plugins/QueuedTracking/Queue/Processor.php(194): Piwik\Plugins\QueuedTracking\Queue\Processor\Handler->process(Object(Piwik\Tracker), Object(Piwik\Tracker\RequestSet)) #12 /var/www/html/plugins/QueuedTracking/Queue/Processor.php(143): Piwik\Plugins\QueuedTracking\Queue\Processor->processRequestSets(Object(Piwik\Tracker), Array) #13 /var/www/html/plugins/QueuedTracking/Commands/Process.php(88): Piwik\Plugins\QueuedTracking\Queue\Processor->process() #14 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\QueuedTracking\Commands\Process->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #15 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #16 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\Plugins\QueuedTracking\Commands\Process), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Outp ut\ConsoleOutput)) #17 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #18 /var/www/html/core/Console.php(135): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #19 /var/www/html/core/Access.php(670): Piwik\Console->Piwik\{closure}() #20 /var/www/html/core/Console.php(136): Piwik\Access::doAsSuperUser(Object(Closure)) #21 /var/www/html/core/Console.php(87): Piwik\Console->doRunImpl(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #22 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #23 /var/www/html/console(32): Symfony\Component\Console\Application->run() #24 {main} ```

Fail to be processed.

Context

These are "bad" tracking hits, in the sense that utm_campaign[]=... are not technically correct, but we should not be failing hard on processing these hits. Specifically, these errors stall out processing while using the QueuedTracking plugin, and stall all processing of the queue, similar to https://github.com/matomo-org/plugin-QueuedTracking/issues/192

Your Environment

  • Matomo Version: 4.12.3
  • PHP Version: 8.0.26
  • Server Operating System: Docker (alpine) (running on Fedora)
  • Additionally installed plugins:
    • CustomReports
    • MarketingCampaignsReporting
    • CustomAlerts
    • LogViewer
    • InvalidateReports
    • TasksTimetable
    • QueuedTracking
    • AbTesting
    • MediaAnalytics
    • FormAnalytics
    • Funnels
    • RollUpReporting
    • SearchEngineKeywordsPerformance
    • MultiChannelConversionAttribution
    • HeatmapSessionRecording
    • SEOWebVitals
    • UsersFlow
    • ActivityLog
    • WhiteLabel
    • Cohorts
    • AdvertisingConversionExport
    • LoginSaml
    • WooCommerceAnalytics
    • CustomVariables
    • GoogleAnalyticsImporter
@MatomoForumNotifications commented on January 10th 2023

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/mediaanalytics-error-message/48642/4

@bx80 commented on January 12th 2023 Contributor

Thanks for reporting this @djmetzle, could you provide a sample tracking URL so we can be sure that the issue is being recreated properly?

@djmetzle commented on January 12th 2023

@bx80
I'm having a hard time running down one of the tracker hits that specifically broke this...
I did find this stack trace snapshot from our debugging:
typeerror
That led us to Base:
https://github.com/matomo-org/matomo/blob/b83ed614a09d175cb9c6a5042cbb48b6967468ea/plugins/Referrers/Columns/Base.php#L379
Note that urldecode here will only work with strings, not if an array is returned.
That led us to UrlHelper:
https://github.com/matomo-org/matomo/blob/b83ed614a09d175cb9c6a5042cbb48b6967468ea/core/UrlHelper.php#L278
but there appears to to be a bug in the docblock:
https://github.com/matomo-org/matomo/blob/b83ed614a09d175cb9c6a5042cbb48b6967468ea/core/UrlHelper.php#L275
However, the test cases for this method assert that this function will return the type array if needed!
https://github.com/matomo-org/matomo/blob/b83ed614a09d175cb9c6a5042cbb48b6967468ea/tests/PHPUnit/Unit/UrlHelperTest.php#L117-L122
This seems like a bug.

This problem is similar to the issues we've faced previous, basically "array found, but string expected".
Ref: https://github.com/matomo-org/plugin-QueuedTracking/issues/194

Specifically, we are using the QueuedTracking plugin, so when these "hard-stop" type errors occur, our data ingestion queue stops processing completely, and we notice immediately that we stop receiving new tracking data.
That issue was a followon to nearly the same issue we'd raised previously:
https://github.com/matomo-org/plugin-QueuedTracking/issues/192
So this seems like an ongoing issue.

Here are our relevant forum posts as well:


Currently, we are applying several patches in order to continue processing queued tracking data. All three seem to address nearly the same issue:

  • core/Tracker/Request.php

    
    --- Request.php 2022-11-09 13:25:33.000000000 -0700
    +++ Patched.php 2022-11-09 13:31:06.000000000 -0700
    @@ -640,7 +640,8 @@
         // use headers as default if no data was send with the tracking request
         $default = Http::getClientHintsFromServerVariables();
  • $clientHints = Common::getRequestVar('uadata', $default, 'json', $this->params);

  • $varType = is_string($this->params['uadata']) ? 'json' : null;

  • $clientHints = Common::getRequestVar('uadata', $default, $varType, $this->params);

     return is_array($clientHints) ? $clientHints : [];

    }

  • core/UrlHelper.php

    
    --- UrlHelper.php       2023-01-10 11:38:01.000000000 -0700
    +++ Patched.php 2023-01-10 11:37:50.000000000 -0700
    @@ -279,7 +279,7 @@
     {
         $nameToValue = self::getArrayFromQueryString($urlQuery);
  • if (isset($nameToValue[$parameter])) {

  • if (isset($nameToValue[$parameter]) && is_string($nameToValue[$parameter])) {
    return $nameToValue[$parameter];
    }
    return null;

  • plugins/AdvertisingConversionExport/Tracker/RequestProcessor.php

    
    --- RequestProcessor.php
    +++ Patched.php
    @@ -93,7 +93,8 @@
    
         $rawParams = $request->getRawParams();
  • parse_str(parse_url($rawParams['url'] ?? '', PHP_URL_QUERY) ?? '', $params);

  • $rawUrl = is_string($rawParams['url']) ? $rawParams['url'] : '';

  • parse_str(parse_url($rawUrl, PHP_URL_QUERY) ?? '', $params);

     foreach ($providers as $provider) {
         if (empty($provider::CLICK_ID_REQUEST_PARAM)) {

please advise.

@sgiehl commented on January 12th 2023 Member

@djmetzle Thanks for tracing this one down. I guess the utm parameters are actually not supposed to be provided as an array.
But providing them as an array should at least not break the Tracking for sure.
For that I would personally not adjust the UrlHelper. Even though the method is documented to return string|null, that doesn't mean it might be used somewhere and expecting to return an array if one is provided in the url.
I'll set up a quick PR, so that values provided as arrays will simply be ignored.

@djmetzle commented on January 12th 2023

@sgiehl if it's helpful, we swept, and all uses of that particular method are expecting only strings:
image

Only the Test for that method asserts arrays get returned. In fact, almost every usage appears to expect that the return docblock annotation is accurate.

@sgiehl commented on January 12th 2023 Member

@djmetzle I want to avoid introducing any regressions with other plugins that might use the method to receive an array.
The way we are currently using request and input parameters is not very smart, as it a bit error prone. I've already started to introduce new type safe methods that can be used for such cases in the future. But that will be introduced as of Matomo 5 and will take a long time, till all stuff using request parameters will be refactored.
Till then I've create the PR above to fix your specific problem for now.

@djmetzle commented on January 12th 2023

Got it. Awesome, thank you @sgiehl

This Issue was closed on January 13th 2023
Powered by GitHub Issue Mirror