It might be caused by #17102
but I have no idea how this can result in different numbers being sent to Matomo

I'll put it for now as a regression in the milestone just to make sure we look into it and see if there is an issue or not.

There might be indeed a problem

new:

old:

Seems mostly it behaves the same but eg not for domLoading

Also maybe some rounding should be added just to be safe to not dealing with issues based on this and to make sure the tracking request isn't unnecessary long.

In Firefox it's pretty much the same

Overall it maybe shouldn't really result in double performance. From what I see it should report maybe a wee bit less. Unless some other browsers behave differently. Is this maybe happening for someone else too? Just to make sure that not the site performance actually regressed on the user's website (meaning tracking is all good).

There is a problem. I've been seeing this for the last days now, ever since the upgrade. Metrics are messed up, server time appears to be near double, DOM metrics have nearly disappeared, and when the page performance icon is opened it doesn't register the visits any more.

Also: I had auto-archiving set with a cron job every hour. Now, though the archiving log shows no errors and neither do the system diagnostics, the metrics take several hours to update. The latest visits show up regularly.

Whatever has happened, maybe it is better to simply restore the previous configuration like version 4.1.1 and leave it like that for some time, until the problems are sorted out.

PS: I'm considering downgrading. Will there be any problems with the database?

UPDATE

I posted about this issue in matomo forum and since the new performance API was the problem, I was advised to replace piwik.js, matomo.js and the files in js/ with those of 4.1.1 version.

Everything returned back to normal. No more crazy server fluctuations and disappearing DOM.

All the other problems are also fixed. The hourly updating is back, visits are no longer missing from the page performance panel, and some unusual server load is also gone and activity is back to normal.

I have two separate sites, very different, and both were badly affected by this 4.2 update. I am not a programmer expert, but this new performance API is NOT READY for live sites and real-world conditions.

Hi @MaghSamana, this might be a bit late, but did you see more metric issues for certain browsers? Ie, did visitors who used certain browsers have less accurate data than others?

Also, when you say everything returned to normal, do you mean old dates that had bad data before now have good data?

I couldn't tell about browsers because in most cases the visits no longer appeared in the page performance tab. And the general performance metrics themselves took over half a day (or a whole day) to update, instead of every 1 hour.

In first picture [capture1] you can see what happened after the update for 3 days - until I copied the .js files from the previous version (with the former performance API). The "server time" appeared to double while the DOM metrics disappeared. The 1st and 3rd day are only from organic traffic, the 2nd include intensive testing from sites like Gtmetrix, Webpagetest.org etc.

The second picture [capture 2] comes from another site, where things happened differently - the "server time" appeared to be lower than usual, and again DOM metrics were gone. Again I applied the same solution, coping the previous .js files.

Hi @MaghSamana thanks for your reply! Just to better understand, are you saying when you downgraded, you only replaced your piwik.js/matomo.js file but kept everything else at the higher Matomo version (ie, php code)?

And if you're still willing to help us find the issue, would you be able to check the browsers report for the days in the second site where there are no dom metrics, and post some of the browsers? It would also be helpful to see the site so we can take a look at the tracking code and see what the browsers do (you can email me at dizzy@innocraft.com if you don't want to post that publicly).

there’s for sure some bug in there like I mentioned in #17259 (comment) eg the dom timing doesn’t work at all and it needs some rounding. Likely not related to the actual issue though but be still great to fix.

Looking at the segmented visitor log in our own instance I see a similar issue on an iPad running iOS9.3 with Safari 9. This might be something to look into if we can reproduce it there.

I was going to say maybe overall it shouldn’t track any unrealistic time as it messes up with the data but in some cases hours of transfer time technically could be realistic but then it would likely also create a new visit or something if it actually took > 4 hours and it’s likely better to see these huge times so we can debug and fix the issues.
Overall it might be an easy fix to first try using the old API which is likely to be more correct across browsers (especially older ones) and only use the newer API if the older is not supported anymore instead of using the newer one by default maybe https://github.com/matomo-org/matomo/pull/17130/files

@tsteur it seems the domLoading attribute has been removed meanwhile. See w3c/navigation-timing@5d7657e

I had built the metrics based on the old definitions:

The new are actually like this:

Guess the most closest approach might be to use responseEnd instead of domLoading

I checked a couple of times in firefox:

let a = performance.getEntriesByType('navigation')[0]
let b = performance.timing
a.connectEnd - a.fetchStart => 0
b.connectEnd - b.fetchStart => 0

a.responseStart - a.requestStart => 676
b.responseStart - b.requestStart => 675

a.responseEnd - a.responseStart => 0
b.responseEnd - b.responseStart => 0

a.domInteractive - a.domLoading => NaN
b.domInteractive - b.domLoading => 22
b.domInteractive - b.responseEnd
a.domInteractive - a.responseEnd => 23

a.domComplete - a.domInteractive => 89
b.domComplete - b.domInteractive => 88

a.loadEventEnd - a.loadEventStart => 50
b.loadEventEnd - b.loadEventStart => 50


TEST2
a.connectEnd - a.fetchStart => 0
b.connectEnd - b.fetchStart => 0

a.responseStart - a.requestStart => 312
b.responseStart - b.requestStart => 313

a.responseEnd - a.responseStart => 0
b.responseEnd - b.responseStart => 0

a.domInteractive - a.domLoading => NaN
b.domInteractive - b.domLoading => 558
b.domInteractive - b.responseEnd => 267
a.domInteractive - a.responseEnd => 888

a.domComplete - a.domInteractive => 5313
b.domComplete - b.domInteractive => 5313

a.loadEventEnd - a.loadEventStart => 9
b.loadEventEnd - b.loadEventStart => 8

domLoading is really missing now, so the time_dom_processing column is null using the new tracker version. I make a pr quickly for fix that.

Hi @MaghSamana thanks for your reply! Just to better understand, are you saying when you downgraded, you only replaced your piwik.js/matomo.js file but kept everything else at the higher Matomo version (ie, php code)?

And if you're still willing to help us find the issue, would you be able to check the browsers report for the days in the second site where there are no dom metrics, and post some of the browsers? It would also be helpful to see the site so we can take a look at the tracking code and see what the browsers do (you can email me at dizzy@innocraft.com if you don't want to post that publicly).

Sorry for the late reply. I found some time to monitor this at varied occasions. The problem isn't tied to specific browsers - any visitor browser and any version, gives wrong metrics. In fact, there aren't any - everything becomes "server time" and all other metrics vanish.

I think the old API is still better for now.

@diosmosis Also: the tracking code I use is the very basic one (which is generated automatically by Matomo in the relevant settings page). No additional js or events.

Hi @MaghSamana, thanks for the reply! #17427 should fix the issue