Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

For special dates in evolution graphs, calculate date & timezone together, to get proper result. #13799

Merged
merged 4 commits into from Dec 14, 2018

Conversation

diosmosis
Copy link
Member

@diosmosis diosmosis commented Dec 4, 2018

When a timezone is specified w/ a string like 'today', the result of Date::factory() is incorrect since it will get today in UTC and then apply the timezone. Since today in UTC removes the hour/minutes, the adjusted time can move into a different, incorrect day. (eg, timezone is UTC-5, time is 1 am Dec 3, UTC today is Dec 3 midnight, applying timezone yields Dec 2 19:00). Fixed by calculating timezone & date together using DateTime.

I'm not sure if this will have a performance penalty, hopefully not.

Refs #13786

@diosmosis diosmosis added the Needs Review PRs that need a code review label Dec 4, 2018
@diosmosis diosmosis added this to the 3.8.0 milestone Dec 4, 2018
@tsteur
Copy link
Member

tsteur commented Dec 4, 2018

@mskala could you give this a try?

@ghost
Copy link

ghost commented Dec 4, 2018

It looks good so far.

@diosmosis
Copy link
Member Author

Note: need to add some tests & I think yesterdaySameTime needs a change.

@ghost
Copy link

ghost commented Dec 5, 2018

No, I'm afraid it isn't quite right. Now - December 4, 22:00 EST, which notably is December 5 UTC - I'm getting all zeroes in the "Visits Overview" box. I think that box (which worked correctly before the patch) may now be getting the time zone correction applied twice or something of the kind. All the reports under headings like "Behaviour" and "Acquisition" are saying "There is no data for this report."

@diosmosis
Copy link
Member Author

diosmosis commented Dec 9, 2018

New changes:

  • Noticed dates were getting timezones applied more than they should be. The timestamp is meant to be in UTC and getTimestamp() is meant to apply the timezone, but Date::factory() was applying the timestamp again.
  • The timestamp application in getTimestamp() didn't seem to work for me. adjustForTimezone works, however, so switched to using that.
  • today could only be computed by first finding 'today' in the requested timezone using now in the requested timezone, then undoing that and getting that time in UTC, then using that in a Date instance so eventually the timezone application in getTimestamp() would convert it to the right time.

Think this change should do the trick. EDIT: or maybe not since the build is failing.

CC @tsteur

@tsteur
Copy link
Member

tsteur commented Dec 9, 2018

It's hard to test and reproduce these things (and probably lots of different combinations). I reckon if user confirms it works it should be fine to merge and ideally we release a new beta soon.

@diosmosis
Copy link
Member Author

Will try and fix the build tomorrow, don't want anyone to use the code until every test passes.

@tsteur
Copy link
Member

tsteur commented Dec 9, 2018

Totally 👍

@diosmosis
Copy link
Member Author

@tsteur fixed the build, looks like the timezone calculation was actually reversed before (UTC+1 calculated as UTC-1).

@ghost
Copy link

ghost commented Dec 10, 2018

I don't think this latest patch works at all. With a few days of testing it's giving me weird results, including multiple consecutive days that show the same summary statistics, and stats not restarting at zero at local midnight. I'm finding it hard to characterize exactly what's wrong - it's not as simple as just showing yesterday's stats in place of today, or resetting things at UTC midnight instead of local midnight - and it seems to differ between different Web sites that collect their stats on the same Matomo server. My best guess is that there's something cache-related being screwed up, but resetting the cache doesn't seem to fix it. I just hope that the problem is only in display, so that when there's a version that works, the data collected during these few days won't be trash.

@diosmosis
Copy link
Member Author

@mskala That's a bit harder to diagnose from this end. Could you elaborate on "including multiple consecutive days that show the same summary statistics, and stats not restarting at zero at local midnight"? Screenshots might help if that's possible.

@diosmosis
Copy link
Member Author

@mskala Actually, I think you're right, this change isn't right, I would undo it. If the data doesn't look right, you may have to re-archive it, but I think the underlying log data should be fine.

I'll let you know explicitly when I have something that's worth testing.

@ghost
Copy link

ghost commented Dec 10, 2018

See screenshot.. the Visits Over Time graph shows exactly 37 visits with bounce rate of 77% for both December 8 and 9, which seems like a surprising coincidence. At one point today, shortly after midnight this morning, it was also showing that same number of visits and bounce rate for December 10 and in the Visits Overview box, with new visits not resulting in changes to the Visits Overview box. Some, not all, of the other Web sites on this Matomo installation were showing similar behaviour (identical stats for Dec 8 and 9). However, I haven't been able to reproduce the three-days-in-a-row behaviour when trying it just now. I've been through several replacements of Date.php since.

screenshot_2018-12-10_16-08-30

@diosmosis diosmosis added Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. and removed Needs Review PRs that need a code review labels Dec 11, 2018
@diosmosis diosmosis changed the title For special dates, calculate date & timezone together, to get proper result. For special dates in evolution graphs, calculate date & timezone together, to get proper result. Dec 11, 2018
@diosmosis
Copy link
Member Author

@mskala this PR is ready for another test, make sure to change Date.php & Range.php. These changes shouldn't affect any other part of matomo, so no side effects, but I noticed much of how Date.php deals w/ timezones is just off, so there may be other issues like this elsewhere.

@diosmosis diosmosis added Needs Review PRs that need a code review and removed Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. labels Dec 12, 2018
@ghost
Copy link

ghost commented Dec 14, 2018

Okay, I think this one works. I've run it for a couple of days, testing stuff like before and after midnight local and before and after midnight UTC, and I haven't caught it in any incorrect behaviour. Looks good!

@diosmosis
Copy link
Member Author

Thanks for testing @mskala!

@diosmosis diosmosis merged commit ef32327 into 3.x-dev Dec 14, 2018
@diosmosis diosmosis deleted the 13786-date-factory-timezone branch December 14, 2018 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs Review PRs that need a code review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants