Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using period=range, an empty list is returned if there is no data for one date in the date range #19460

Open
Situphen opened this issue Jul 4, 2022 · 10 comments
Labels
answered For when a question was asked and we referred to forum or answered it.

Comments

@Situphen
Copy link

Situphen commented Jul 4, 2022

I am trying to get some statistics such as nb_visits for a specific label (corresponding to a URL) over a specific range of dates, but sometimes Matomo response is just an empty list. While investigating, I found that if there is no data for one date in the date range, then the response is an empty list.

Request with period=day

Request parameters:

/index.php?module=API&format=JSON&idSite=6&period=day&date=2022-06-28,2022-07-02&method=Actions.getPageUrls&label=tutoriels+>+3645+>+demontrer-par-labsurde

Current and correct response:

{
  "2022-06-28": [],
  "2022-06-29": [],
  "2022-06-30": [
    {
      "label": "demontrer-par-labsurde",
      "nb_visits": 8,
      ...
    }
  ],
  "2022-07-01": [
    {
      "label": "demontrer-par-labsurde",
      "nb_visits": 7,
      ...
    }
  ],
  "2022-07-02": [
    {
      "label": "demontrer-par-labsurde",
      "nb_visits": 3,
      ...
    }
  ]
}

Request with period=range

Request parameters:

/index.php?module=API&format=JSON&idSite=6&period=range&date=2022-06-28,2022-07-02&method=Actions.getPageUrls&label=tutoriels+>+3645+>+demontrer-par-labsurde

Current Behavior

Current response:

[]

Expected Behavior

Correct response:

[
  {
    "label": "demontrer-par-labsurde",
    "nb_visits": 18,
    ...
  }
]

Steps to Reproduce (for Bugs)

I guess any requests with method=Actions.getPageUrls, some correct label specified, period=range and date with a date range that as at least one date without data.

Context

See the first paragraph.

Your Environment

  • Matomo Version: 4.10.1
  • PHP Version: at least 7.3
  • Server Operating System: Debian 10
  • Additionally installed plugins: only Google Analytics Importer as far as I know
@Situphen Situphen added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Jul 4, 2022
@peterhashair
Copy link
Contributor

@Situphen thank you for reporting this, our product will review this.

@peterhashair peterhashair added the Needs priority decision This issue may need to be added to the current milestone by Product Manager label Jul 5, 2022
@justinvelluppillai
Copy link
Contributor

@peterhashair it'd be good to confirm we can reproduce this issue and check it's not a regression.

@justinvelluppillai justinvelluppillai removed the Needs priority decision This issue may need to be added to the current milestone by Product Manager label Jul 6, 2022
@peterhashair
Copy link
Contributor

@Situphen trying to reproduce this, but it seems like it works on my local. I notice your label param is tutoriels+>+3645+>+demontrer-par-labsurde, and the actual response label is demontrer-par-labsurde. Can you try a request with label=demontrer-par-labsurde

@Situphen
Copy link
Author

Situphen commented Jul 6, 2022

I tried a request with label=demontrer-par-labsurde as you suggested and I got an empty response : {"2022-06-28":[],"2022-06-29":[],"2022-06-30":[],"2022-07-01":[],"2022-07-02":[]} with period=day and [] with period=range.

@peterhashair
Copy link
Contributor

@Situphen how about without label, and do a search on-page (ctrl+f) for demontrer maybe, just to confirm what the actual label is in the response, it could be caused by special characters.

@Situphen
Copy link
Author

Situphen commented Jul 6, 2022

With period=day

Request parameters without label=... but with expanded=1:

/index.php?module=API&format=JSON&idSite=6&period=day&date=2022-06-28,2022-07-02&method=Actions.getPageUrls&expanded=1

Correct response (3 responses with the label demontrer-par-labsurde as before and 2 with the label /demontrer-par-labsurde.pdf when filtering with demontrer in Firefox) :

image

With period=range

Request parameters without label=... but with expanded=1:

/index.php?module=API&format=JSON&idSite=6&period=range&date=2022-06-28,2022-07-02&method=Actions.getPageUrls&expanded=1

Incorrect response (nothing when filtering with demontrer in Firefox):

image

@peterhashair
Copy link
Contributor

@Situphen thanks for providing this, I will do more investigation on this

@Situphen
Copy link
Author

Situphen commented Jul 7, 2022

As you were not able to reproduce this behavior, I investigated further and for some date range it works and for some other it doesn't. So I downloaded the data with period=day with a date range of 30 days (filename data_day_*.json) as well as the data with period=range for all possible date range combinaison (filename data_range_*.json) with a Python script (get_data_range.py). For each date range combinaison, I compared the nb_visits result I got from period=range to the sum of each daily nb_visits with another Python script (analysis.py). I sorted the date ranges in two lists: correct and incorrect (filename analysis_*.json). Everything is inside this zipfile below. I was not able to find a strict pattern common to all incorrect date ranges, but maybe you will be able to?

test-matomo.zip

@peterhashair
Copy link
Contributor

@Situphen thank you very much for providing the additional info and scripts, it really helps, I will come back to you ASAP.

@sgiehl
Copy link
Member

sgiehl commented Jul 7, 2022

@Situphen I haven't had a look in detail, but that might be an issue of data truncation. When reports are archived, the aggregated data is limited to a certain amount of records. For actions the default is 500 for the base report and 100 for all subtables. Depending on the amount of page you are tracking it might happen that pages that are visited quite few are summarized into a Others row. So in theory a page can be visible on each day report, but if each day has varying pages tracked, a bigger period (e.g. week, month, year or range) might not contain a certain row as the report would have too many records.

@peterhashair peterhashair added answered For when a question was asked and we referred to forum or answered it. and removed Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. labels Nov 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Development

No branches or pull requests

5 participants