Graph of returning visitors doesn't show returning visitors #17123

I've recognized this a few days after the Update to 4.1.1, but it must have started with the latest beta before the update to 4.1.1 (see screenshot).

Behaviour > Engagement > Returning visitors

@robles-design do you see any data in the reports below this graph and all the other reports?

Here is a screenshot of what I see from the 09. January on. The blue line is the returning visitors line.

@robles-design do you only have this problem for date ranges or also when you select a single day, week or month?

@tsteur same for day and week. Selecting the month there is data, cause there was data until the 8 January (see first screenshot).

@robles-design sorry for all the questions. When you go to "Administration -> General Settings". What does it say for these settings in "archiving settings"?

If you have configured "Archive reports when viewed from the browser: No", I suppose you have a cronjob configured that runs regularly and generates the reports? If that's the case, any chance you can try to execute this manually and see if it shows any error?

@tsteur Yes, you're right. I have a cronjob. No errors after runnig ./console core:archive manually.

INFO [2021-01-22 02:39:11] 7988  ---------------------------
INFO [2021-01-22 02:39:11] 7988  SUMMARY
INFO [2021-01-22 02:39:11] 7988  Processed 457 archives.
INFO [2021-01-22 02:39:11] 7988  Total API requests: 457
INFO [2021-01-22 02:39:11] 7988  done: 457 req, 309011 ms, no error
INFO [2021-01-22 02:39:11] 7988  Time elapsed: 309.011s
INFO [2021-01-22 02:39:11] 7988  ---------------------------
INFO [2021-01-22 02:39:11] 7988  SCHEDULED TASKS
INFO [2021-01-22 02:39:11] 7988  Starting Scheduled tasks... 
INFO [2021-01-22 02:39:11] 7988  done
INFO [2021-01-22 02:39:11] 7988  ---------------------------

@robles-design can you try adding -vvv to the command? just in case it shows some more information

INFO [2021-01-22 03:00:34] 5940 ---------------------------
INFO [2021-01-22 03:00:34] 5940 SUMMARY
INFO [2021-01-22 03:00:34] 5940 Processed 428 archives.
INFO [2021-01-22 03:00:34] 5940 Total API requests: 428
INFO [2021-01-22 03:00:34] 5940 done: 428 req, 300979 ms, no error
INFO [2021-01-22 03:00:34] 5940 Time elapsed: 300.979s
INFO [2021-01-22 03:00:34] 5940 ---------------------------
INFO [2021-01-22 03:00:34] 5940 SCHEDULED TASKS
DEBUG [2021-01-22 03:00:34] 5940 42 scheduled tasks loaded
INFO [2021-01-22 03:00:34] 5940 Starting Scheduled tasks...
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 0:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 1:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 2:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 3:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 4:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 5:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 6:
DEBUG [2021-01-22 03:00:34] 5940 Task Piwik\Plugins\CustomJsTracker\Tasks.updateTracker is scheduled to run again for 2021-01-22.
INFO [2021-01-22 03:00:34] 5940 Scheduler: executing task Piwik\Plugins\CustomJsTracker\Tasks.updateTracker...
DEBUG [2021-01-22 03:00:34] 5940 General tracker cache was re-created.
INFO [2021-01-22 03:00:34] 5940 Scheduler: finished. Time elapsed: 0.020s
DEBUG [2021-01-22 03:00:34] 5940 Task Piwik\Plugins\AbTesting\Tasks.changeExperimentStatusIfNeeded is scheduled to run again for 2021-01-22.
INFO [2021-01-22 03:00:34] 5940 Scheduler: executing task Piwik\Plugins\AbTesting\Tasks.changeExperimentStatusIfNeeded...
INFO [2021-01-22 03:00:34] 5940 Scheduler: finished. Time elapsed: 0.000s
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 7:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 8:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 9:
DEBUG [2021-01-22 03:00:34] 5940 Task Piwik\Plugins\PrivacyManager\Tasks.deleteLogData is scheduled to run again for 2021-01-22.
INFO [2021-01-22 03:00:34] 5940 Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.deleteLogData...
INFO [2021-01-22 03:00:34] 5940 Scheduler: finished. Time elapsed: 0.000s
DEBUG [2021-01-22 03:00:34] 5940 Task Piwik\Plugins\PrivacyManager\Tasks.anonymizePastData is scheduled to run again for 2021-01-22.
INFO [2021-01-22 03:00:34] 5940 Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.anonymizePastData...
INFO [2021-01-22 03:00:34] 5940 Scheduler: finished. Time elapsed: 0.000s
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 10:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 11:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 12:
INFO [2021-01-22 03:00:34] 5940 done
INFO [2021-01-22 03:00:34] 5940 ---------------------------

@robles-design I guess the output before the summary could be quite useful too.

Sorry another question: When you click on Visitors -> Visits log do you see data there?

@tsteur Yes, I see data there.

Sorry, I can't send you the output before the summary, cause these are client sites. What are you searching for? Maybe I can find it.

i have the same ques.

view this in page index.php?module=CoreHome&action=index&idSite=1&period=day&date=today&activated=&limit=100&mode=admin&type=&show=#?idSite=1&period=day&date=2021-01-01&segment=&category=General_Actions&subcategory=VisitorInterest_Engagement

@robles-design is it maybe possible to share the information with some site data removed? Like the name or URL of the sites or if it says the number of visits you could also change that value randomly. We'd ideally have most of the output to fully understand if reports get invalidated and archived etc.

@tsteur do you have an email address for me?

@robles-design hello@matomo.org

@tsteur I send you a mail.

Thanks @robles-design I suppose above screen you see in idSite=4? Do you also have the same problem in other sites?

@robles-design it be also great to post the output of the system report here which you find in Matomo -> Administration -> System Report. There you can copy the system report and it should be automatically anonymised.

@tsteur yes in all sites as I wrote in the mail.

@tsteur you have another mail. Please search for XXX in the file. Not all data was anonymised.

Hi @robles-design, if there are visits alongside the zero returning visitors metric, then the archive process is probably not at fault. Could you try the following things:

• check the API output of the Behaviour > Engagement > Returning visitors graph (via export URL). Does it also report 0 returning visitors?
• check the API output of the VisitsSummary.get report with the following segment: visitorType%3D%3Dreturning%2CvisitorType%3D%3DreturningCustomer. Does this report 0 visits?
• check the Visits Log with the segment visitorType%3D%3Dreturning%2CvisitorType%3D%3DreturningCustomer for a day with no returning visitors. Does it select any visits?

Hi @diosmosis

• check the API output of the Behaviour > Engagement > Returning visitors graph (via export URL). Does it also report 0 returning visitors?

Yes

• check the API output of the VisitsSummary.get report with the following segment: visitorType%3D%3Dreturning%2CvisitorType%3D%3DreturningCustomer. Does this report 0 visits?

Yes

• check the Visits Log with the segment visitorType%3D%3Dreturning%2CvisitorType%3D%3DreturningCustomer for a day with no returning visitors. Does it select any visits?

Yes, it selects visitits.

@robles-design can you run the following sql queries and show the output for both:

SELECT idarchive, name FROM archive_numeric_2021_01 WHERE idsite = {idSite} AND period = 1 AND date1 = '2021-01-18 00:00:00' AND date2 = '2021-01-18 00:00:00' AND name IN ('done', 'donefea44bece172bc9696ae57c26888bf8a', 'donefea44bece172bc9696ae57c26888bf8a.VisitsSummary');

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN ({idArchivesFromPreviousQuery}) AND name = 'nb_visits';

Replace {idSite} with the site ID, and {idArchivesFromPreviousQuery} with the idarchive results from the first query.

@diosmosis I'm not really sure if I did this right with phpMyAdmin, cause I'm not a SQL guy.

SELECT idarchive, name FROM archive_numeric_2021_01 WHERE idsite = 4 AND period = 1 AND date1 = '2021-01-18 00:00:00' AND date2 = '2021-01-18 00:00:00' AND name IN ('done', 'donefea44bece172bc9696ae57c26888bf8a', 'donefea44bece172bc9696ae57c26888bf8a.VisitsSummary');

Results:

idarchive | name
74        | donefea44bece172bc9696ae57c26888bf8a.VisitsSummary
1897591   | done

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN (74) AND name = 'nb_visits';
Results: 0

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN (1897591) AND name = 'nb_visits';
Results:

idarchive | name      | idsite | date1      | date2      | period | ts_archived         | value
1897591   | nb_visits | 4      | 2021-01-18 | 2021-01-18 | 1      | 2021-01-19 00:01:05 | 804

@robles-design thanks! You ran the SQL correctly. Looks like there is an archive for the returning visits metric, but it has no data (this is the one with idarchive = 74; it's there, but has no nb_visits metric). Can you run the following query and show us the output?

SELECT DISTINCT name FROM archive_numeric_2021_01 WHERE idarchive = 74;

Afterwards can you try the following:

1. first check that there are visits in the visits log for 2021-01-18 (if no visits, then don't do the rest of the steps)

2. run the queries:

   DELETE FROM archive_numeric_2021_01 WHERE idarchive = 74;
   DELETE FROM archive_blob_2021_01 WHERE idarchive = 74;

   (this will delete the archive that is supposed to contain the returning visits metric)

3. invalidate the day 2021-01-18 either via the plugin or the CLI command
4. run core:archive with --force-idsites={idSite} --force-date-range=2021-01-18,2021-01-18 -vvv
5. check if there are returning visits for the day 2021-01-18 in the UI
6. (if not, then run the same queries in my last comment again and post the results)

@diosmosis I forgot to run the first SQL query (sorry for that) and started with steps 0–4 with no success. The first time I invalidated the last month via the plugin, because there is no option for a date or date range. The second time I invalidated the date via the CLI. Both times core:archive runs without errors, but no returning visits were shown in the UI.

Step 5

SELECT idarchive, name FROM archive_numeric_2021_01 WHERE idsite = 4 AND period = 1 AND date1 = '2021-01-18 00:00:00' AND date2 = '2021-01-18 00:00:00' AND name IN ('done', 'donefea44bece172bc9696ae57c26888bf8a', 'donefea44bece172bc9696ae57c26888bf8a.VisitsSummary');
idarchive | name
2629994 | done
2630716 | donefea44bece172bc9696ae57c26888bf8a.VisitsSummary

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN (2629994) AND name = 'nb_visits';
idarchive | name      | idsite | date1      | date2      | period | ts_archived         | value
2629994 | nb_visits | 4      | 2021-01-18 | 2021-01-18 | 1      | 2021-01-26 00:23:11 | 804

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN (2630716) AND name = 'nb_visits';
idarchive | name      | idsite | date1      | date2      | period | ts_archived         | value
2630716   | nb_visits | 4      | 2021-01-18 | 2021-01-18 | 1      | 2021-01-26 00:32:30 | 174

Then I remembered a similar case with a self created segment months ago where invalidating didn't work though all commands run without error. The solution was to disable cron in the admin UI, switching to the user UI, click the segments and wondering why it works. After that I switched back to cron and had no problems. Same this time. After these steps the returning visitors are back again :-) I'm back on cron now and will report you if the returning visitors will continue to work.

@diosmosis returning visitors with cron enabled is still not working. All other reports work.

@robles-design would you be able to apply this PR: https://github.com/matomo-org/matomo/pull/17168 and see if it solves your issue? You'll have to invalidate and re-archive the an old period to check.

@diosmosis it works! Thank you!