@robles-design do you see any data in the reports below this graph and all the other reports?

Here is a screenshot of what I see from the 09. January on. The blue line is the returning visitors line.

@robles-design do you only have this problem for date ranges or also when you select a single day, week or month?

@tsteur same for day and week. Selecting the month there is data, cause there was data until the 8 January (see first screenshot).

@robles-design sorry for all the questions. When you go to "Administration -> General Settings". What does it say for these settings in "archiving settings"?

If you have configured "Archive reports when viewed from the browser: No", I suppose you have a cronjob configured that runs regularly and generates the reports? If that's the case, any chance you can try to execute this manually and see if it shows any error?

@tsteur Yes, you're right. I have a cronjob. No errors after runnig ./console core:archive manually.

INFO [2021-01-22 02:39:11] 7988  ---------------------------
INFO [2021-01-22 02:39:11] 7988  SUMMARY
INFO [2021-01-22 02:39:11] 7988  Processed 457 archives.
INFO [2021-01-22 02:39:11] 7988  Total API requests: 457
INFO [2021-01-22 02:39:11] 7988  done: 457 req, 309011 ms, no error
INFO [2021-01-22 02:39:11] 7988  Time elapsed: 309.011s
INFO [2021-01-22 02:39:11] 7988  ---------------------------
INFO [2021-01-22 02:39:11] 7988  SCHEDULED TASKS
INFO [2021-01-22 02:39:11] 7988  Starting Scheduled tasks... 
INFO [2021-01-22 02:39:11] 7988  done
INFO [2021-01-22 02:39:11] 7988  ---------------------------

@robles-design can you try adding -vvv to the command? just in case it shows some more information

INFO [2021-01-22 03:00:34] 5940 ---------------------------
INFO [2021-01-22 03:00:34] 5940 SUMMARY
INFO [2021-01-22 03:00:34] 5940 Processed 428 archives.
INFO [2021-01-22 03:00:34] 5940 Total API requests: 428
INFO [2021-01-22 03:00:34] 5940 done: 428 req, 300979 ms, no error
INFO [2021-01-22 03:00:34] 5940 Time elapsed: 300.979s
INFO [2021-01-22 03:00:34] 5940 ---------------------------
INFO [2021-01-22 03:00:34] 5940 SCHEDULED TASKS
DEBUG [2021-01-22 03:00:34] 5940 42 scheduled tasks loaded
INFO [2021-01-22 03:00:34] 5940 Starting Scheduled tasks...
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 0:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 1:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 2:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 3:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 4:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 5:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 6:
DEBUG [2021-01-22 03:00:34] 5940 Task Piwik\Plugins\CustomJsTracker\Tasks.updateTracker is scheduled to run again for 2021-01-22.
INFO [2021-01-22 03:00:34] 5940 Scheduler: executing task Piwik\Plugins\CustomJsTracker\Tasks.updateTracker...
DEBUG [2021-01-22 03:00:34] 5940 General tracker cache was re-created.
INFO [2021-01-22 03:00:34] 5940 Scheduler: finished. Time elapsed: 0.020s
DEBUG [2021-01-22 03:00:34] 5940 Task Piwik\Plugins\AbTesting\Tasks.changeExperimentStatusIfNeeded is scheduled to run again for 2021-01-22.
INFO [2021-01-22 03:00:34] 5940 Scheduler: executing task Piwik\Plugins\AbTesting\Tasks.changeExperimentStatusIfNeeded...
INFO [2021-01-22 03:00:34] 5940 Scheduler: finished. Time elapsed: 0.000s
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 7:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 8:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 9:
DEBUG [2021-01-22 03:00:34] 5940 Task Piwik\Plugins\PrivacyManager\Tasks.deleteLogData is scheduled to run again for 2021-01-22.
INFO [2021-01-22 03:00:34] 5940 Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.deleteLogData...
INFO [2021-01-22 03:00:34] 5940 Scheduler: finished. Time elapsed: 0.000s
DEBUG [2021-01-22 03:00:34] 5940 Task Piwik\Plugins\PrivacyManager\Tasks.anonymizePastData is scheduled to run again for 2021-01-22.
INFO [2021-01-22 03:00:34] 5940 Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.anonymizePastData...
INFO [2021-01-22 03:00:34] 5940 Scheduler: finished. Time elapsed: 0.000s
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 10:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 11:
DEBUG [2021-01-22 03:00:34] 5940 Executing tasks with priority 12:
INFO [2021-01-22 03:00:34] 5940 done
INFO [2021-01-22 03:00:34] 5940 ---------------------------

@robles-design I guess the output before the summary could be quite useful too.

Sorry another question: When you click on Visitors -> Visits log do you see data there?

@tsteur Yes, I see data there.

Sorry, I can't send you the output before the summary, cause these are client sites. What are you searching for? Maybe I can find it.

i have the same ques.

view this in page index.php?module=CoreHome&action=index&idSite=1&period=day&date=today&activated=&limit=100&mode=admin&type=&show=#?idSite=1&period=day&date=2021-01-01&segment=&category=General_Actions&subcategory=VisitorInterest_Engagement

@robles-design is it maybe possible to share the information with some site data removed? Like the name or URL of the sites or if it says the number of visits you could also change that value randomly. We'd ideally have most of the output to fully understand if reports get invalidated and archived etc.

@tsteur do you have an email address for me?

@robles-design hello@matomo.org

@tsteur I send you a mail.

Thanks @robles-design I suppose above screen you see in idSite=4? Do you also have the same problem in other sites?

@robles-design it be also great to post the output of the system report here which you find in Matomo -> Administration -> System Report. There you can copy the system report and it should be automatically anonymised.

@tsteur yes in all sites as I wrote in the mail.

@tsteur you have another mail. Please search for XXX in the file. Not all data was anonymised.

Hi @robles-design, if there are visits alongside the zero returning visitors metric, then the archive process is probably not at fault. Could you try the following things:

• check the API output of the Behaviour > Engagement > Returning visitors graph (via export URL). Does it also report 0 returning visitors?
• check the API output of the VisitsSummary.get report with the following segment: visitorType%3D%3Dreturning%2CvisitorType%3D%3DreturningCustomer. Does this report 0 visits?
• check the Visits Log with the segment visitorType%3D%3Dreturning%2CvisitorType%3D%3DreturningCustomer for a day with no returning visitors. Does it select any visits?

Hi @diosmosis

• check the API output of the Behaviour > Engagement > Returning visitors graph (via export URL). Does it also report 0 returning visitors?

Yes

• check the API output of the VisitsSummary.get report with the following segment: visitorType%3D%3Dreturning%2CvisitorType%3D%3DreturningCustomer. Does this report 0 visits?

Yes

• check the Visits Log with the segment visitorType%3D%3Dreturning%2CvisitorType%3D%3DreturningCustomer for a day with no returning visitors. Does it select any visits?

Yes, it selects visitits.

@robles-design can you run the following sql queries and show the output for both:

SELECT idarchive, name FROM archive_numeric_2021_01 WHERE idsite = {idSite} AND period = 1 AND date1 = '2021-01-18 00:00:00' AND date2 = '2021-01-18 00:00:00' AND name IN ('done', 'donefea44bece172bc9696ae57c26888bf8a', 'donefea44bece172bc9696ae57c26888bf8a.VisitsSummary');

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN ({idArchivesFromPreviousQuery}) AND name = 'nb_visits';

Replace {idSite} with the site ID, and {idArchivesFromPreviousQuery} with the idarchive results from the first query.

@diosmosis I'm not really sure if I did this right with phpMyAdmin, cause I'm not a SQL guy.

SELECT idarchive, name FROM archive_numeric_2021_01 WHERE idsite = 4 AND period = 1 AND date1 = '2021-01-18 00:00:00' AND date2 = '2021-01-18 00:00:00' AND name IN ('done', 'donefea44bece172bc9696ae57c26888bf8a', 'donefea44bece172bc9696ae57c26888bf8a.VisitsSummary');

Results:

idarchive | name
74        | donefea44bece172bc9696ae57c26888bf8a.VisitsSummary
1897591   | done

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN (74) AND name = 'nb_visits';
Results: 0

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN (1897591) AND name = 'nb_visits';
Results:

idarchive | name      | idsite | date1      | date2      | period | ts_archived         | value
1897591   | nb_visits | 4      | 2021-01-18 | 2021-01-18 | 1      | 2021-01-19 00:01:05 | 804

@robles-design thanks! You ran the SQL correctly. Looks like there is an archive for the returning visits metric, but it has no data (this is the one with idarchive = 74; it's there, but has no nb_visits metric). Can you run the following query and show us the output?

SELECT DISTINCT name FROM archive_numeric_2021_01 WHERE idarchive = 74;

Afterwards can you try the following:

0. first check that there are visits in the visits log for 2021-01-18 (if no visits, then don't do the rest of the steps)

1. run the queries:

   DELETE FROM archive_numeric_2021_01 WHERE idarchive = 74;
   DELETE FROM archive_blob_2021_01 WHERE idarchive = 74;
   

   (this will delete the archive that is supposed to contain the returning visits metric)

2. invalidate the day 2021-01-18 either via the plugin or the CLI command

3. run core:archive with --force-idsites={idSite} --force-date-range=2021-01-18,2021-01-18 -vvv

4. check if there are returning visits for the day 2021-01-18 in the UI

5. (if not, then run the same queries in my last comment again and post the results)

@diosmosis I forgot to run the first SQL query (sorry for that) and started with steps 0–4 with no success. The first time I invalidated the last month via the plugin, because there is no option for a date or date range. The second time I invalidated the date via the CLI. Both times core:archive runs without errors, but no returning visits were shown in the UI.

Step 5

SELECT idarchive, name FROM archive_numeric_2021_01 WHERE idsite = 4 AND period = 1 AND date1 = '2021-01-18 00:00:00' AND date2 = '2021-01-18 00:00:00' AND name IN ('done', 'donefea44bece172bc9696ae57c26888bf8a', 'donefea44bece172bc9696ae57c26888bf8a.VisitsSummary');
idarchive | name
2629994 | done
2630716 | donefea44bece172bc9696ae57c26888bf8a.VisitsSummary

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN (2629994) AND name = 'nb_visits';
idarchive | name      | idsite | date1      | date2      | period | ts_archived         | value
2629994 | nb_visits | 4      | 2021-01-18 | 2021-01-18 | 1      | 2021-01-26 00:23:11 | 804

SELECT * FROM archive_numeric_2021_01 WHERE idarchive IN (2630716) AND name = 'nb_visits';
idarchive | name      | idsite | date1      | date2      | period | ts_archived         | value
2630716   | nb_visits | 4      | 2021-01-18 | 2021-01-18 | 1      | 2021-01-26 00:32:30 | 174

Then I remembered a similar case with a self created segment months ago where invalidating didn't work though all commands run without error. The solution was to disable cron in the admin UI, switching to the user UI, click the segments and wondering why it works. After that I switched back to cron and had no problems. Same this time. After these steps the returning visitors are back again :-) I'm back on cron now and will report you if the returning visitors will continue to work.

Having this problem in multiple instances after updating from <=4.5.0 to >=4.6.1 (all of these are "older" instances that also have been updated from 3.x to 4.x):

@peterbo Are those instances using browser archiving or cron task only?
This might be related to #18362 maybe...

Hey @sgiehl ! These instances are all running on Cron archiving with Browser archiving disabled (and most of them also with browser_archiving_disabled_enforce = 1)

Just wanted to add I couldn't reproduce it on one of our instances (browser archiving disabled, also for segments). Worked for "All Visits" on all our instances and when choosing a segment. Also seems to work on https://demo.matomo.cloud/ ?

For Segments, it is working, just not for "All Visits". I'll gather some more data that could be useful to reproduce this.

Interesting @peterbo I have a feeling this might be fixed by https://github.com/matomo-org/matomo/pull/18403/files then. Let me know in case there's more to reproduce this

Implemented the change on the 11th, but still no engagement metrics for "All Visits". Segmented engagement metrics are still working normally.

@tsteur - figured it out!

In all instances that showed that error, I'm using:

[Segments]
Segments[] = "visitorType==new"
Segments[] = "visitorType==returning,visitorType==returningCustomer"

to speed up the engagement metrics processing. After commenting this out, the Engagement metrics are coming back.

@peterbo That's interesting. I guess that somehow causes problems with the segments matomo tries to process internally. Actually those segments are defined "escaped" (like visitorType%3D%3Dnew). So maybe it considers them as already archived, but doesn't use them later as the segment doesn't exactly match or something like that.
@tsteur guess we should investigate where the problem is. Afaik we don't have any tests with predefined segments. might be good to add some as well.

Great find @peterbo and glad it works now!

As it's probably not too often the case that this happens I moved it out of the milestone for now.

Thanks for contributing to this issue. As it has been a few months since the last activity and we believe this is likely not an issue anymore, we will now close this. If that's not the case, please do feel free to either reopen this issue or open a new one. We will gladly take a look again!