Thanks for the report @robles-design I think @diosmosis is already working on this one

Matomo 4.0.4 Same problem. Some senments are still processed, but most of them are not.

@gusevmaksim for one of the segments that isn't being archived, would you be able to post a screenshot of the segment editor w/ that segment loaded? (Or select the row of the segment in the segment table and post the result?)

For me it works now with 4.0.4 but I had some similar problems after upgrading. Not all segments were created and I got a lot of server errors like Error: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away. After logging in via SSH and running the commands manually all segments were created and the server errors are gone. Maybe this helps.

I updated the database via the console, there were no errors in the process.

@gusevmaksim I think this might be fixed by one of the existing PRs, but just to be certain, can you run php console core:archive -vvv and provide the output? (you can post it here or if you'd like you can email benaka@innocraft.com)

Similar logs for each segment
DEBUG [2020-12-07 06:55:18] 2111952 No next invalidated archive. DEBUG [2020-12-07 06:55:18] 2111952 Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=8&period=day&date=2020-12-07&format=json&segment=pageUrl%3D%3Dhttps%25253A%25252F%25252Fname.com%25252Fn%25252Fn2&trigger=archivephp DEBUG [2020-12-07 06:55:18] 2111952 Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=8&period=day&date=2020-12-06&format=json&segment=pageUrl%3D%3Dhttps%25253A%25252F%25252Fname.com%25252Fn%25252Fn2&trigger=archivephp DEBUG [2020-12-07 06:55:18] 2111952 Running command: /usr/bin/php7.4 -q console climulti:request -q --matomo-domain='name.com' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=8&period=day&date=2020-12-07&format=json&segment=pageUrl%3D%3Dhttps%25253A%25252F%25252Fname.com%25252Fn%25252Fn2&trigger=archivephp&pid=4ab09ab4b0fcfcf280e535719f72b88288762382c41d1d2264b67918aec6a6afbbf6692143defb2ca997a4a6552f67146b510&runid=2111952' > tmp/climulti/4ab09ab4b0fcfcf280e535719f72b88288762382c41d1d2264b67918aec6a6afbbf6692143defb2ca997a4a6552f67146b510.output 2>&1 & DEBUG [2020-12-07 06:55:18] 2111952 Running command: /usr/bin/php7.4 -q console climulti:request -q --matomo-domain='name.com' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=8&period=day&date=2020-12-06&format=json&segment=pageUrl%3D%3Dhttps%25253A%25252F%25252Fname.com%25252Fn%25252Fn2&trigger=archivephp&pid=deea214f77f51a8e85d128284863bb3d14c3edeec6c71dc119ea94b65131d47d4adb40d8a0924ce42fcd6e30c54c6a72d1e61&runid=2111952' > tmp/climulti/deea214f77f51a8e85d128284863bb3d14c3edeec6c71dc119ea94b65131d47d4adb40d8a0924ce42fcd6e30c54c6a72d1e61.output 2>&1 & INFO [2020-12-07 06:55:19] 2111952 Archived website id 8, period = day, date = 2020-12-07, segment = 'pageUrl%3D%3Dhttps%25253A%25252F%25252Fname.com%25252Fn%25252Fn2', 28 visits found. Time elapsed: 1.182s INFO [2020-12-07 06:55:19] 2111952 Archived website id 8, period = day, date = 2020-12-06, segment = 'pageUrl%3D%3Dhttps%25253A%25252F%25252Fname.com%25252Fn%25252Fn2', 80 visits found. Time elapsed: 1.457s DEBUG [2020-12-07 06:55:19] 2111952 No usable archive exists (ts_archived of existing = 2020-12-06 23:08:07, now = 2020-12-07 06:55:19). DEBUG [2020-12-07 06:55:19] 2111952 Processing invalidation: [idinvalidation = 3938, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = done7bfc35dc23212ae2a3d1e39027464615]. DEBUG [2020-12-07 06:55:19] 2111952 Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 3934, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = done397a1a090d98156dfd7993e328151f88] DEBUG [2020-12-07 06:55:19] 2111952 Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 3930, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = done5660f5b1c7c4beaf08aa6bdbbc435272] DEBUG [2020-12-07 06:55:19] 2111952 Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 3926, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = done24f0b143ebdd837993519ac6c80eb00d] DEBUG [2020-12-07 06:55:19] 2111952 Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 3922, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = done39bc6aeb08cc2396db6a8b5da8395850] DEBUG [2020-12-07 06:55:19] 2111952 Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 3918, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = doned7cef626a1d7ae3e99433fc51fb092a8] DEBUG [2020-12-07 06:55:19] 2111952 Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 3914, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = doned038fd355b20a900c3a406d222987711] DEBUG [2020-12-07 06:55:19] 2111952 Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 3910, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = done8d993298787ab91b86e35d6894105000] DEBUG [2020-12-07 06:55:19] 2111952 Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 3906, idsite = 8, period = day(2020-12-07 - 2020-12-07), name = donebaabf09e0b400ef0fe9344f16b4622d3]

@gusevmaksim can you check and see if version 4.0.5 works for you?

The problem remained in place. The segments are empty. In the logs by segments, everything goes well, no errors are visible
INFO [2020-12-09 07:03:01] 3164099 Archived website id 25, period = day, date = 2020-12-09, segment = 'pageUrl%3D%40https%25253A%25252F%25252Fname.com%25252Fn', 127 visits found. Time elapsed: 1.481s

@gusevmaksim In this case I think it may not be related to the archiving changes... can you try requesting the API directly for those requests, eg, sending requests:

http://mymatomo.com/?module=API&method=VisitsSummary.get&idSite=25&period=day&date=2020-12-09&segment=pageUrl%3D%40https%25253A%25252F%25252Fname.com%25252Fn&token_auth=...

http://mymatomo.com/?module=API&method=VisitsSummary.get&idSite=25&period=day&date=2020-12-09&segment=pageUrl%3D%40https%253A%252F%252Fname.com%252Fn&token_auth=...

And for this specific segment, can you load it in matomo and see what the segment query parameter is?

<result> <error message="These reports have no data, because the Segment you requested (syntax) has not yet been processed by the system. Data for this Segment should become available in a few hours when processing completes. (If it does not, there may be a problem.) Please note that you can test whether your segment will work without having to wait for it to be processed by using the Live.getLastVisitsDetails API. When using this API method, you will see which users and actions were matched by your &segment= parameter. This can help you confirm your Segment matches the users and actions you expect it to."/> </result>
2.
<result> <nb_uniq_visitors>13</nb_uniq_visitors> <nb_users>0</nb_users> <nb_visits>14</nb_visits> <nb_actions>22</nb_actions> <nb_visits_converted>11</nb_visits_converted> <bounce_count>7</bounce_count> <sum_visit_length>939</sum_visit_length> <max_actions>3</max_actions> <bounce_rate>50 %</bounce_rate> <nb_actions_per_visit>1.6</nb_actions_per_visit> <avg_time_on_site>67</avg_time_on_site> </result>

@gusevmaksim thanks, this narrows the problem down, I should be able to find a fix.

@gusevmaksim I was able to reproduce the issue and found a fix that worked for me here: #16947 . Would you be able to apply this fix, run core:archive and see if you see a change?

Yes, the segments started working after the fix

But now there is a problem with recalculating segments, the information on the page https://matomo.org/faq/how-to/faq_155/ does not help. Dropping tables piwik_archive_numeric_2020_12 and piwik_archive_blob_2020_12 also. The general statistics are recalculated normally, and the segments are only for today and yesterday.

@gusevmaksim can you try running the core:invalidate-report-data command? Something like:

./console core:invalidate-report-data --dates=2020-11-01,2020-12-14 --periods=day --sites=1 --segment=...

then run core:archive again? (Note: invalidations cascade upwards, so invalidating days automatically invalidates weeks/months/years).

I tried running ./console core: invalidate-report-data --dates = didn't help. The segments are recalculated only for today and yesterday

@gusevmaksim can you post the command you used for core:invalidate-report-data? Specifically the segment. It has to be the value of the segment= param in the URL, urldecoded once. (So if the segment parameter is pageUrl%3D@%25252C, the command line argument would be pageUrl=@%252C.)

./console core:invalidate-report-data --dates=2020-12-04,2020-12-12 Without specifying segments and other parameters. I also tried to delete talits from the database with report archives piwik_archive_numeric_2020_12 и piwik_archive_blob_2020_12. General statistics are recalculated, segments are not.

@gusevmaksim this isn't something I can reproduce, and it's not something I can debug without the command or core:archive output. Would you be able to provide this information by email? Or could you create a new segment that does not have sensitive information, but has one character from the original segment that would be encoded (ie, '/' or ',') and then provide the command + outputs?

I just noticed that the segment, the name of which consists only of the numbers "404", was recalculated normally.

console core:invalidate-report-data --dates=2020-12-12,2020-12-12 --sites=8 --segment=pageUrl%3D%40https%253A%252F%252Fname.com%252Fn Invalidating day periods in 2020-12-12,2020-12-12 [segment = pageUrl=@https%3A%2F%2Fname.com%2Fn]... Invalidating week periods in 2020-12-12,2020-12-12 [segment = pageUrl=@https%3A%2F%2Fname.com%2Fn]... Invalidating month periods in 2020-12-12,2020-12-12 [segment = pageUrl=@https%3A%2F%2Fname.com%2Fn]... Invalidating year periods in 2020-12-12,2020-12-12 [segment = pageUrl=@https%3A%2F%2Fname.com%2Fn]...

This segment is recalculated normally

@gusevmaksim can you try w/ a --segment= value of pageUrl=@https%253A%252F%252Fname.com%252Fn

console core:invalidate-report-data --dates=2020-12-11,2020-12-11 --sites=8 --segment=pageUrl=@https%253A%252F%252Fname.com%252Fn Invalidating day periods in 2020-12-11,2020-12-11 [segment = pageUrl=@https%3A%2F%2Fname.com%2Fn]... Invalidating week periods in 2020-12-11,2020-12-11 [segment = pageUrl=@https%3A%2F%2Fname.com%2Fn]... Invalidating month periods in 2020-12-11,2020-12-11 [segment = pageUrl=@https%3A%2F%2Fname.com%2Fn]... Invalidating year periods in 2020-12-11,2020-12-11 [segment = pageUrl=@https%3A%2F%2Fname.com%2Fn]...

@gusevmaksim did you run console core:archive afterwards? If not, can you? EDIT: Or you can wait & let the cron run as normal.

It gave nothing

@gusevmaksim I'm going to need more information than that to debug your issue. If you can email me, please do so.

Just FYI I'm facing same problem on Matomo 4.0.5.

@aqustiq do you mean you aren't seeing any data for any segments? Or no data for specific segments? Is there data for some days but not all?

@diosmosis I cannot see the data in browser but when I go to segment and click to edit then "Test" I can see data.
When I cannot see data I see blue message:
"These reports have no data, because the Segment you requested (Remote work toolkit) has not yet been processed by the system. Data for this Segment should become available in a few hours when processing completes. (If it does not, there may be a problem.)
Meanwhile you can use the Visitor Log to test whether your segment will match your users correctly by applying it there. When applied, you can see immediately which visits and actions were matched by your segment. This can help you confirm your Segment matches the users and actions you expect it to."

I have update today to version 4.1.0. I have invalidate all reports using the plugin then manually archived.
However segments still not show for days, weeks, and all previous months. For December 2020 month and default "All visits" works.
Also I found weird during manual archiving that segments name not showing. It's just quoted.
Ex:
INFO [2020-12-22 15:34:38] 479595 Archived website id 1, period = day, date = 2020-04-24, segment = '', 502 visits found. Time elapsed: 1.730s
INFO [2020-12-22 15:34:40] 479595 Archived website id 1, period = day, date = 2020-04-23, segment = '', 402 visits found. Time elapsed: 1.179s
INFO [2020-12-22 15:34:40] 479595 Archived website id 1, period = day, date = 2020-04-22, segment = '', 362 visits found. Time elapsed: 1.179s

@aqustiq

Also I found weird during manual archiving that segments name not showing. It's just quoted.

This signals the "All Visits" segment was archived, it just means the segment parameter was empty.

Can you provide the segment the definition for the "Remote work toolkit" segment along with what appears in the segment= query parameter in the URL when you load the segment?

Since updating to 4.1.0 yesterday I have the same issue as in my entry post and errors like this:

`INFO [2020-12-23 14:35:21] 11342 Error: Got invalid response from API request: ?module=API&method=CoreAdminHome.archivent=deviceType%253D%253DSmartphone%252CdeviceType%253D%253DTablet%252CdeviceType%253D%253DPhablet&trigger=archivephp. Resig_device_type' segment value Smartphone,deviceType==Tablet,deviceType==Phablet"}'
ERROR [2020-12-23 14:35:21] 11342 65 total errors during this script execution, please investigate and try and fix thes
ERROR [2020-12-23 14:35:21] 11342 Uncaught exception: /www/htdocs/piwik/core/CronArchive.php(605): 6 and try and fix these errors.
65 total errors during this script execution, please investigate and try and fix these errors.

[Exception]
65 total errors during this script execution, please investigate and try and fix these errors.`

After updating to version 4.1.0, segments were no longer archived again.

@diosmosis
This is the part for URL for working page:
module=CoreHome&action=index&date=today&period=day&idSite=1#?idSite=1&period=day&date=2020-12-23&segment=pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fremote-work-toolkit,pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fworking-securely-home,pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fsetting-your-home-network,pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fsetting-your-home-office,pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fholding-virtual-meetings&category=General_Actions&subcategory=General_Pages

And this is the for not working page:
module=CoreHome&action=index&date=today&period=day&idSite=1#?idSite=1&period=day&date=2020-12-18&segment=pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fremote-work-toolkit,pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fworking-securely-home,pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fsetting-your-home-network,pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fsetting-your-home-office,pageUrl%3D%3Dhttps%25253A%25252F%25252Fintranet.internal.XXXX.xx%25252Fen%25252Fholding-virtual-meetings&category=General_Actions&subcategory=General_Pages

And I checked for all the days before 2020-12-18 doesn't work the page.

@robles-design what you're experiencing should be fixed by #17004

@aqustiq this is the same issue @gusevmaksim is experiencing. I haven't been able to isolate the cause though. Would you be able to provide the full core:archive output w/ --force-idsites=1 --force-idsegments=... (w/ the idsegment of the problematic segment in there). If you can do this, can you email the result to dizzy@innocraft.com? It would also be very helpful if you were able to provide read-only access to your matomo, though it's perfectly understandable if this isn't an option for you. (And one further note, make sure to run the command after invalidating the segment reports.)

@diosmosis thanks, seems to work.

@diosmosis using Invalidate plugin i have invalidate all archives for site 1 and segment "Remote work toolkit".
Then I have run: su apache -s /bin/bash -c "/opt/webapps/matomo/console core:archive --url=https://matomo-prod.internal.XXXX.xx --force-idsites=1 --force-idsegments='Remote work toolkit'"
Now all segments are started to show. Weird fix but worked :)
Thank you,

Note: there are still some issues in 4.1.0 which are hopefully (after #17004 and possibly others) now fixed in 4.1.1 beta. You can use the beta easily, see:
https://matomo.org/faq/how-to-update/faq_159/

if you still have any issue with the beta, please open a new issue, and we'll continue investigating. Thanks!

@gusevmaksim I'm experience the same issue as you with a similar segment.

"Page URL" - "Starts with" - "https://example.com"

All other segments are being processed as expected but this one doesn't. Did you find a way forward for this?