It would be also good to have information what is currently being processed. Now we get information only about last task and we don't know what is in progress.

@quba we had a look with @mattab and there is a maybe a quick win we can do: showing progress of current requests (when there are many segments). For example:

Will pre-process for website id = 2, range period, 30 segments
Querying URLs 1, 2, 3 of 30
Querying URLs 4, 5, 6 of 30
Querying URLs 7, 8, 9 of 30
...
[...]

Would that be a progress from the current situation?

The problem is that a better solution (e.g. showing the name of the segments) would require much more work because we need to do extensive changes to how the archiving work. So in the meantime we try to find a simpler solution that could help.

@mnapoli I had an idea on how we could log as suggested in issue description: instead of sending all segment URLs to CliMulti at once, we could create chunks of concurrent-requests-per-website elements and then loop through the chunks, log Starting processing for website id = 2, range period, of these 3 Segments { .... } and call Climulti for this chunk only. would that work?

Yes but that would make the feature of CliMulti to chunk and parallelize obsolete, so do we remove multi-threading from the CliMulti to put it in CronArchive? And if we do that that's still not easy as we need to find a way to keep trace of the segment (or message) from the moment we list the segments and add them to the url list, and the moment we iterate through that url list to query each one of them. I.e. same problem as today except we stay in the same class (and add more logic to cronarchive). It's possible, but I don't think it's especially easier or cleaner than another solution.

Quick change for this would be to add callbacks to CliMulti so we can execute code when an archiving job is about to be executed.

I have a POC that has this output:

INFO CoreConsole[2015-04-16 23:22:25] Starting Piwik reports archiving...
INFO CoreConsole[2015-04-16 23:22:25] Will pre-process for website id = 1, day period, 2 segments
INFO CoreConsole[2015-04-16 23:22:25] - pre-processing segment entryPageTitle=@Web%20Analytics
INFO CoreConsole[2015-04-16 23:22:26] - pre-processing segment pageTitle=@Piwik
INFO CoreConsole[2015-04-16 23:22:26] Archived website id = 1, period = day, 6 visits in last last2 days, 0 visits today, Time elapsed: 1.380s
INFO CoreConsole[2015-04-16 23:22:26] Will pre-process for website id = 1, week period, 2 segments
INFO CoreConsole[2015-04-16 23:22:26] - pre-processing segment entryPageTitle=@Web%20Analytics
INFO CoreConsole[2015-04-16 23:22:26] - pre-processing segment pageTitle=@Piwik
INFO CoreConsole[2015-04-16 23:22:36] Archived website id = 1, period = week, 459 visits in last last3 weeks, 301 visits this week, Time elapsed: 9.440s

@mattab & @quba would that be OK?

It looks good to me 👍

here is an idea: add - pre-processing all visits line so we can see from the datetime how long the request 'without segment' took

PR: #7723