Hi @Guite
Thanks for the report. it is possible that the core:archive command does not process correctly the ecommerce metrics. If so, this would be a bug.

I generated ecommerce visits and archives were correctly written for day and month via web and via CLI. To test it I deleted the related archive tables and verified that archives were generated.

In the next Piwik release Ecommerce will be split into 4 separate categories. It would be interesting to know which reports are actually slow for you. By any chance are you using Piwik beta versions? The latest beta version already includes that change.

As I am using the latest beta another possibility could be that it was fixed in this version somehow.

The following requests seem to be responsible:

1. index.php?date=2014-10-31&module=Goals&action=ecommerceReport&idSite=123&period=month&idGoal=ecommerceOrder
2. index.php?date=2014-10-31&module=Dashboard&action=getAvailableWidgets&idSite=123&period=month

First I tested different months using one ecommerce site which was very fast. The two requests above were the slowest ones nevertheless - each required about 400 to 500 ms.

Then with having opened the view of October 2014 I switched to another site.
Both requests then required 3m 9s (in this case).
Reloading the page again they needed only a few milliseconds again.

Sorry, a beta version is not applicable. But you could close the issue and I'll reopen if the problem persists in 2.11.0.

Tried to reproduce this for a while again and I think I found a problem. Do you have any Segments configured for that site?

See screenshots what I did is the following. At 19:39 I archived via CLI and it archived correctly day, week, month, year. After the archiving was finished at 19:44 I opened the Ecommerce page and noticed there are more archives. After a while I noticed that it created more archiving because of a Segment that I have configured for that site, at least I presume so. It created those archives even though I was not viewing one of them. This could explain it and would be probably a bug? Presume a segment should be only archived in case it is actually viewed?

Another question is whether you are viewing stats for today? If so, you might want to disable archiving triggered by the browser see http://piwik.org/docs/setup-auto-archiving/#disable-browser-triggers-for-piwik-archiving-and-limit-piwik-reports-to-updating-every-hour . Then the archives won't be reprocessed over and over again while browsing through the stats.

The behaviour that you describe sounds like Segments though. In case you are a super user you can edit the segments and configure them to not process them in real time (triggered via browser) but to process them with the CLI archiver. This would result in more stored data and archiving would take longer but reports are viewed faster.

Nonetheless, I think it would be worth investigating to archive segments only when they are actually viewed?

There are no segments used for any ecommerce site.

Browser-based archiving is already disabled.

Would it help if you get access to my installation (Piwik and database)?

This would be most likely helpful indeed. Feel free to send us details to hello at piwik.org. I'd recommend to create a new user for the Piwik installation that you can delete afterwards. If possible with super user access.

Thx for that! I had a look and learned when opening Goals / Ecommerce there are two segments applied in the background, automatically: visitorType==new and visitorType==returning,visitorType==returningCustomer

Those segments are processed in real time and there is a config setting to process them during CLI archiving see https://github.com/piwik/piwik/blob/2.11.0-b3/config/global.ini.php#L646-L657

To pre-archive those segments during CLI archiving add this to your config:

[Segments]
Segments[]="visitorType==new"
Segments[]="visitorType==returning,visitorType==returningCustomer"

The annoying part is that those segments will be archived even for websites not using Ecommerce or Goals. As explained pre-archiving segments results in longer archiving time and takes more space in the database but makes accessing the data in the browser faster but it would be definitely worth a try to see whether it improves the performance.

Another setting that is interesting is the following:

; if browser trigger archiving is disabled, API requests with a &segment= parameter will still trigger archiving.
; You can force the browser archiving to be disabled in most cases by setting this setting to 1
; The only time that the browser will still trigger archiving is when requesting a custom date range that is not pre-processed yet
browser_archiving_disabled_enforce = 0

In case you decide to pre-archive those segments by adding the mentioned config entry you might also want to set this config entry in the [General] section:

browser_archiving_disabled_enforce = 1

I think this is how one is supposed to fix this issue. Do you mind giving it a try? If it makes it faster feel free to remove the account you created for us again.

@mattab I was a bit surprised by this. We should do something here and I'm not sure whether it is done by mentioning it in FAQ and User Guide as it makes Ecommerce / Goals very slow. Especially in this case.. browser archiving is disabled yet it still triggers browser archiving because of the segments. This is even more of an issue if you do not log in daily and many archives need to be built for many days.

Thanks @tsteur for your investigation. Going to check this out and report back afterwards.

@tsteur The changes you described above works great! Thank you very much.

(One minor thing I noticed is that the widgetVisitFrequencygetSparklines shows always 0 values now. I do not know whether this is related or not though.)

This shouldn't be related to this. Can you maybe create a separate issue for this? Maybe with a screenshot?

I will leave this issue open as I think we have to do more here. At least FAQ / User Guide. Maybe show some notifications in Ecommerce etc. in case request takes longer than X seconds explaining how to make it faster etc

Created a separate issue at #7095

It created those archives even though I was not viewing one of them. This could explain it and would be probably a bug? Presume a segment should be only archived in case it is actually viewed?

The segments should be archived if the segments are set to "Pre-processed by cron" (one of the segment properties) then it's expected that those segments are pre-processed during core:archive.

Otherwise the other segments that we need to pre-process are the Ecommerce ones, and the Returning/New visitor one.

I will leave this issue open as I think we have to do more here.

Desired behavior is that when user has setup core:archive as cron, then the data for Ecommerce and Returning visits reports should always be pre-archived as user would expect. Never should any user have to manually change some INI setting or other tweak to make the data appear, as this is really basic functionality for Piwik. Also as you point out it's important that we only process data with these segments, for the subset of reports that we know the UI queries (eg. the few API that are used to create the Ecommerce report, and the
API used to create the New/Returning report)

(btw I thought this was already working like this and if not then it's definitely Major bug)

I'm not sure if that would work... could we define eg in the report classes if a report uses "hardcoded" segments eg

public function getSegmentsNeededToGenerateReport() { 
    return array('visitorType==new', 'visitorType==returning,...')
}

And in CLI archiver trigger the archiving for those segments and reports via $this->request($urlToRequestReportAPI . $segmentNeededToGenerateReport. 'trigger=archivephp')?

I'm not sure if that would work... could we define eg in the report classes if a report uses "hardcoded" segments eg

I think that's a great idea, and that it would work!

Btw if someone else works on this, we have credentials to hopefully reproduce the issue sent to us by @Guite (ask me)

Tried to reproduce the problem:

• General Settings > Allow Piwik archiving to trigger when reports are viewed from the browser -> NO
• set browser_archiving_disabled_enforce = 1 under [General]
• enable Ecommerce for website X
• run ./console visitorgenerator:generate-visits --idsite=X
• delete archive tables
• go to UI: there is no data in Ecommerce
• run ./console core:archive
• I go to Piwik UI and I see the data in "Ecommerce tab"

-> It works for me...

Another question is whether you are viewing stats for today? If so, you might want to disable archiving triggered by the browser see http://piwik.org/docs/setup-auto-archiving/#disable-browser-triggers-for-piwik-archiving-and-limit-piwik-reports-to-updating-every-hour . Then the archives won't be reprocessed over and over again while browsing through the stats.

+1, Piwik UI could even suggest to user these kind of tips: #6210

Note to self: i need to try reproduce by setting browser_archiving_disabled_enforce = 0

Note: with regards to applying custom segments to Ecommerce report, please note that this is currently broken and was reported in: #6593

Are the default segments commented out?

[Segments]
;Segments[]="visitorType==new"
;Segments[]="visitorType==returning,visitorType==returningCustomer"

By default those segments are commented and therefore not pre-archived. See previous comments simply uncommenting them by default can lead to much longer archiving time + increased DB size I reckon. Ideally we would do this only for reports that actually need the segments.

Yes they are commented out - and still, the data was showing up in the Ecommerce dashboard for those "core segments". It looks to me that the "desired behavior" (in #6986 (comment)) actually works already today as desired

But still @Guite clearly had some problem so i'm not sure - @Guite can you upgrade to 2.12.1 and confirm if you still have the issue?

Already using 2.12.1. What exactly should I test? Reverting all settings done for this issue?

You don't need to do anything. I can reproduce. Sorry for that.

@Guite FYI once this issue is closed and you have updated to Piwik 2.13 you might be able to remove those config entries again as we will take core of this in Piwik itself. There's a chance that you still need it as it might not find some existing archives otherwise. Just give it a try and remove those lines and check whether you can still see data of the previous days.

[Segments]
Segments[]="visitorType==new"
Segments[]="visitorType==returning,visitorType==returningCustomer"

Thanks again @Guite for reporting this issue as otherwise we wouldn't have even noticed there was such problem in Piwik 👍

Works fine with 2.13.0, thank you.