fixes #7822
fixes #8462

This PR brings many changes and it was not really possible to do smaller PRs as it all affected each other:

• New Widgets API
• New API methods API.getWidgetMetadata(), API.getReportPagesMetadata()
• Many new classes that will be API and many breaking changes see CHANGELOG.md
• We render the page layout in browser, widgets/reports on any page are now requested in parallel
• New Sparklines visualization
• Any report / plugin can add sparklines to other reports
• One can customize pages (partially)
• Order of reports / widgets are no longer hard coded, they are determined by categories and subcategories
• Widgets & Dashboard menu changed. It should be more intuitive to find a report (similar to the actual reporting menu)
• Url for reporting pages changed: Instead of ...#/idSite=... we now have ...#?idSite= which is more correct (as those are query parameters) and easier to use as it is supported by Angular.

Overall I'd say we got rid of some duplicated code and different UI behaviour. Data is reused where possible so that the order of reports and widgets etc is more consistent and when updated in one position, no update somewhere else is needed. Eg the menu items in "Widgets & Dashboard" selector were completely different to the position of reports in the UI as we forgot to update it.

Pages

We got rid of many controllers and actions that were responsible to render reporting pages. From now on any page is constructed pased on widgets. A page is identifed by a categoryId (eg 'General_Actions') and a subcategoryId (eg 'General_Pages').

On initial reporting page load

There is a new API API.getReportPagesMetadata that returns all reporting pages including the widgets on each page and the structure of each page looks like this:

{
    uniqueId: "General_Actions.General_Downloads",
    category: {
        id: "General_Actions",
        name: "Actions",
        order: 10
    },
    subcategory: {
        id: "General_Downloads",
        name: "Downloads",
        order: 35
    },
    widgets: [
        {
            name: "Downloads",
            module: "Actions",
            action: "getDownloads",
            order: 109,
            parameters: {
                module: "Actions",
                action: "getDownloads"
            },
            uniqueId: "widgetActionsgetDownloads",
            viewDataTable: "table",
            isReport: true
        }
    ]
}

It defines a page for Actions => Downloads and only one widget shall be shown within that page.

The frontend fetches the data of all pages and creates the reporting menu based on this information as well as the currently selected reporting page. It's quite a lot of data to load but with gzipped it's not that much. On my local server it takes about 300ms to generate and load this data, on a fast server it should be even faster. We only load this list on the initial page load and as long as one doesn't change the date or idSite this list does not need to be reloaded. The good thing is, from then on we have the information of all pages. Meaning when clicking on a menu item, we can build the new page very fast and at least the headlines of the reports will be visible immediately.

Having everything based on the above API will make it possible to have a very flexible UI that can be entirely changed eg by a measurable type like "Mobile app".

To build this list, the system looks for all Widget classes provided by plugins, it triggers an event Widget.addWidgetConfigs and it asks each report to add widgets.

Reporting menu

As you may notice for each page there is a category and subcategory. Based on this we can build the menu. Each category represents a main menu item, and each subcategory represents a submenu item of a particular main menu item. They are sorted depending on the specified order. The id of a category or a subcategory will be used in the URL. Eg category=General_Actions&subcategory=General_Downloads will be used to render that page. If there's no categroy or subcategory given, we cannot render a page. If both are not given, we select the inital page which is the page having the lowest category and subcategory order (in our case currently "Dashboard" page).

For the rendering responsible is the new angular directive <div piwik-reporting-menu/>. As soon as someone clicks on a menu item, the URL will be changed and a new page will be rendered immediately because AngularJS triggers a $locationChangeSuccess event.

A typical URL for a reporting page looks like this:

http://apache.piwik/index.php?module=CoreHome&action=index&idSite=1&period=day&date=2015-07-02#?idSite=1&period=day&date=2015-07-02&category=General_Actions&subcategory=General_Pages

Reporting page

The new angular directive <div piwik-reporting-page/> is listening to changes in the URL. As soon as there is a change it renders the requested page depending on the category and subcategory URL parameter. As mentioned the data for each page is already present so we can replace the old page immediately with the new one. Each widget containing the actual data still needs a bit of time to render. As the widgets can now be rendered in parallel (eg some pages have 5 or more widgets on one page) it should be faster to load a particular page.

The reporting page directive is responsible for rendering a particular page. Eg if an evolution and sparkline widget is defined for a page, it will make sure to show these widgets on the top. If there's only one widget it will show that widget with full width (eg "Action pages") unless specified differently via CSS. If there's more than one widget, it will put these widgets automatically into two columns and also take care of proper headline spacing etc.

It is no longer possible to add any menu items via ReportingMenu::add(...). The reporting menu and reporting pages are fully based on the API API.getReportPagesMetadata. If one wants to have something shown in a reporting page, one needs to create a widget (eg via a Report or via Widget class).

Widgets within a page

Each widget that is defined by a widget class or by any report will be shown on a reporting page as long as it defines a categoryId and subcategoryId. If a widget does not define a subcategoryId it won't be visible in a reporting page but it would be still possible to put it on a dashboard or to export it unless $widget->setIsNotWidgetizable() is called. setIsNotWidgetizable() makes it eg possible to put a widget on a reporting page but not have it available in the list of exportable widgets. This is useful eg for Manage Goals and many other widgets.

The order of each widget defines the position of a widget within a page. If a widget is based on a report that has a related report, and this related report would be visible on that page again (eg Devices.getOs and Devices.getOsFamilies are related reports and they would be both visible on one page), then we show only one of those reports (the one having the lower order).

New widget API

Instead of having one Widgets class per plugin that defines multiple widgets we now have one Widget class per widget and it is basically configured like this:

class GetVisitorProfilePopup extends \Piwik\Plugin\Widget
{
    public static function configure(WidgetConfig $config)
    {
        $config->setCategory('Live!');
        $config->setName('Live_VisitorProfile');
        $config->setOrder(25);
        $config->disable();
    }

    public function render()
    {
    }
}

They can be located in Widgets directory and we can now generate more than one widget (as it is one file per widget). Internally we work with WidgetConfig which is a rather stupid data object. The actual widget is only needed to render it. Dependencies can be defined in __construct, the dependencies will be only created if needed to keep gathering widget config information fast. To also keep widget generation fast and cachable we always work with translation keys (eg ->setName('Live_VisitorProfile') instead of ->setName(Piwik::translate('Live_VisitorProfile'))).

WidgetConfig can be also used in a Widget.addWidgetConfigs event:

$config = new WidgetConfig();
$config->setCategory('Live!');
$config->setName('UserCountryMap_RealTimeMap');
$config->setModule('UserCountryMap');
$config->setAction('realtimeMap');
$config->setOrder(5);
$widgetConfigs[] = $config;

// instead of 
WidgetsList::add('Live!', Piwik::translate('UserCountryMap_RealTimeMap'), 'UserCountryMap', 'realtimeMap', array(), $order = 5);

If one wants to generate a widget from a report all one has to do is to define a subcategoryId like this:

    protected function init()
    {
        $this->categoryId    = 'General_Actions';
        $this->subcategoryId = 'General_Pages';
    }
    // this will create a default widget (using default visualization) for this report and it will be displayed in a reporting page having the menu item "Actions -> Pages"

It is the same as doing this manually:

    public function configureWidgets(WidgetsList $widgetsList, ReportWidgetFactory $factory)
    {
        $widgetsList->addWidgetConfig($factory->createWidget());

        // $factory->createWidget() will created a `WidgetConfig` based on all information that is already present in the report (Report name, category, order, subcategory, module, action, ...)
    }

Many widgets can be generated from one report and they can be customized. Eg

    public function configureWidgets(WidgetsList $widgetsList, ReportWidgetFactory $factory)
    {
        $widgetsList->addWidgetConfig(
            $factory->createWidget()
                ->setName('VisitsSummary_WidgetLastVisits')
                ->forceViewDataTable(Evolution::ID)
                ->setAction('getEvolutionGraph')
                ->setOrder(5)
        );

        $widgetsList->addWidgetConfig(
            $factory->createWidget()
                ->setName('VisitsSummary_WidgetVisits')
                ->forceViewDataTable(Sparklines::ID)
                ->setOrder(10)
        );
    }

Widget middleware parameters

There are widgets that should be only shown if an archives contains specific data. For example the "Goals Conversion Overview" widget should be only shown if there are actually conversions for the current selected date/period. As mentioned earlier when requesting the initial page we collect all information about all pages and all widgets. Loading this list would be very slow if we had to archive data and fetch reports in order to build this list. Also as we only load this list initially, a widget might not be added to this list if at the point of the initial page load there are no conversions, but at a later point there might be conversions. Also the list of available widgets wouldn't be cachable if it would depend on archived data since we'd have to invalidate the cache whenever new data is tracked or archived.

To have the gathering of all widgets fast I introduced "Middleware parameters". I named it this way as it is kinda used similar in other frameworks like Slim and Laravel. If a widget should be only shown based on archived data, one can set middleware parameters like this:

$config->setMiddlewareParameters(array('module' => 'Goals', 'action' => 'hasConversions'));

Whenever a page is requested that contains this widget, we will first perform a request to the specified URL (idSite, period, date is added automatically). If the response contains a JSON true we will display the widget, otherwise not. If one clicks on the same page again, we will again perform this check. Meaning even if initially there are no conversions, if one clicks on the link again the widget might be shown if there were conversions meanwhile.

Widget container

Sometimes the structure of the UI might be a bit more complex. As mentioned before: As soon as there is more than one widget on a page, they are shown in two columns. If you want to have two independent widgets below each other (and not next to each other) you need to group them as done for example in the getContinent report:

public function configureWidgets(WidgetsList $widgetsList, ReportWidgetFactory $factory)
    {
    $containerId = 'Continent';
    $container = $factory->createContainerWidget($containerId);
    $widgetsList->addWidgetConfig($container);

    $container->addWidgetConfig($factory->createWidget());

    $widget = $factory->createWidget()->setAction('getDistinctCountries');
    $container->addWidgetConfig($containerId, $widget);
}

Any other report can add reports to this container via the $containerId as well:

public function configureWidgets(WidgetsList $widgetsList, ReportWidgetFactory $factory)
    {
    $widget = $factory->createWidget();
    $widgetsList->addToContainerWidget($containerId = 'Container', $widget);
}

Container can as well have a layout in case they are not supposed to be shown below each other. For example you could set a layout ByDimension which will show only one of the widgets at a time on the right side and a menu to select a widget on the left side.

$container->setLayout(CoreHome::WIDGET_CONTAINER_LAYOUT_BY_DIMENSION);

Widget containers can contain any number of widgets and even widget container itself.

Changing the order of categories and subcategories

As seen earlier in the output a category and subcategory have an order to find the correct position:

    category: {
        id: "General_Actions",
        name: "Actions",
        order: 10
    },
    subcategory: {
        id: "General_Downloads",
        name: "Downloads",
        order: 35
    },

A report consists of a category and optionally a subcategory. The order of the category / subcategory is now used to sort reports (eg for API.getReportMetadata). This was hard coded in the past.

A widget conists of a category and optionally a subcategory as well. We use this to sort eg the output of API.getReportPagesMetadata and API.getWidgetMetadata. Meaning the lower the order of the category, the further left the menu item for that category will be shown. Same for subcategories.

By default each category and subcategory have an order of 99. It is possible to customize this behavior though by defining a category class:

class VisitorsCategory extends Category
{
    protected $id = 'General_Visitors';
    protected $order = 5;
}

or by defining a subcategory class:

class VisitorsOverviewSubcategory extends Subcategory
{
    protected $categoryId = 'General_Visitors';
    protected $id = 'General_Overview';
    protected $order = 2;

}

Those classes can be placed in a plugin in a Categories folder. I did this for all of our plugins to make sure we have the same order in reporting menu as before. A plugin developer would not have to take care of this as those menu items would be just shown after our core menu items (because orderId is 99 by default).

Sparklines Visualization

The new sparklines visualization shows one or multiple sparklines. Each sparkline item consists of a sparkline graph, at least one metric and one description. Optionally it can show an evolution beside each item. For more documentation see the Sparklines and Sparklines\Config PHP docs. An example usage in a report looks like this:

    public function configureWidgets(WidgetsList $widgetsList, ReportWidgetFactory $factory)
    {
        $widgetsList->addWidgetConfig(
            $factory->createWidget()->forceViewDataTable(Sparklines::ID)
        );
    }

    public function configureView(ViewDataTable $view)
    {
        if ($view->isViewDataTableId(Sparklines::ID)) {
            $view->config->addSparklineMetric(array('nb_visits_returning'));
            $view->config->addSparklineMetric(array('avg_time_on_site_returning'));
            $view->config->addSparklineMetric(array('nb_actions_returning', 'nb_actions'));
            $view->config->addTranslations(array(
                'nb_visits_returning' => 'returning visits',
                'avg_time_on_site_returning' => 'avg time on site',
                'nb_actions_returning' => 'returning actions: %s', // %s can be used optionally, by default value is shown first before the description,
                'nb_actions' => 'of %s total actions'
            ));
        }
    }

Any report can listen to the ViewDataTable.configure event and add more sparklines.

    public function getListHooksRegistered()
    {
        return array(
            'ViewDataTable.configure' => 'configureViewDataTable',
        );
    }

    public function configureViewDataTable(ViewDataTable $view)
    {
        if ($view->requestConfig->apiMethodToRequestDataTable === 'Goals.get' && 
            $view->isViewDataTableId(Sparklines::ID)) {
            $view->config->addSparklineMetric(array('nb_visits_returning'));
            $view->config->addTranslations(array(
                'nb_visits_returning' => 'returning visits'
            ));
        }
    }

New angular directives

Besides the mentioned directives piwik-reporting-menu and piwik-reporting-page there are following new directives:

• <div piwik-activity-indicator loading="true/false"/> Shows our general loading message and the typical spinning wheel.
• <div piwik-dashboard dashboard-id="5"/> Shows the dashboard having the specified ID.
• <div piwik-popover-handler/> If present on any page will listen to the popover URL parameter and open or close a popover. Only needed on a page initially.
• <div piwik-widget="widget" widetized="true/false"/> renders any kind of widget as returned by the widget metadata API.
• <div piwik-widget-container="containerWidget"/> renders a container widget as returned by the widget metadata API. All widgets within this container are rendered below each other.
• <div piwik-widget-by-dimension-container="containerWidget"/> renders a container widget as returned by the widget metadata API. It shows a menu for each widget on the left and the select widget on the right.
• <div piwik-widget-loader="{module: '', action: '', ...}"/> similar to ng-include can request any action and shows the returned response of this action within this <div/>.

A reporting page looks pretty much like this:

<div piwik-popover-handler/>
<div piwik-reporting-menu/>
<div piwik-reporting-page>
    <div piwik-widget><div piwik-widget-loader></div>
    <div piwik-widget>
        <div piwik-widget-container>
            <div piwik-widget><div piwik-widget-loader></div>
            <div piwik-widget><div piwik-widget-loader></div>
            <div piwik-widget><div piwik-widget-loader></div>
        </div>
    </div>
    <div piwik-widget><div piwik-widget-loader></div>
    <div piwik-widget><div piwik-widget-loader></div>
</div>

Furthermore we got kinda rid of the broadcast JS object. It is only used by Piwik Overlays now. It's still used to get params though. However, it is no longer used to fetch pages, to handle the history etc. Instead we use Angulars builtin $location where possible. This is the first step towards using Angulars builtin $routing but to actually use it, we need to do more work.

Other new API's

There's as well a new API method API.getWidgetMetadata. It is only kinda similar to API.getReportPagesMetadata.

• API.getReportPagesMetadata returns information about all reporting UI pages. They are grouped by category and subcategory and only widgets will be shown in this list, if they have actually both a category and a subcategory defined. A widget that has no subcategory defined won't be displayed in a reporting page.
• API.getWidgetMetadata is a simple list of all available widgets that can be added to the dashboard or exported via an iframe etc. It also contains widgets that have no subcategory defined. A category is mandatory though.

If a widget should be shown in a reporting page but not added to the widget metadata, one can call $widgetConfig->setIsNotWidgetizable().

If a widget shall be not shown in a reporting page but widgetizable one should simply not set a subcategoryId

Small overview of new classes

The diff looks really big but it is not that much actually. New are mainly the following classes. Otherwise there are more or less only changes in the plugin to use the new page layout rendering mechanism instead of the old controller logic.

• Piwik\Category\CategoryList Holds a list of all available categories
• Piwik\Category\Category Defines a single category. A category holds a list of subcategories
• Piwik\Category\Subcategory (API) Defines a single subcategory.
• Piwik\Plugin\Categories Finds all categories and subcategories implemented by plugins
• Piwik\Widget\Widget (API) Defines a standalone widget
• Piwik\Widget\WidgetConfig (API) Data object that defines a widget configuration.
• Piwik\Widget\WidgetContainerConfig (API) Data object that defines a widget container configuration.
• Piwik\Widget\WidgetsList (API) Holds a list of all configured widgets and widget container
• Piwik\Plugin\Widgets Finds all widgets and widget configs implemented by plugins
• Piwik\Report\ReportWidgetConfig (API) Data object for a widget configuration that is based on a report or created by a report. Allows you to set eg a viewDataTable to render etc
• Piwik\Report\ReportWidgetFactory (API) Factory that lets you easily created widgets from a report
• Piwik\Plugin\Reports Finds a specific report or all reports implemented by plugins

Code organization

I made a few changes to the code organization. Eg Widgets is no longer under Piwik\Plugin\Widget, it is under Piwik\Widget\Widget and everything under this namespace contains so far only code related to Widgets (not really any other dependencies). All category related classes are under Piwik\Category. They do not really know anything about Piwik and certainly not know anything about Widgets or Reports.

I moved some code that finds all components (reports/widgets/categories) that are defined by plugins to Piwik\Plugin. For example moved Piwik\Plugin\Report::factory() and Piwik\Plugin\Report::getAllReports() to a different class named Piwik\Plugin\Reports. Same for Piwik\Plugin\Categories and Piwik\Plugin\Widgets. For example:

• Piwik\Plugin\Categories::getAllCategories()
• Piwik\Plugin\Categories::getAllSubcategories()
• Piwik\Plugin\Widgets::getAllWidgets()
• Piwik\Plugin\Widgets::getAllWidgetConfigs()
• Piwik\Plugin\Widgets::getAllWidgetContainerConfigs()

They are under Piwik\Plugin as they find only concrete Report, Widget, Category, ... classes implemented by plugins. It would be nice to move most other classes at some point out of Piwik\Plugin and eg have instead of a Piwik\Plugin\Report a Piwik\Report\Report class.

Better widgets list

See following image. We make sure it is more intuitive to find a widget by reusing the structure of the menu. As soon as a subcategory contains 3 or more widgets, we add a new menu item to the list. Eg Visitors - Times or Ecommerce - Products, otherwise they are listed in the related category eg Visitors.

Todo

• We no longer have different naming for an exported widget and for the headline within a reporting page. They use the same naming for consistency, simplicity and because it is easier to implement and understand. We will probably need to adjust some namings of some widgets.
• There might be still a view TODO in the code but these can be ignored
• At some point we should maybe adjust the output of API.getReportMetadata to instead of return category: 'Visits', 'subcategory': 'Engagement' the same structure as here with name, id and order: category: {id: '...', name: '...', order: '...'}

UI tests:

• http://builds-artifacts.piwik.org/ui-tests.7822_4/14473.7/UIIntegrationTest_menu_apidisallowed this one is expected to be changed as we no longer request the API or any controller or so when a reporting page is selected. We could alternatively remove it