Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Piwik is not using log_* indexes when segments are applied #9073

Closed
quba opened this issue Oct 21, 2015 · 3 comments
Closed

Piwik is not using log_* indexes when segments are applied #9073

quba opened this issue Oct 21, 2015 · 3 comments
Labels
c: Performance For when we could improve the performance / speed of Matomo. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.

Comments

@quba
Copy link
Contributor

quba commented Oct 21, 2015

As in topic. I'll start with an example from a realtime widget with a simple page scope custom var segment:

Id: 5170860
    User: XXXXXX
    Host: XXXXXX
      db: XXXXXX
 Command: Query
    Time: 208
   State: Queried about 60710000 rows
    Info: SELECT
                COUNT(log_inner.visit_last_action_time)
            FROM

        (

            SELECT
                log_visit.visit_last_action_time
            FROM
                piwik_log_visit AS log_visit
                LEFT JOIN piwik_log_link_visit_action AS log_link_visit_action ON log_link_visit_action.idvisit = log_visit.idvisit
            WHERE
                ( log_visit.idsite in ('2') AND log_visit.visit_last_action_time >= '2015-10-20 08:50:48' )
                AND
                ( log_link_visit_action.custom_var_v4 = 'XXXXXX' )
            GROUP BY
                log_visit.idvisit
            ORDER BY
                NULL
        ) AS log_inner

This query is used to calculate number of visits and actions in last X minutes.

I think that instead of this query, we could simply run something like that:

SELECT count(distinct log_link_visit_action.idvisit), count(distinct log_link_visit_action.idlink_va) 

FROM piwik_log_visit AS log_visit 

LEFT JOIN piwik_log_link_visit_action AS log_link_visit_action ON log_link_visit_action.idvisit = log_visit.idvisit 

WHERE (log_link_visit_action.idsite = 2 
AND log_link_visit_action.server_time >= '2015-10-20 08:50:48') 
AND log_link_visit_action.custom_var_v4 = 'XXXXXX';

In this case Piwik is using idsite and server_time columns which are indexed. In my case the difference is huge, first query takes 45sec and second one a few milliseconds.

Additionally it may also be the case that it's slow also for standard archiving queries that require joining log_link_visit_action table when there's an action-scope segment (like event, custom var, page title, etc.). I think that it's worth investigating.

@mattab mattab added the c: Performance For when we could improve the performance / speed of Matomo. label Oct 22, 2015
@mattab mattab modified the milestone: 2.15.1 Nov 12, 2015
@mattab
Copy link
Member

mattab commented Nov 12, 2015

Good find, we have a chance to improve performance of Live API when Segments URL are used. A SQL query that looks at 60 million rows is slow and a bit of a monster... Here you show that if we query another datetime field then the query uses the INDEX and is much faster.

Usually, it does not work to say that log_visit.visit_last_action_time >= '2015-10-20 08:50:48' == log_link_visit_action.server_time >= '2015-10-20 08:50:48', but maybe for this particular API and use case it actually could work?

when it comes to performance, archiving w/ segment seems to be the next bottleneck for Piwik archiving. I'm tentatively adding to 2.15.1 in case this one would be do-able, let's take a look?

@mattab mattab added this to the 2.15.1 milestone Nov 12, 2015
@quba
Copy link
Contributor Author

quba commented Nov 20, 2015

It's possible that this behaviour was bacause of a bug in TokuDB. It started using indexes after executing analyze table on all log_* tables.

@mattab mattab removed this from the 2.15.1 milestone Nov 22, 2015
@mattab
Copy link
Member

mattab commented Nov 23, 2015

@quba closing as this is not something I'm comfortable working on anyway :-)

@mattab mattab closed this as completed Nov 23, 2015
@mattab mattab added the wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it. label Nov 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.
Projects
None yet
Development

No branches or pull requests

2 participants