Disable Custom Variables feature by default #11524

We need to deprecate Custom Variables in favor of Custom Dimensions.

In Piwik 4, maybe we could deactivate the CustomVariables plugin, if no Custom Variable has been tracked so far in this Piwik?

Additionally we would need to find ways to promote the Custom Dimensions feature within the product. By showing some ads or notifications in some way.

Thoughts?

Sounds good. I reckon in general it will be a hard transition to get rid of it in terms of providing a good experience to users who still use it. We could for sure mention it in Custom Variables report etc.

Also what about custom variables in scope event?

@tsteur AFAIK the same benefits can be achieved with custom dimensions on events?

Ideas so far for Custom Variables deprecation

• display a message below Custom Variables reports, where we recommend to use Custom Dimensions instead
• in Piwik 4, for new installations, do not enable Custom Variables
• in Piwik 4, for users upgrading, possibly we can detect whether any custom variable has been used so far in this Piwik server, and if never used, disable the plugin automatically
• possibly we can go as far as removing the source code from Piwik release package, and move this plugin to the Marketplace?

Before we can disable custom variables in core, we need to change any core features which currently use Custom Variables (all of it is "hacky"):

• Ecommerce view tracking uses custom variables, which leads to bugs such as https://github.com/piwik/piwik/issues/12035 -> we should rewrite Ecommerce tracking to work without custom variables. ✔️
• Custom vars also used in the Log Importer to
  • set Http-Status codes,
  • whether the user is a bot or not,
  • track the HTTP method GET VS POST
  • etc.
    This would need some adjustements (refs)
• Custom variables also used for Site Search tracking, implementation should be rewritten see #12072 ✔️

The Log Analytics conversion work is well underway in https://github.com/matomo-org/matomo-log-analytics/pull/214 :+1:

Another thing i noticed:

• in the following config settings should be renamed (of course with BC kept)

  datatable_archiving_maximum_rows_custom_variables = 5000
  datatable_archiving_maximum_rows_subtable_custom_variables = 5000

We need to move Custom Variables plugin to Marketplace and add Custom Dimensions directly just fyi

@tsteur so we will move custom variables into a submodule repo and include custom dimensions plugin to core instead of having a submodule?

Yes exactly 👍

I suppose we would also automatically enable the custom dimensions plugin during upgrade.

Ok. Most stuff should be done. We still have open issues for that changes in php-tracker and log-analytics.
Additionally those tasks are still open, if we still want to do that:

• the following config settings should be renamed (of course with BC kept)

  datatable_archiving_maximum_rows_custom_variables = 5000
  datatable_archiving_maximum_rows_subtable_custom_variables = 5000

• In Piwik 4, maybe we could deactivate the CustomVariables plugin, if no Custom Variable has been tracked so far in this Piwik?

• Additionally we would need to find ways to promote the Custom Dimensions feature within the product. By showing some ads or notifications in some way.

In Piwik 4, maybe we could deactivate the CustomVariables plugin, if no Custom Variable has been tracked so far in this Piwik?

That could work but bit scared of such a full table scan. I would say we could check the archives if any custom variable was tracked say in the last 3 years but then you'd need to fetch this for each site... Maybe we would do this logic only if less than 5 sites are used? Actually, I do see few other problems though maybe like it might try to start an archive while the DB is not upgraded yet etc. Maybe we rather not do this. If they haven't used it in a year, they likely also won't start using it maybe and it be then eventually save to remove.

BTW seeing the plugin description is too long: https://github.com/matomo-org/plugin-CustomVariables/blob/4.x-dev/plugin.json#L3 I wonder if we can tweak the description to somehow incorporate "use custom dimensions as the feature is deprecated" to ideally prevent someone from using it again. Could also show a similar message maybe in a footer note in the reports?

Custom Dimensions we enable by the way by default so there shouldn't be any need for extra promotion we only need to convert people from custom variables to custom dimensions.

@sgiehl is there much to do for this one?

Guess it's only "renaming the config value" left to do and the adjustments you mentioned:

BTW seeing the plugin description is too long: https://github.com/matomo-org/plugin-CustomVariables/blob/4.x-dev/plugin.json#L3 I wonder if we can tweak the description to somehow incorporate "use custom dimensions as the feature is deprecated" to ideally prevent someone from using it again. Could also show a similar message maybe in a footer note in the reports?

@mattab in https://github.com/matomo-org/matomo/issues/11524#issuecomment-420207544

I don't know what you mean by renaming eg datatable_archiving_maximum_rows_custom_variables? To what should it be renamed?

@tsteur maybe we can ignore this comment actually... (was only thinking to move it to [CustomVariables] section but that's not needed?

👍 not needed.

@sgiehl I've adjusted the plugin description. Can we close this issue now?

datatable_archiving_maximum_rows_custom_variables is actually also used for custom dimension archiving. If that's fine, we can close the issue...

Sounds good to fix that and use _custom_dimensions. Then we need to update https://matomo.org/faq/how-to/faq_54/ and mention from Matomo 4 the variable name changes.

I've updated the FAQ. So there's nothing left to do.