Allow custom dimensions of any size (custom_dimension_X fields of unlimited length column (TEXT type) instead of VARCHAR 255) #16150

Currently custom dimension values are limited to 255 characters, but in some cases we want to track more than 255 characters in a custom dimension.

Currently, if trying to issue a request with a dimension value longer than 255 chars, the tracker API request will fail with an error like

Exception: Error query: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'custom_dimension_2' at row 1 In query: INSERT INTO piwik_log_visit (idvisitor, config_id, location_ip, idsite, visit_first_action_time, visit_goal_buyer, visit_goal_converted, visit_last_action_time, visitor_days_since_first, visitor_days_since_order, visitor_returning, visitor_count_visits, visit_entry_idaction_name, visit_entry_idaction_url, [...], campaign_content, campaign_id, campaign_keyword, campaign_medium, campaign_name, campaign_source, custom_dimension_1, custom_dimension_2)

When manually changing the column type from VARCHAR 255 to TEXT, then the data is tracked properly, and displayed correct in the reports.

---> For users who need to track more than 255 chars, can we recommend them to manually change the DB schema column type to TEXT field? is there any possible problem with this solution?

Yes would definitely not making this the default. And would also possibly trim after 255 characters to avoid errors and rather not fully tracking it instead of failing.

Maybe the columns could be to TEXT type in the Custom Dimensions release for Piwik 4.0.0 in the future.

@tsteur @mattab is this change something that should be considered for Matomo 4?

Not sure. Personally would keep things the way they are maybe. Eg TEXT can have performance problems creating temporary tables etc. Having an FAQ on how to change this be good though.

Btw before closing we should also update the link in the page: https://matomo.org/docs/data-limits/
which currently link to this issue. It could link to the FAQ explaining how to make the alter table columns to TEXT.

Is there any update on this issue? Is this change going to be included in Matomo 4?

It's not included in Matomo 4 but you can change the type manually in your database if needed.

@tsteur Is there already an FAQ entry for this? Could not find anything until now. Is it just an ALTER TABLE where the type gets changed to TEXT on all custom dimension fields that should get this change?

What happens if a future migration of piwik changes this type back?

@luflow there isn't. AFAIK it's only an alter table query to change field to text like below:

ALTER TABLE matomo_log_visit MODIFY COLUMN custom_dimension_1 TEXT, MODIFY COLUMN custom_dimension_2 TEXT, MODIFY COLUMN custom_dimension_3 TEXT, MODIFY COLUMN custom_dimension_4 TEXT, MODIFY COLUMN custom_dimension_5 TEXT;

I'm not sure what would happen if we were to change it back. I'm assuming the content would be truncated but the historical archived reporting data would still include the full names.

I am using matomo 4.6.2
Even after manually changing the column type to text in matomo DB for custom dimension columns, the values are still being truncated to 255 characters.
Is there anyway to get around this limitation at this time?

@mattab @tsteur could you please help me with this? We have now upgraded to version 4.10.1 and still have this issue.
Even after manually changing the column type to text in matomo DB for custom dimension columns, the values are still being truncated to 255 characters.
Is there anything else that I need to do after changing column type?
Is there anyway to get around this limit at this time?

Hi @ashutosh-hs we haven't got this on the list of currently prioritised work. You may have a better chance of getting help with this over on our forums https://forum.matomo.org/

@justinvelluppillai thanks.
I have already created a topic on the form : https://forum.matomo.org/t/manually-changing-custom-dimension-x-fields-to-be-unlimited-length-column/46287
haven't received any replies on this yet

Hi @ashutosh-hs the custom dimension value is hard coded to 250 characters: https://github.com/matomo-org/matomo/blob/4.11.0-rc1/plugins/CustomDimensions/Tracker/CustomDimensionsRequestProcessor.php#L175

It was added in Matomo 3.1.8 https://github.com/matomo-org/plugin-CustomDimensions/commit/823e8e9faab673f16c5964e832f2b926d9cdf7ee#diff-bbc47867ddfd7fe85fe96b6a846a35ca5064315fb8a12ec8615a4ec69adf6fd3R132-R136 meaning the workaround to use TEXT columns doesn't work anymore since that release.

oh, okay. @tsteur thanks for the confirmation

Hi @tsteur
Maybe you know if other variables are also hard coded to be specific size? For example 'Event Name/Action/Category/Value'?
Your answer would help me a lot
Thanks

@Sonofendor I may be wrong but I believe event name/action/category are limited to 4000 characters as per the log_action.name field definition. I don't think it's limited in the code though (unless I missed it). I can't really say for any other fields right now

@tsteur Thanks for your answer!