Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow custom dimensions of any size (custom_dimension_X fields of unlimited length column (TEXT type) instead of VARCHAR 255) #16150

Open
mattab opened this issue Dec 8, 2017 · 20 comments
Labels
c: Custom Dimensions For issues related to the Custom Dimensions plugin. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.

Comments

@mattab
Copy link
Member

mattab commented Dec 8, 2017

Currently custom dimension values are limited to 255 characters, but in some cases we want to track more than 255 characters in a custom dimension.

Currently, if trying to issue a request with a dimension value longer than 255 chars, the tracker API request will fail with an error like

Exception: Error query: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'custom_dimension_2' at row 1 In query: INSERT INTO piwik_log_visit (idvisitor, config_id, location_ip, idsite, visit_first_action_time, visit_goal_buyer, visit_goal_converted, visit_last_action_time, visitor_days_since_first, visitor_days_since_order, visitor_returning, visitor_count_visits, visit_entry_idaction_name, visit_entry_idaction_url, [...], campaign_content, campaign_id, campaign_keyword, campaign_medium, campaign_name, campaign_source, custom_dimension_1, custom_dimension_2)

When manually changing the column type from VARCHAR 255 to TEXT, then the data is tracked properly, and displayed correct in the reports.

---> For users who need to track more than 255 chars, can we recommend them to manually change the DB schema column type to TEXT field? is there any possible problem with this solution?

@mattab mattab changed the title Manually changing the custom_dimension_X fields to be unlimited length fields instead of VARCHAR 255? Manually changing the custom_dimension_X fields to be unlimited length column (TEXT type) instead of VARCHAR 255? Dec 8, 2017
@tsteur
Copy link
Member

tsteur commented Dec 8, 2017

Yes would definitely not making this the default. And would also possibly trim after 255 characters to avoid errors and rather not fully tracking it instead of failing.

@mattab
Copy link
Member Author

mattab commented Dec 10, 2017

Maybe the columns could be to TEXT type in the Custom Dimensions release for Piwik 4.0.0 in the future.

@tsteur tsteur transferred this issue from matomo-org/plugin-CustomDimensions Jun 29, 2020
@tsteur tsteur added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: Custom Dimensions For issues related to the Custom Dimensions plugin. labels Jun 29, 2020
@sgiehl
Copy link
Member

sgiehl commented Jul 21, 2020

@tsteur @mattab is this change something that should be considered for Matomo 4?

@tsteur
Copy link
Member

tsteur commented Jul 21, 2020

Not sure. Personally would keep things the way they are maybe. Eg TEXT can have performance problems creating temporary tables etc. Having an FAQ on how to change this be good though.

@mattab
Copy link
Member Author

mattab commented Jul 23, 2020

Btw before closing we should also update the link in the page: https://matomo.org/docs/data-limits/
which currently link to this issue. It could link to the FAQ explaining how to make the alter table columns to TEXT.

@cah-erinblake
Copy link

Is there any update on this issue? Is this change going to be included in Matomo 4?

@tsteur
Copy link
Member

tsteur commented Aug 9, 2021

It's not included in Matomo 4 but you can change the type manually in your database if needed.

@luflow
Copy link

luflow commented Aug 30, 2021

@tsteur Is there already an FAQ entry for this? Could not find anything until now. Is it just an ALTER TABLE where the type gets changed to TEXT on all custom dimension fields that should get this change?

What happens if a future migration of piwik changes this type back?

@tsteur
Copy link
Member

tsteur commented Aug 30, 2021

@luflow there isn't. AFAIK it's only an alter table query to change field to text like below:

ALTER TABLE matomo_log_visit MODIFY COLUMN custom_dimension_1 TEXT, MODIFY COLUMN custom_dimension_2 TEXT, MODIFY COLUMN custom_dimension_3 TEXT, MODIFY COLUMN custom_dimension_4 TEXT, MODIFY COLUMN custom_dimension_5 TEXT;

I'm not sure what would happen if we were to change it back. I'm assuming the content would be truncated but the historical archived reporting data would still include the full names.

@ashutosh-hs
Copy link

ashutosh-hs commented Jun 10, 2022

I am using matomo 4.6.2
Even after manually changing the column type to text in matomo DB for custom dimension columns, the values are still being truncated to 255 characters.
Is there anyway to get around this limitation at this time?

@ashutosh-hs
Copy link

@mattab @tsteur could you please help me with this? We have now upgraded to version 4.10.1 and still have this issue.
Even after manually changing the column type to text in matomo DB for custom dimension columns, the values are still being truncated to 255 characters.
Is there anything else that I need to do after changing column type?
Is there anyway to get around this limit at this time?

@justinvelluppillai
Copy link
Contributor

Hi @ashutosh-hs we haven't got this on the list of currently prioritised work. You may have a better chance of getting help with this over on our forums https://forum.matomo.org/

@ashutosh-hs
Copy link

@justinvelluppillai thanks.
I have already created a topic on the form : https://forum.matomo.org/t/manually-changing-custom-dimension-x-fields-to-be-unlimited-length-column/46287
haven't received any replies on this yet

@tsteur
Copy link
Member

tsteur commented Jun 24, 2022

Hi @ashutosh-hs the custom dimension value is hard coded to 250 characters: https://github.com/matomo-org/matomo/blob/4.11.0-rc1/plugins/CustomDimensions/Tracker/CustomDimensionsRequestProcessor.php#L175

It was added in Matomo 3.1.8 matomo-org/plugin-CustomDimensions@823e8e9#diff-bbc47867ddfd7fe85fe96b6a846a35ca5064315fb8a12ec8615a4ec69adf6fd3R132-R136 meaning the workaround to use TEXT columns doesn't work anymore since that release.

@ashutosh-hs
Copy link

oh, okay. @tsteur thanks for the confirmation

@justinvelluppillai justinvelluppillai added this to the For Prioritization milestone Jul 6, 2022
@mattab mattab changed the title Manually changing the custom_dimension_X fields to be unlimited length column (TEXT type) instead of VARCHAR 255? Allow custom dimensions of any size (custom_dimension_X fields of unlimited length column (TEXT type) instead of VARCHAR 255) Jul 9, 2022
@Sonofendor
Copy link

Hi @tsteur
Maybe you know if other variables are also hard coded to be specific size? For example 'Event Name/Action/Category/Value'?
Your answer would help me a lot
Thanks

@tsteur
Copy link
Member

tsteur commented Sep 20, 2022

@Sonofendor I may be wrong but I believe event name/action/category are limited to 4000 characters as per the log_action.name field definition. I don't think it's limited in the code though (unless I missed it). I can't really say for any other fields right now

@Sonofendor
Copy link

@tsteur Thanks for your answer!

@Webmasterei
Copy link

@tsteur as time moves on and data collections are getting bigger, I see a larger demand for this to be happening.
I have a usecase now where we want to push all "ecommerce items" on a page/cart/checkout to a dimension but due to this restriction it's not possible. So a big +1 from my side to bring this into priorisation. The Plans to do so are long-time announced.

@Eismann82
Copy link

Hello @tsteur tsteur and the Matomo team,

I would like to make a quick plea for the implementation of unlimited text lengths for the custom_dimension_X fields. In our projects, we regularly hit the limits of the current 255 character limit, which prevents us from utilizing the full depth of our data. An extension to the TEXT type would make us much more flexible and improve the quality of our analyses. We hope for a positive development in this matter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Custom Dimensions For issues related to the Custom Dimensions plugin. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

10 participants