Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DB field piwik_log_visit.location_provider too small #9564

Closed
theyosh opened this issue Jan 19, 2016 · 5 comments · Fixed by #10003
Closed

DB field piwik_log_visit.location_provider too small #9564

theyosh opened this issue Jan 19, 2016 · 5 comments · Fixed by #10003
Assignees
Labels
Bug For errors / faults / flaws / inconsistencies etc.
Milestone

Comments

@theyosh
Copy link
Contributor

theyosh commented Jan 19, 2016

Default the field piwik_log_visit.location_provider is 100 chars. But that is to small. I have already data that is 153 chars long. So I manually changed the length to about 200 chars.

ex of an location entry:
Institute of Bioorganic Chemistry Polish Academy o - AS9112 Institute of Bioorganic Chemistry Polish Academy of Science, Poznan Supercomputing and Networ

So I think this field should be updated to a length of 200 chars.
Else you will get exceptions in PHP, and the loglines are not imported / processed.

This is found in version 2.15.0

Error:
2016/01/19 14:23:36 [error] 30919#0: *14833 FastCGI sent in stderr: "PHP message: Error in Piwik (tracker): Error query: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'location_provider' at row 1 In query: INSERT INTO piwik_log_visit (idvisitor, config_id, location_ip, idsite, visit_first_action_time, visit_goal_buyer, visit_goal_converted, visit_last_action_time, visitor_days_since_first, visitor_days_since_order, visitor_returning, visitor_count_visits, visit_entry_idaction_name, visit_entry_idaction_url, visit_exit_idaction_name, visit_exit_idaction_url, visit_total_actions, visit_total_searches, referer_keyword, referer_name, referer_type, referer_url, location_browser_lang, config_browser_engine, config_browser_name, config_browser_version, config_device_brand, config_device_model, config_device_type, config_os, config_os_version, visit_total_events, visitor_localtime, visitor_days_since_last, config_resolution, config_cookie, config_director, config_flash, config_gears, config_java, config_pdf, config_quicktime, config_realplayer, confi" while reading response header from upstream, client: 82.150.152.12, server: stats.streamzillacdn.com, request: "POST /piwik.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "stats.streamzillacdn.com"

@tsteur
Copy link
Member

tsteur commented Jan 19, 2016

We should increase it, however we cannot do this in Piwik 2 as the update could take too long for some database. I'll move it to Piwik 3.0 for now @mattab feel free to change or close if you think it is not needed.

Not sure why there is a PHP error as we trim the length of a provider to max 100 characters here: https://github.com/piwik/piwik/blob/2.16.0-b3/plugins/Provider/Columns/Provider.php#L77

@tsteur tsteur added the Bug For errors / faults / flaws / inconsistencies etc. label Jan 19, 2016
@tsteur tsteur added this to the 3.0.0-b2 milestone Jan 19, 2016
@theyosh
Copy link
Contributor Author

theyosh commented Jan 19, 2016

Not sure if the javascript tracker was causing this or the python import script. I did see that the python import script did not process the data. I would expect that both trackers would use the same php code.
Maybe an difference in character set? Not sure, but could be that a multi byte character is counted as one character in PHP and double in MySQL?

And yes, changing this will take a long time with a full table. But I think this could be put with the other issues I had with to small database fields. Like the int vs bigint issue with auto increment fields. And I had also another issue with a small int field.

@tsteur
Copy link
Member

tsteur commented Jan 19, 2016

Yes, it must be a character set problem since both use same PHP code. We should possibly switch to mb_substr instead of substr. Also for other fields.

We can provide this update with the other issues you created #9565 and #3288 but we committed to not have any such db updates in Piwik 2.X. Piwik 2.16.0 should be the last minor release of Piwik and then only some patch releases will follow see http://piwik.org/blog/2016/01/announcing-long-term-support-in-piwik-2-the-analytics-platform-for-your-mission-critical-projects/ so we can only include it in Piwik 3

@theyosh
Copy link
Contributor Author

theyosh commented Jan 19, 2016

I agree with the upgrade path. So for now, just collect those bugs, and with fix them in a fresh 3.0 version.

@mattab mattab changed the title DB field piwik_log_visit.location_provider to small DB field piwik_log_visit.location_provider too small Jan 22, 2016
@mattab mattab modified the milestones: 3.0.0-b2, 3.0.0-b1 Jan 22, 2016
@tsteur tsteur self-assigned this Apr 4, 2016
tsteur added a commit that referenced this issue Apr 18, 2016
* fixes #9564 fix DB field piwik_log_visit.location_provider too small

* use new plugins updater API
tsteur added a commit that referenced this issue Jun 3, 2016
* fixes #9564 fix DB field piwik_log_visit.location_provider too small

* use new plugins updater API
tsteur added a commit that referenced this issue Aug 29, 2016
…10397)

* improved ui and responsiveness

* improve rss widget

* commit changes for ui again, got lost after the last commit

* fix more tests

* restoring files

* fix fonts

* fix more tests

* more test fixes

* fix some system tests

* fix tests

* fix system and ui tests

* fix updater tests

* make a page as loaded once the callback is called

* enable verbose

* more verbose output

* enable phantomjs debug flag

* debug should be a phantomjs option

* trying to fix installation tests

* fixes #10173 to not compile css files as less

* trying to minimize js/css requests to hopefully prevent random ui test fails

* disable verbose mode

* fix updater and installation

* lots of bugfixes and ui tweaks

* fix reset dashboard

* various bugfixes

* fix integration tests

* fix text color

* hoping to fix installation tests this way

* cache css/js resources for an hour, should speed up tests and prevent some random issues

* we need to avoid installing plugins multiple times at the same time when requesting resources

* finally getting the colors right again

* fix most tests, more tests for theme

* use an h2 element for titles for better accessibility

* fix headline color

* use actual theme text color (piwik-black)

* fix small font size was applied on all p elements

* fix tests

* now improving all the datatables

* trying to ignore images for visitor log

* Revert "trying to ignore images for visitor log"

This reverts commit ad1ff72.

* fix tests

* fix we had always ignored a max label width

* trying to fix file permissions

* fix more file permissions

* Improved plugins update API (#10028)

* refs #7983 let plugins add or remove fields to websites and better settings api

* * Hide CorePluginsAdmin API methods
* More documentation
* Added some more tests

* improved updates API for plugins

* better error code as duplicate column cannot really happen when not actually renaming a colum

Conflicts:
	core/Updates/3.0.0-b1.php
	plugins/CoreUpdater/Commands/Update/CliUpdateObserver.php

* fix DB field piwik_log_visit.location_provider too small (#10003)

* fixes #9564 fix DB field piwik_log_visit.location_provider too small

* use new plugins updater API

* DB field piwik_log_visit.visit_total_actions too small (#10002)

* fixes #9565 DB field piwik_log_visit.visit_total_actions too small

* change type of some db columns that are too small

* fix tests (#10040)
Conflicts:
	plugins/CoreAdminHome/Menu.php
	plugins/Goals/Menu.php
	plugins/MobileMessaging/Menu.php
	plugins/SitesManager/Menu.php
	plugins/UsersManager/Menu.php
	tests/PHPUnit/System/expected/test_apiGetReportMetadata__API.getWidgetMetadata.xml

* fix more file permissions

* repair more file permissions

* repair more file permissions

* trying to make ui tests work again, the table was missing

* fix some encoding issues

* cross browser fixes and usability improvement

* move back the config icon, need to find a better solution later

* more cross browser fixes

* bugfixes

* fix ui tests

* fix encoding issue

* fix various issues with the ui tests when a test gets aborted

* also skip this visitor log test when aborted

* there were 3 css files that were loaded separately, merge them instead into one css

* forgot to add the actual manifest

* do not add manifest if custom logo is specified

* load font css files first as it was before merging them into big css

* fix link icon was not aligned anymore

* minor fixes

* setting it back to 4px

* in popovers the font variable was always ignored and a different font loaded

* forgot to update screenshots

* fix remaining tests

* this should fix an update error

* added 3 new widgets system check, system summary and plugin updates

* tweak new widgets content

* no page reload when changing date or segment

* in admin home show only enabled widgets

* refs #10295 use getMockBuilder instead of deprecated getMock

* fix some ui tests

* fix various bugs

* fix more tests

* fix ui tests

* add a space between loading image and loading message

* fix docs so they appear on developer.piwik.org

* improved documentation

* introduce new Widget::renderTemplate method for consistency with controllers

* remove no longer needed files

* testing system fonts

* fix strong was not really bold

* more useful system summary

* remove ubuntu font

* fix most tests and removed most em elements

* fix tests

* fix headline was very thin

* update submodule

* update submodules

* update submodule

* fix failing ui tests

* update submodules
@tsteur
Copy link
Member

tsteur commented Sep 13, 2016

fixed in #10003

@tsteur tsteur closed this as completed Sep 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants