Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mysqli statement execute error : Incorrect string #19069

Closed
dr0i opened this issue Apr 8, 2022 · 7 comments
Closed

Mysqli statement execute error : Incorrect string #19069

dr0i opened this issue Apr 8, 2022 · 7 comments
Labels
answered For when a question was asked and we referred to forum or answered it.

Comments

@dr0i
Copy link

dr0i commented Apr 8, 2022

Importing logs using matomo/misc/log-analytics/import_logs.pysometimes breaks. Webserver's error log reveals a [php7:notice] [...] Mysqli statement execute error : Incorrect string [...].

Searched issues here with similar problems, like e.g. #11136.

Matomo: 4.8.0
Mysql: Ver 8.0.25 for Linux on x86_64 (made sure to have innodb_file_per_table=ON and using utf8mb4 as described in #9785)

Found a solution at https://forum.matomo.org/t/error-in-piwik-tracker-error-query-sqlstate-hy000-general-error-1366-incorrect-string-value-xd0--xd0-xb2-xd1-x82-for-column-name-at-row-1-in-query-insert-into-piwik-log-action/21800.

If $name = iconv('UTF-8', 'ASCII//TRANSLIT', $name); is put before

$db->query($sql, array($name, $name, $type, $urlPrefix));
the import runs successfully.

Don't have clue about php re impact of that iconv considering performance or if it breaks showing all/some UTF-8 URLs in matomo - but this may give a good hint what to do meanwhile fixes my problem (until the next matomo update).

dr0i added a commit to dr0i/matomo that referenced this issue Apr 8, 2022
Some UTF8 strings may stop the import. Webserver's error-log reveals:
[php7:notice] [...] Mysqli statement execute error : Incorrect string [...].

Converting the string to ASCII before importing fixes the issue so that
importing the logs runs successfully.
@MatomoForumNotifications

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/problem-with-url-context/44978/6

@dr0i
Copy link
Author

dr0i commented Apr 12, 2022

A bit of background:
This problem came up (incidentally?) just after an update of my matomo 3.* installation (which doesn't ran that smooth for I had to do alter table matomo_log_link_visit_action add pageview_position MEDIUMINT UNSIGNED DEFAULT NULL; manually in the mysqldb.)
Also, I updated mysqld a year or so before from 5.x to 8.x .
Don't know if one/all of these changes require a rewriting of table metadata. See also the discussion at the corresponding PR.

@dr0i
Copy link
Author

dr0i commented Apr 12, 2022

May be linked to #15711 ? I would have to do ALTER TABLE matomo_log_visit CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; (which I didn't yet - also in the referenced issue this hasn't solved the problem).

@dr0i
Copy link
Author

dr0i commented Apr 12, 2022

Thx again @sgiehl for the pointer - I think I can go on my own now, since SHOW TABLE STATUS FROM piwik; reveals that
all tables use utf8_unicode_ci as collation( which is pointed out to be problematic in #9785 (comment)). So I would have to change the collation of (at least all log* tables) to utf8mb4_unicode_ci

@dr0i
Copy link
Author

dr0i commented Apr 12, 2022

This should to the trick:

mysql > use piwik;
mysql> ALTER TABLE matomo_log_link_visit_action CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ;

This may take a long time!
(having several 100GBs of data and count of rows for matomo_log_link_visit_action is > 10^9 I expect the time needed for the conversion ~ 2 days. It just seems to be a matter of CPU time (and IO throughput) though - memory is alright, also file space seems to be fine (meaning not doubling the db space temporarily . Mysql seems to be clever enough to remove converted COLUMNS (batch wise or so)).

Note that it is not enough to only change the DEFAULTS like with:

ALTER TABLE matomo_log_link_visit_action DEFAULT CHARACTER SET utf8mb4 ;

(hoped for this to only take effect on newly inserted data, but that's not the way it is proposed to work (tested it)).
Did this none the less since a 'SHOW TABLE STATUS FROM piwik;' will (only then) show the use of utf8mb4_unicode_ci.

Note also that you may need to update all other matomo_log* tables, but I guess this one table is enough.
Will come back here to report if this works out ok.

@sgiehl
Copy link
Member

sgiehl commented Apr 12, 2022

@dr0i There is actually a command to perform all the required (database) changes to do this: ./console core:convert-to-utf8mb4
See https://matomo.org/faq/how-to-update/how-to-convert-the-database-to-utf8mb4-charset/

@dr0i
Copy link
Author

dr0i commented Apr 14, 2022

Mysql ended up with :

ERROR 1034 (HY000): Incorrect key file for table 'matomo_log_link_visit_action'; try to repair it

Didn't thought of having enough space left on device for temporary mysql files:

mysql> SELECT * FROM performance_schema.error_log\G
...
DATA: Error number 28 means 'No space left on device'
...

My comment was obviously wrong:

file space seems to be fine (meaning not doubling the db space temporarily . Mysql seems to be clever enough to remove converted COLUMNS (batch wise or so)).

because I only had a look at the mount where the matomo db resides.

I close this issue here because this is a new issue a myslq issue due my misconfiguration having nothing to do with matomo.
Thx for your help again!

@dr0i dr0i closed this as completed Apr 14, 2022
@sgiehl sgiehl added the answered For when a question was asked and we referred to forum or answered it. label Apr 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants