Thanks for creating the issue @adrianlambertz
Maybe we should avoid updating the usage timestamp for token_auths when doing log imports

Is there a quick fix to disable the timestamp update while using the python log import? Because we are heavily relying on this script - we import logs from about 100 sites every night.

Because of this issue the imported statistics arent complete or missing.

You can change the code so the update isn't done here:

Perfect thanks, temporarily commenting out the updateTokenAuthTable function seems to work. I'll try to report tomorrow morning after the import of all sites.

Hi @adrianlambertz

Have you managed to solve this problem? If yes could please share your change?

Thanks!

Hi @adrianlambertz

Have you managed to solve this problem? If yes could please share your change?

Thanks!

See this pull request from @sqiehl

@adrianlambertz thanks, I have updated code according to @sgiehl pull request, but problem still exists, very poor import speed.

Hi @L3on1d are there also other queries causing these errors maybe?

Is the Provider plugin enabled maybe?

Provider plugin is disabled. I'm just started another log import, if these errors will appear again, I will update you. Anyway speed is very slow.

@tsteur yes you are right, this error:

"PHP message: Error in Matomo (tracker): Error query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction In query: UPDATE mtm_log_visit SET profilable = ?, visitor_seconds_since_order = ?, visit_exit_idaction_url = ?, visit_total_actions = visit_total_actions + 1 , visit_total_interactions = visit_total_interactions + 1 , visit_total_time = ? WHERE idsite = ? AND idvisit = ? Parameters: array ( 0 => 0, 1 => NULL, 2 => 2588680546, 3 => 589, 4 => 6, 5 => '3037957010', )" while reading response header from upstream

not related to log import,

but main problem that I have pretty big log volume, and with such big drop in import speed I'm unable to import logs.

What's your log import command parameters?

In config/config.ini.php you could also try setting this as a test:

[Tracker]
bulk_requests_use_transaction = 0

Might not help but worth a try. Otherwise you could also try and play around with different settings to see if that helps.

At the moment, I use this command:

python3.9 /path/to/misc/log-analytics/import_logs.py --url=https://mymatomourl --idsite=6 --recorders=8 --enable-bots --recorder-max-payload-size=400 /path/to/logs

UPD

I have tried import with bulk_requests_use_transaction = 0 - same results.

Thank you.

I found another thing which can be directly related to speed problems, according to monitoring, after upgrade to Matomo 4 log import start consuming more CPU and load average increased x3 times. :(

Before upgrade:

After upgrade:

Perfect thanks, temporarily commenting out the updateTokenAuthTable function seems to work. I'll try to report tomorrow morning after the import of all sites.

This night our import worked smoothly like a charm. No SQL Timeouts and all statistics/logs are correctly imported. Thank you @sgiehl

EDIT:
But it seems that it was much slower than before the update. This night it took 02:18 hours to import (but i have to say: with 4 recorders instead of 8).
The last import on version 3 took only 23 Minutes with 8 recorders.

I changed the amount of recorders to 8 today and i'll report tomorrow morning.

Yes, tonight it took 1:38 hour to complete the import with 8 recorders. It's much slower than before the update.

@adrianlambertz have you checked your server load during import?

@adrianlambertz have you checked your server load during import?

Yes, nothing unusual. Same load as before.