To reproduce this issue have Form Analytics installed and track data.

Then configure log data deletion as defined in https://matomo.org/faq/troubleshooting/faq_42/ . At least in the beginning you will want to configure a high minimum amount of days like 5000 in order to not delete all your tracked log data.

When the task runs to delete old log data, which you can do eg using ./console core:run-scheduled-tasks --force "Piwik\Plugins\PrivacyManager\Tasks.deleteLogData"

You will then notice in LogDeleter::deleteVisits() while debugging the code that it will try to delete the data from the log_form tables, but not from the tables that log_form references to which is eg log_form_page and log_form_field. Other custom tracker plugins may have similar problems.

It's only deleting data from tables that have an idvisit column but the other form tables don't have this. instead they reference to log_form using the idlogform table.

on the contrary in https://github.com/matomo-org/matomo/blob/4.4.1/plugins/PrivacyManager/Model/DataSubjects.php#L72-L77 where we have the logic for deleting individual visit data it does delete data from these tables.

Not sure if any of the logic can be reused as at least the DataSubjects API doesn't have any limits on how many visits would be deleted at once and there may be a risk that these log tables would be locked for too long possibly causing downtime. But maybe it can be reused as long as not too many visits are being sent to the method (which may be already the case).

We need to make sure that we delete the correct data from all the needed tables when log purging runs. At the same time we need to make sure these tables won't be locked for too long as otherwise there will be server issues while trying to track data into these tables.

At least in the beginning you will want to configure a high minimum amount of days like 5000 in order to not delete all your tracked log data.

Does that mean to reproduce I will need to create data going back more than 5000 days? Would the VisitorGenerator be the tool to use for this or another way I would need to automate or do manually?

I currently have no form data logs and would create a form, submit it and see what it logs. Then I would try to repeat the request with some different parameters to figure out how it all works.

I debugged the script like so:

php -dxdebug.start_with_request=yes ./console core:run-scheduled-tasks --force "Piwik\Plugins\PrivacyManager\Tasks.deleteLogData"

but it didn't hit LogDeleter::deleteVisits().

Might this be part of the problem or does it hit LogDeleter::deleteVisits() when there are logs only?

Answering partly my own question:

seems to call this method only if there are logs.

The question that remains though what is the most effective and fastest way to generate the logs to be deleted?

The question that remains though what is the most effective and fastest way to generate the logs to be deleted?

I think the Visitor Generator which purpose is to generate test logs? it's pretty slow but it should work fine..

Thanks @mattab .

For future reference, I use this command:

./console development:disable; ./console visitorgenerator:generate-visits --start-date 2007-12-20 --idsite 1 --days 6 to overlap with 5000 days ago and to avoid it taking too long.

@geekdenz you could also set a different number of days like 300 or so. You just want to probably not delete any recent data by accident.