Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add setting to disable LOAD DATA INFILE, for easy Mysql replication in AWS #4358

Closed
anonymous-matomo-user opened this issue Nov 28, 2013 · 12 comments
Assignees
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@anonymous-matomo-user
Copy link

I'm a fan of Piwik so far, but when we deployed Piwik to our live site yesterday, it broke our MySQL replication.

Here's the error I'm getting from the slave:

Error 'Access denied for user @ (using password: NO)' on query. Default database: 'piwik'. Query: 'LOAD DATA INFILE '/rdsdbdata/tmp/(numbers removed by me in case they're sensitive).data' REPLACE INTO TABLE piwik_archive_blob_2013_11 FIELDS TERMINATED BY '\t' ENCLOSED BY '"' ESCAPED BY '' LINES TERMINATED BY '\r\n' (idarchive, idsite, date1, date2, period, ts_archived, name, value)'

I thought it was weird that there was no username or hostname in the error, as well as the password being absent. Anyone have any thoughts on what this could be? We can't use our slave until this is figured out.

@anonymous-matomo-user
Copy link
Author

I forgot to mention this is all within Amazon's RDS, I didn't set up the replication myself. Let me know if you need more information from me.

@anonymous-matomo-user
Copy link
Author

I dug into this a little more and it looks like AWS just does not allow this:

https://forums.aws.amazon.com/thread.jspa?messageID=494201&#494201

Is there any way to turn off the Load Infile stuff from the config files? I realize this is going to slow everything down, but we apparently can't use Piwik given our setup if the load infile stuff is happening.

@mattab
Copy link
Member

mattab commented Dec 8, 2013

We could add a config setting to disable the use of LOAD DATA INFILE. Thanks for the suggestion!

@anonymous-matomo-user
Copy link
Author

Thanks for checking on on this. Any thoughts on how I can stop the LOAD DATA INFILE from happening in the mean time?

@mattab
Copy link
Member

mattab commented Dec 9, 2013

at line 66 of core/Db/BatchInsert
replace

    if (Db::get()->hasBulkLoader()) {

by

    if (false && Db::get()->hasBulkLoader()) {

it will skip the batch load.

@anonymous-matomo-user
Copy link
Author

This is on line 2358 of core/Piwik.php if you're in 1.12, I think. Thanks for your help!

@anonymous-matomo-user
Copy link
Author

Nope, this still isn't working. I've tried commenting out lines in the MySQL adapters that enable the load data infile options, but my replication is still being broken. Any other thoughts on how I can fix this in 1.12?

@mattab
Copy link
Member

mattab commented Dec 9, 2013

in the function createTableFromCSVFile

can you add the first line:

    debug_print_backtrace();

Then it will show you which code is calling this function. Maybe there's one more place to disable bulk loader. As far as I know there's only the one caller.

@anonymous-matomo-user
Copy link
Author

Hi Matt,

Thanks for your help on this. Once I killed the existing slave and started a new one, the problems went away. So the first edit you suggested was probably enough to fix the replication problem.

Thanks again!

@mattab
Copy link
Member

mattab commented Dec 26, 2013

Actually I'm not so keen to add a new setting for this. Maybe it would be better to catch the exception?

If anyone experiences the issue please comment and i'll reopen the ticket

@anonymous-matomo-user
Copy link
Author

Currently, Piwik set up to use LOAD INFILE will break Mysql replication in AWS RDS. Are you saying you won't fix that issue or that you won't add a config option? All I care about is that it doesn't break our replication in future versions.

@mattab
Copy link
Member

mattab commented Dec 27, 2013

In 1894442: Adding config setting to explicitely disable LOAD DATA INFILE to prevent error when doing mysql replication on AWS Fixes #4358

@anonymous-matomo-user anonymous-matomo-user added this to the 2.0.3 - Piwik 2.0.3 milestone Jul 8, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

2 participants