New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LOAD DATA INFILE: Security issues in documentation and in general #7519
Comments
Hi @thomaszbz Thanks for the suggestion to improve the documentation about this feature. since you seem knowledgeable about the security implications, maybe you would be able to paste here your suggested updated text for this FAQ? it would help us make this change faster. cheers |
As @thomaszbz explain on https://issues.piwik.org/7519 and @mattab commented about improving the documentation these where the steps I did on my Piwik installation just in case someone of you want to update https://piwik.org/faq/troubleshooting/#faq_194:How do I get LOAD DATA INFILE to work on my server? Piwik processes huge amount of data and then stores this data in the database. For improved performance, Piwik tries to import a lot of data in the database at once using a Mysql feature called “LOAD DATA INFILE”. You can check whether your server supports this performance improvement in Administration > System Check menu. The system checks two commands: LOAD DATA INFILE and LOAD DATA LOCAL INFILE. It’s enough if either one of these work. In order to do this, connect to your webserver with SSH. You'll need ROOT ACCESS OR ROOT PERMISSIONS.
First check that the sql user that Piwik is using has the permission to import the files:
Check the “mysqld” process can access the file created in the tmp/assets directory in the piwik-installation: give the process mysqld read-access to files in path/to/piwik/tmp/assets/* and all parent folders.
check that the request isn’t blocked by apparmor or any other security-software when accessing this folder. If you are using Ubuntu, you may have to disable apparmor which prevents mysql from accessing files in path/to/piwik/tmp/* Edit the file /etc/apparmor.d/usr.sbin.mysqld and add the following path in the file:
Restart apparmor:
in your Mysql configuration (my.cnf) set the following options: [mysqld] local-infile, secure-file-priv = "" and [mysql] local-infile
Restart mysql:
Now you can have that nice "Huzzah! There are no problems with your Piwik setup. Give yourself a pat on the back." message on Piwik's System Check page. |
@fortinux Nice catch. It's just that ordinary web space hosters won't let anyone change an apparmor profile. Now that the documentation already suggests to do exactly that, it should be documented that this step requires root access. |
Hi, I just started out using Matomo and you should really rework this mentioned section in the help. EDIT : |
Nothing seems to have happend for a very long time. What is the status here? |
I found some issues in the FAQ, which somehow disable security features to get MySQL's LOAD DATA INFILE to work:
It should be
or even better
and not
which would affect all databases (not only the piwik database).
Why would mysqld need to execute files in tmp/assets? Instead, read access to files should do, execution is only needed for directories.
The local-infile would be system-wide, not only piwik-specific. In general, this opens up mysql. Plus, disabling apparmor is not an option for many users. Plus, why is writeable access proposed for mysqld (rw)?
Switching to MYSQLI is not an option for everyone. Playing around with open_basedir and safe_mode does not have something to do with PDO/MYSQLI as the reader might assume when reading the text. Plus, open_basedir and safe_mode are important security features which should NOT be disabled just to fix some errors. Users should be taught about that. Plus: What does file read access in the context of mysql have to do with PHP's open_basedir/safe_mode settings? These settings will only affect how apache/PHP process behaves. MySQL is a separate process which will not respect these settings at all!
Yes, that's the best option for most users who would otherwise insecure their servers.
Documentation needs to be reviewed in terms of security (I just red the part I was writing about). Unexperienced users will insecure their systems otherwise.
And for the code: In general, I would suggest to not use LOAD DATA INFILE in the context of mysqld at all. For PostgreSQL there exists a set of PDO-functions allowing bulk inserts in the context of the PHP-client via file or STDIN. Check out https://bugs.php.net/bug.php?id=63656 . Eventually, there is also an option for MySQL, but I did not search for that right now. That's the way to go: No file access via the database process -> no hassle. And it's fast as hell on PostgreSQL, still allowing transactions by reusing the context of client's connection. I even found a way to reuse doctrine connections for that on PostgreSQL. If you are interested in that, read my comment on the mentioned php.net site (I feel sorry that you used MySQL...).
For users who don't use BULK INSERTs, please check out if a MySQL transaction is used around the whole dataset which is to be inserted. Plus, check out if the MySQL Multi Row syntax insert is used. Both measurements could massively improve performance.
If inserts are done via ORM (doctrine, etc.), please make sure that you use MySQL transactions (doctrine supports that) around the mass INSERTs. Within the transaction, you can flush multiple times using flush(array). Make sure you destroy objects between flushes, and make sure that only few entity objects are instantiated at any time. Having lots of objects instantiated heavily slows down doctrine performance.
The text was updated successfully, but these errors were encountered: