Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New FAQ How do I configure Piwik to use Redis for better scalability? #6932

Closed
mattab opened this issue Jan 6, 2015 · 4 comments
Closed
Assignees
Labels
c: Usability For issues that let users achieve a defined goal more effectively or efficiently. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Jan 6, 2015

The goal of this issue is to create a new FAQ that answers "How do I configure Redis to store tracking requests, sessions and/or caches, for better scalability?".

Some ideas of the answer to the question:

  • list requirements as currently listed in QueuedTracking: redis 2.8+, phpredis
  • explain how to enable the features:
    • Tracker: use QueuedTracking plugin
    • Sessions: change config file?
    • Caches: change config file?
@mattab mattab added Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. c: Usability For issues that let users achieve a defined goal more effectively or efficiently. labels Jan 6, 2015
@mattab mattab added this to the Piwik 2.12.0 milestone Jan 6, 2015
@mattab mattab self-assigned this Mar 17, 2015
@mattab
Copy link
Member Author

mattab commented Mar 17, 2015

I'm removing some content from this other FAQ How do I configure Piwik Tracking for high reliability?

Content removed:


**How do I improve reliability of Piwik tracking as to not be dependant on the Mysql servers being up and running at all times?** 
Follow these steps:

*   On the Piwik tracking server(s), disable synchronous tracking in the config file: in config.ini.php, below the `[Tracker]` section, set: <pre>record_statistics=0</pre> 

*   At this stage, your web server(s) still receive the piwik.php requests but will directly return the 1*1 transparent gif beacon. These requests to piwik.php are still logged into your webserver access log files. They are not stored in your  Piwik database yet.
*   Regularly (eg. every hour or once a day), merge all these Tracking servers access log files into one file. This merged access log file will contain all requests to piwik.php including for each: the user agent, IP address, datetime, the piwik.php? URL, etc.
*   Sort this merged log file by date. 

*   To import the log file, in the Piwik server running the import, edit config.ini.php, and below the `[Tracker]` section, set: <pre>record_statistics=1</pre> 

*   Import this log file by running the [Log Analytics tool](http://piwik.org/log-analytics/). Use the option <pre>--replay-tracking</pre> This option will "replay" the piwik.php tracking requests into Piwik, effectively storing the visitors, visits, pageviews and goals data into your Mysql Database. Learn more about replaying logs [in this FAQ](http://piwik.org/faq/log-analytics-tool/faq_19221/).

*   Run the [archive pre-processing](http://piwik.org/docs/setup-auto-archiving/) after the visitors were successfully imported.

**Advantages:**

*   Tracker does not depend on the Mysql server being up. If Mysql is down, tracking requests are still collected in the server access log files. Piwik is more reliable.

**Challenges:**

*   If you have several frontends servers collecting the piwik.php tracking pixels requests, then make sure you merge all files prior to importing them with [Log Analytics](http://piwik.org/log-analytics/). 
*   After merging all server log files, sort them by date to ensure a given user's requests are always in the right order prior to import in Piwik. Sorting can be complex on very large log files.
*   Check your log rotation works well. Make sure not to replay the same log data multiple times. Piwik does not de-duplicate records and will import everything thrown at the [Tracking API](http://developer.piwik.org/api-reference/tracking-api).
*   Unless you import the log files often (eg. every minute), you will lose the [Real time](http://piwik.org/docs/real-time/) capability of Piwik. Typically, one would import logs every hour or every day.
*   Do not run the cron archiving process until after all visitors have been imported for the previous date. If you run the archive.php process too early (before visitors are imported for yesterday or past dates), then analytics data will be incomplete, and you would have [to re-process the report data](http://piwik.org/faq/how-to/faq_59/).

@mattab
Copy link
Member Author

mattab commented Mar 17, 2015

Created FAQ: How do I configure Piwik to use Redis for better scalability?

and edited this FAQ to link to the new one: How do I configure Piwik Tracking for high reliability?

(removed content is in above comment)

@mattab
Copy link
Member Author

mattab commented Mar 17, 2015

I chose not to document for now how to use Redis for Sessions and Caches. @tsteur maybe we should consider documenting this if it works already, thoughts?

@mattab mattab closed this as completed Mar 17, 2015
@tsteur
Copy link
Member

tsteur commented Mar 17, 2015

It does work already, I'm not sure what you need?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Usability For issues that let users achieve a defined goal more effectively or efficiently. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Development

No branches or pull requests

2 participants