New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
configurable Thread-Queue size #277
configurable Thread-Queue size #277
Conversation
added debug for PHP calls / Recorder usage
Reason for this change is, that the importer slows markably down if one recorder gets far more hits, so the PHP is slower than the parser, and then the Importer get's stuck in Queue.put (=>threading.acquire). ( improved our import speed from almost 1h to 12 minutes ). |
Thanks for PR! What value do you set --queue-size to, to go from 1h to 12min? |
we had set it to 12, but it did not use more than 5 in our case. To be precise: we found in python cProfile, that it spend really much time in waiting for add_hits()->Queue.put .. causing the cpython process hung, so we just enable the main thread to continue working when making Thread-Queue configurable if you like, i can deliver precise performance data next week. |
ok thanks for explanation. Do you understand why the importer gets stuck in Queue.put? Also, do you think it would make sense to increase from default 2 to for example 5 for all users? |
yes, and thats why i am uncertain, if this is a specific problem to us. We have several very busy sites, but some IPs belong to customers, and they do alot traffic from one IP. The effect is: we have the more or less common traffic, which fills recorders evenly, and we have customers, which fill recorder for example recorder 6. if recorder 6 is full, the importer waits. if the customer ends his session, we have random traffic, until another customer fills for example recorder 11, where it gets stuck again. so, i don't know how other big sites are equipped, how they do get along, but as the PHP Part is the slow part, keeping as much php-fpm childs as busy as possible is a big gain. 5 suits our needs now, but i dont dare to say, if thats true for ech other user - and probably its not necessary for most small sites. so, the only way to solve that is to modify the backlog, which is currently hardcoded (and changed by this PR). |
Ok I get it now! For sure that it is a problem that many other users will experience. Because it's common to have one or several IPs with hundreds of requests... |
How many Maybe we could set |
i found that recorders works best for me if it's * ( 1.5 - 2 ) |
Nice, always better to not add new code that makes harder to understand 👍 But, I still think, it could be a good idea, to process the Queue size based on numbers of recorders (automatically) ;) |
configurable Thread-Queue size
added debug for PHP calls / Recorder usage