Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add information about the memory consumption #11816

Closed
wants to merge 3 commits into from

Conversation

mikeSimonson
Copy link

of the archive cronjob.
This can be helpfull when you approach the memory limit.

of the archive cronjob.
This can be helpfull when you approach the memory limit.
@mikeSimonson
Copy link
Author

The build is failing for reason that are unrelated to this PR.
All the tests that this PR broke are fixed.

@@ -413,6 +414,7 @@ public function run()
Piwik::postEvent('CronArchive.archiveSingleSite.start', array($idSite));

$completed = $this->archiveSingleSite($idSite, $requestsBefore);
$this->logger->info("Mermory usage: " . $this->getHumanReadableBytesSize(memory_get_usage()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo should be Memory

@@ -460,6 +462,12 @@ public function run()
);

$this->logger->info($timer->__toString());
$this->logger->info("Mermory peak usage: " . $this->getHumanReadableBytesSize(memory_get_peak_usage()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo should be Memory

$this->logger->info("Mermory peak usage: " . $this->getHumanReadableBytesSize(memory_get_peak_usage()));
}

private function getHumanReadableBytesSize($memoryInBytes, $precision = 2) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please format the code consistently with our code / PSR notation, ie add line breaks and spacing

@mattab
Copy link
Member

mattab commented Aug 3, 2017

Hi @mikeSimonson
Thanks for the PR. Started reviewing it but then realised that this script is not actually the script which runs the archiving tasks. They are either

  1. Run in a separate thread using our console command climulti:request (if supported)
  2. Or they are run by calling the Piwik API over HTTP

In both cases the memory of the archiving script will not raise much as the actual memory hungry processes are in separate threads.

Therefore I would say this PR is not needed. Could you clarify whether it was helpful for you at some point, and why?

This is a little bit crazy but apparently the only way I can answer
your questions on my PR are via this commit message.

Apparently the owner of the piwik repo restricted comment to only
prior contributor. You would think that this would allow you to
comment your own PR but apparently not.

So abouts yours questions:

In my case the archiving tasks is run by a cron job via
```./console core:archive```, because of that it's quite
convenient to see in the output if it's the memory
constraints that made the cronjob fail.

For the conformance to the code format I hope that this commit fix it.
Don't hesitate to point me other formatting issue if there are others.

Thanks
@mattab
Copy link
Member

mattab commented Aug 16, 2017

@mikeSimonson Please see my last comment for questions. For now closing the PR but happy to re-open if it's actually useful 👍

@mattab mattab closed this Aug 16, 2017
@mikeSimonson
Copy link
Author

@mattab Sorry I thought I had already replied.

That commands is also directly called by cronjobs ( from the docs ).
It's in this case that it's useful for me as the log will give me more info as to why it might have failed.

@mattab
Copy link
Member

mattab commented Aug 27, 2017

@mikeSimonson would you mind sending here an example of the archive log output which shows the memory increase/decrease or another interesting use case? Thanks

@mikeSimonson
Copy link
Author

INFO [2017-08-27 22:40:19] Archived website id = 3, period = year, 0 segments, 1495404 visits in last 2 years, 1495404 visits this year, Time elapsed: 45.997s
INFO [2017-08-27 22:40:19] Archived website id = 3, 4 API requests, Time elapsed: 35.564s [3/9 done]
INFO [2017-08-27 22:40:19] Mermory usage: 123.55MB
INFO [2017-08-27 22:40:19] - no new tracking data for website id 4 since 2017-08-27 22:35:11 UTC (since the last successful archiving)
INFO [2017-08-27 22:40:19] Skipped website id 4 as archiving is not needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants