Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skip CLIMulti check for already running commands in tests #12764

Merged
merged 1 commit into from Apr 23, 2018
Merged

Skip CLIMulti check for already running commands in tests #12764

merged 1 commit into from Apr 23, 2018

Conversation

sgiehl
Copy link
Member

@sgiehl sgiehl commented Apr 23, 2018

Since merging #12702 the ArchiveCronTest started failing.
To temporary fix the tests this will disable the check for already running command in tests.

@tsteur could you please have a look why this made the test fail? I had enabled the output of archiving to check for an error.

Guess the problem is skipping archiving for period 'month' because processing the period 'week' is already in progress..

-------------------------------------------------------
Using this 'archive.php' script is no longer recommended.
Please use '/path/to/php /home/travis/build/matomo-org/matomo/tests/PHPUnit/proxy/../../..//console core:archive --url=http://localhost/tests/PHPUnit/proxy/index.php' instead.
To get help use '/path/to/php /home/travis/build/matomo-org/matomo/tests/PHPUnit/proxy/../../..//console core:archive --help'
See also: https://matomo.org/docs/setup-auto-archiving/
If you cannot use the console because it requires CLI
try 'php archive.php --url=http://your.piwik/path'
-------------------------------------------------------
INFO [2018-04-23 12:23:15] 8421  ---------------------------
INFO [2018-04-23 12:23:15] 8421  INIT
INFO [2018-04-23 12:23:15] 8421  Running Matomo 3.5.0-b1 as Super User
INFO [2018-04-23 12:23:15] 8421  ---------------------------
INFO [2018-04-23 12:23:15] 8421  NOTES
INFO [2018-04-23 12:23:15] 8421  - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Matomo UI > Settings > General Settings.
INFO [2018-04-23 12:23:15] 8421    See the doc at: https://matomo.org/docs/setup-auto-archiving/
INFO [2018-04-23 12:23:15] 8421  - Reports for today will be processed at most every 150 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO [2018-04-23 12:23:15] 8421  - Reports for the current week/month/year will be requested at most every 3600 seconds.
INFO [2018-04-23 12:23:15] 8421  - Will invalidate archived reports for 2012-08-09 for following websites ids: 1
INFO [2018-04-23 12:23:15] 8421  - Will invalidate archived reports for 2012-08-10 for following websites ids: 1
INFO [2018-04-23 12:23:15] 8421  - Will invalidate archived reports for 2012-08-11 for following websites ids: 1
INFO [2018-04-23 12:23:15] 8421  - Will invalidate archived reports for 2012-08-15 for following websites ids: 1,2,3
INFO [2018-04-23 12:23:15] 8421  - Will invalidate archived reports for 2012-08-19 for following websites ids: 1
INFO [2018-04-23 12:23:15] 8421  - Will invalidate archived reports for 2012-09-30 for following websites ids: 1
INFO [2018-04-23 12:23:15] 8421  - Will invalidate archived reports for 2014-03-12 for following websites ids: 1,3
INFO [2018-04-23 12:23:15] 8421  - Will invalidate archived reports for 2014-03-13 for following websites ids: 1,3
INFO [2018-04-23 12:23:15] 8421  - Will process 3 other websites because some old data reports have been invalidated (eg. using the Log Import script or the InvalidateReports plugin) , IDs: 1, 2, 3
INFO [2018-04-23 12:23:15] 8421  ---------------------------
INFO [2018-04-23 12:23:15] 8421  START
INFO [2018-04-23 12:23:15] 8421  Starting Matomo reports archiving...
INFO [2018-04-23 12:23:15] 8421  Old report was invalidated for website id 1
INFO [2018-04-23 12:23:15] 8421  Will pre-process for website id = 1, period = day, date = last52
INFO [2018-04-23 12:23:15] 8421  - pre-processing all visits
INFO [2018-04-23 12:23:17] 8421  - pre-processing segment 1/3 browserCode==IE
INFO [2018-04-23 12:23:17] 8421  - pre-processing segment 2/3 visitCount<=5;visitorType!=non-existing-type;daysSinceFirstVisit<=50
INFO [2018-04-23 12:23:17] 8421  - pre-processing segment 3/3 visitCount<=5;visitorType!=re%2C%3Btest%20is%20encoded;daysSinceFirstVisit<=50
INFO [2018-04-23 12:23:22] 8421  Archived website id = 1, period = day, 3 segments, 0 visits in last 52 days, 0 visits today, Time elapsed: 7.533s
INFO [2018-04-23 12:23:22] 8421  Will pre-process for website id = 1, period = week, date = last260
INFO [2018-04-23 12:23:22] 8421  - pre-processing all visits
INFO [2018-04-23 12:23:23] 8421  - pre-processing segment 1/3 browserCode==IE
INFO [2018-04-23 12:23:23] 8421  - pre-processing segment 2/3 visitCount<=5;visitorType!=non-existing-type;daysSinceFirstVisit<=50
INFO [2018-04-23 12:25:00] 8421  - pre-processing segment 3/3 visitCount<=5;visitorType!=re%2C%3Btest%20is%20encoded;daysSinceFirstVisit<=50
INFO [2018-04-23 12:25:51] 8421  Archived website id = 1, period = week, 3 segments, 12 visits in last 260 weeks, 0 visits this week, Time elapsed: 148.048s
INFO [2018-04-23 12:25:51] 8421  Will pre-process for website id = 1, period = month, date = last52
INFO [2018-04-23 12:25:51] 8421  - pre-processing all visits
INFO [2018-04-23 12:25:51] 8421  - pre-processing segment 1/3 browserCode==IE
INFO [2018-04-23 12:25:51] 8421  - pre-processing segment 2/3 visitCount<=5;visitorType!=non-existing-type;daysSinceFirstVisit<=50
INFO [2018-04-23 12:26:02] 8421  - pre-processing segment 3/3 visitCount<=5;visitorType!=re%2C%3Btest%20is%20encoded;daysSinceFirstVisit<=50
INFO [2018-04-23 12:26:09] 8421  Archived website id = 1, period = month, 3 segments, 12 visits in last 52 months, 0 visits this month, Time elapsed: 18.404s
INFO [2018-04-23 12:26:09] 8421  Will pre-process for website id = 1, period = year, date = last7
INFO [2018-04-23 12:26:09] 8421  - pre-processing all visits
INFO [2018-04-23 12:26:09] 8421  - pre-processing segment 1/3 browserCode==IE
INFO [2018-04-23 12:26:09] 8421  - pre-processing segment 2/3 visitCount<=5;visitorType!=non-existing-type;daysSinceFirstVisit<=50
INFO [2018-04-23 12:26:36] 8421  - pre-processing segment 3/3 visitCount<=5;visitorType!=re%2C%3Btest%20is%20encoded;daysSinceFirstVisit<=50
INFO [2018-04-23 12:26:49] 8421  Archived website id = 1, period = year, 3 segments, 46 visits in last 7 years, 0 visits this year, Time elapsed: 40.267s
INFO [2018-04-23 12:26:49] 8421  Archived website id = 1, 16 API requests, Time elapsed: 214.260s [1/3 done]
INFO [2018-04-23 12:26:49] 8421  Old report was invalidated for website id 2
INFO [2018-04-23 12:26:49] 8421  Will pre-process for website id = 2, period = day, date = last52
INFO [2018-04-23 12:26:49] 8421  - pre-processing all visits
INFO [2018-04-23 12:26:51] 8421  Archived website id = 2, period = day, 0 segments, 0 visits in last 52 days, 0 visits today, Time elapsed: 2.176s
INFO [2018-04-23 12:26:51] 8421  Will pre-process for website id = 2, period = week, date = last260
INFO [2018-04-23 12:26:51] 8421  - pre-processing all visits
INFO [2018-04-23 12:27:29] 8421  Archived website id = 2, period = week, 0 segments, 0 visits in last 260 weeks, 0 visits this week, Time elapsed: 37.805s
INFO [2018-04-23 12:27:29] 8421  - skipping archiving for period 'month' because processing the period 'week' is already in progress.
INFO [2018-04-23 12:27:29] 8421  Archived website id = 2, 2 API requests, Time elapsed: 40.004s [2/3 done]
INFO [2018-04-23 12:27:29] 8421  Old report was invalidated for website id 3
INFO [2018-04-23 12:27:29] 8421  Will pre-process for website id = 3, period = day, date = last52
INFO [2018-04-23 12:27:29] 8421  - pre-processing all visits
INFO [2018-04-23 12:27:31] 8421  Archived website id = 3, period = day, 0 segments, 0 visits in last 52 days, 0 visits today, Time elapsed: 2.161s
INFO [2018-04-23 12:27:31] 8421  Will pre-process for website id = 3, period = week, date = last260
INFO [2018-04-23 12:27:31] 8421  - pre-processing all visits
INFO [2018-04-23 12:28:10] 8421  Archived website id = 3, period = week, 0 segments, 12 visits in last 260 weeks, 0 visits this week, Time elapsed: 38.812s
INFO [2018-04-23 12:28:10] 8421  - skipping archiving for period 'month' because processing the period 'week' is already in progress.
INFO [2018-04-23 12:28:10] 8421  Archived website id = 3, 2 API requests, Time elapsed: 41.000s [3/3 done]
INFO [2018-04-23 12:28:10] 8421  Done archiving!
INFO [2018-04-23 12:28:10] 8421  ---------------------------
INFO [2018-04-23 12:28:10] 8421  SUMMARY
INFO [2018-04-23 12:28:10] 8421  Total visits for today across archived websites: 0
INFO [2018-04-23 12:28:10] 8421  Archived today's reports for 3 websites
INFO [2018-04-23 12:28:10] 8421  Archived week/month/year for 3 websites
INFO [2018-04-23 12:28:10] 8421  Skipped 0 websites
INFO [2018-04-23 12:28:10] 8421  - 0 skipped because no new visit since the last script execution
INFO [2018-04-23 12:28:10] 8421  - 0 skipped because existing daily reports are less than 150 seconds old
INFO [2018-04-23 12:28:10] 8421  - 0 skipped because existing week/month/year periods reports are less than 3600 seconds old
INFO [2018-04-23 12:28:10] 8421  Total API requests: 20
INFO [2018-04-23 12:28:10] 8421  done: 3/3 100%, 0 vtoday, 3 wtoday, 3 wperiods, 20 req, 295473 ms, no error
INFO [2018-04-23 12:28:10] 8421  Time elapsed: 295.473s
INFO [2018-04-23 12:28:10] 8421  ---------------------------
INFO [2018-04-23 12:28:10] 8421  SCHEDULED TASKS
INFO [2018-04-23 12:28:10] 8421  Starting Scheduled tasks...
INFO [2018-04-23 12:28:10] 8421  done
INFO [2018-04-23 12:28:10] 8421  ---------------------------

@sgiehl sgiehl added c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. labels Apr 23, 2018
@sgiehl sgiehl merged commit 880dbef into 3.x-dev Apr 23, 2018
@sgiehl sgiehl deleted the skipcheck branch April 23, 2018 13:17
@mattab
Copy link
Member

mattab commented Apr 23, 2018

This is one of those really surprising things, really cannot explain this bug so far... especially that it is random and on my dev box it doesn't happen

@tsteur
Copy link
Member

tsteur commented Apr 23, 2018

I tried to run the tests 10 times and they always fail on my local machine even after increasing memory limit in CLI + web to 1GB...

Only thing I can think of is to disable this feature in tests...

INFO [2018-04-23 22:00:37] 19432 SUMMARY OF ERRORS
INFO [2018-04-23 22:00:37] 19432 Error: Got invalid response from API request: ?module=API&method=API.get&idSite=1&period=day&date=last52&format=php&trigger=archivephp. The response was empty. This usually means a server error. A solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. For more information and the error message please check in your PHP CLI error log file. As this core:archive command triggers PHP processes over the CLI, you can find where PHP CLI logs are stored by running this command: php -i | grep error_log
INFO [2018-04-23 22:00:37] 19432 Error: Empty or invalid response '' for website id 1, Time elapsed: 8.404s, skipping
INFO [2018-04-23 22:00:37] 19432 Error: Got invalid response from API request: ?module=API&method=API.get&idSite=2&period=day&date=last52&format=php&trigger=archivephp. The response was empty. This usually means a server error. A solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. For more information and the error message please check in your PHP CLI error log file. As this core:archive command triggers PHP processes over the CLI, you can find where PHP CLI logs are stored by running this command: php -i | grep error_log
INFO [2018-04-23 22:00:37] 19432 Error: Empty or invalid response '' for website id 2, Time elapsed: 9.140s, skipping
INFO [2018-04-23 22:00:37] 19432 Error: Got invalid response from API request: ?module=API&method=API.get&idSite=3&period=day&date=last52&format=php&trigger=archivephp. The response was empty. This usually means a server error. A solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. For more information and the error message please check in your PHP CLI error log file. As this core:archive command triggers PHP processes over the CLI, you can find where PHP CLI logs are stored by running this command: php -i | grep error_log
INFO [2018-04-23 22:00:37] 19432 Error: Empty or invalid response '' for website id 3, Time elapsed: 8.389s, skipping
ERROR [2018-04-23 22:00:37] 19432 6 total errors during this script execution, please investigate and try and fix these errors.

InfinityVoid pushed a commit to InfinityVoid/matomo that referenced this pull request Oct 11, 2018
@sgiehl @InfinityVoid
Skip CLIMulti check for already running commands in tests (matomo-org…
603bcfe 
…#12764)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@sgiehl @mattab @tsteur