@szepeviktor opened this Pull Request on April 23rd 2018 Contributor

The purpose of this Pr is to measure long-running unit tests.

Was #12760

@szepeviktor commented on April 23rd 2018 Contributor

Results

SystemTests

You should really fix these slow tests (>500ms)...
 1. 215204ms to run Piwik\Tests\System\ArchiveCronTest:testArchivePhpCron
 2. 59605ms to run Piwik\Tests\System\TwoVisitorsTwoWebsitesDifferentDaysTest:testApi with data set <a href='/7'>#7</a>
 3. 54062ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/92'>#92</a>
 4. 51416ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/96'>#96</a>
 5. 51372ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/90'>#90</a>
 6. 50215ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/74'>#74</a>
 7. 48881ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/72'>#72</a>
 8. 48671ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/94'>#94</a>
 9. 47777ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/66'>#66</a>
 10. 47513ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/58'>#58</a>
...and there are 202 more above your threshold hidden from view
Time: 32.84 minutes, Memory: 368.00MB
@szepeviktor commented on April 23rd 2018 Contributor

Something very fundamental takes ~50 seconds per unit test

@mattab commented on May 8th 2018 Member

Thank you @szepeviktor for creating your first PR, that's really useful :+1: making our tests faster has become a top priority as the System tests often fail as they exceed 40min.

From this output it looks like possibly Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi should be tackled first as this causes probably 15+min or more in itself. (these tests check that each dimension can be used as segment, and have at least a visit matching in the fixture.)

You should really fix these slow tests (>500ms)...
 1. 215204ms to run Piwik\Tests\System\ArchiveCronTest:testArchivePhpCron
 2. 59605ms to run Piwik\Tests\System\TwoVisitorsTwoWebsitesDifferentDaysTest:testApi with data set <a href='/7'>#7</a>
 3. 54062ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/92'>#92</a>
 4. 51416ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/96'>#96</a>
 5. 51372ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/90'>#90</a>
 6. 50215ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/74'>#74</a>
 7. 48881ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/72'>#72</a>
 8. 48671ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/94'>#94</a>
 9. 47777ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/66'>#66</a>
 10. 47513ms to run Piwik\Tests\System\AutoSuggestAPITest:testAnotherApi with data set <a href='/58'>#58</a>

edit: @szepeviktor would it be maybe possible to display the top 100 slow tests instead of top 10?

@szepeviktor commented on May 8th 2018 Contributor

@mattab Do you have a 1 page summary on how to get from cloning matomo repo to running tests?
Maybe a Dockerfile??

@sgiehl commented on May 8th 2018 Member
@szepeviktor commented on May 8th 2018 Contributor

I think I need to create a Dockerfile to be able to breath

@szepeviktor commented on May 8th 2018 Contributor

Could you help me how piwik_tests db gets populated on Travis?

@szepeviktor commented on May 8th 2018 Contributor

Adding comments to the testing system (travis.yml, shell scripts) may help a lot.

@diosmosis commented on May 18th 2018 Member

@szepeviktor I use docker half of the time for development: https://github.com/diosmosis/matomo-docker-dev . It's definitely not official, so it may or may not work for you, I can't promise any real support and I will make changes to it as I need them.

Use docker-compose up to start it up (and all the other containers I use when developing). Install matomo by visiting localhost:3000, and run commands via the console script in the readme, eg, /path/to/matomo-docker-dev/console tests:run ./tests/PHPUnit/System/OneVisitorTwoVisitsTest.php (this will run the console php script in the matomo container).

As for how the test database is populated, it is created, setup & filled by the https://github.com/matomo-org/matomo/blob/3.x-dev/tests/PHPUnit/Framework/Fixture.php class and its descendants. (Fair warning, it sets up the entire matomo test environment, and so is a very large, hard to read class, and may make your eyes bleed.) Some test setup code is located elsewhere like in the https://github.com/matomo-org/matomo/blob/3.x-dev/tests/PHPUnit/Framework/TestingEnvironmentManipulator.php class.

@szepeviktor commented on May 18th 2018 Contributor

Thank you.

@szepeviktor commented on May 22nd 2018 Contributor

Thank you @diosmosis!

This PR is a rather like a debugging session.
It's goal is to find out what takes ~50 second per test.
Up to now I wasn't able to find out.

@diosmosis commented on May 24th 2018 Member

@szepeviktor Gotcha, will remove this from the current milestone then, feel free to create PRs when you find some optimizations :)

@mattab commented on June 26th 2018 Member

Thanks @szepeviktor for creating this PR. I've added a comment in the related issue here: https://github.com/matomo-org/matomo/issues/12691#issuecomment-400475879
I'll close this PR now, but feel free to re-open it as we could in theory add the speedtrap by default in Matomo.

This Pull Request was closed on June 26th 2018
Powered by GitHub Issue Mirror