Reducing test failures due to non-bugs (eg, order of API output) #6983
Labels
not-in-changelog
For issues or pull requests that should not be included in our release changelog on matomo.org.
RFC
Indicates the issue is a request for comments where the author is looking for feedback.
Task
Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Refs #6544
Tests (mainly system) can fail for many reasons that aren't actual bugs. This creates pointless work that should be avoided. This ticket contains a list of possible changes that can remove these bugs:
Do not test API rendering in system tests.
Avoids test failures that occur when formatting bugs are fixed or formats are changed.
All tests should either compare against PHP arrays or DataTables themselves instead of rendering to XML/CSV/JSON and comparing against an expected file. The expected data can still be stored in files, we just shouldn't compare the rendered output.
When comparing API output, do not take column ordering into account.
Avoids test failures that occur when API output reorders columns.
Remove all PHP test code and put into composer managed component
Avoids test failures that occur when tests are run against older Piwik versions.
Since the test related infrastructure will be managed by composer and not git, checking out a tag won't make new test code unavailable.
Instead of testing all available API methods, System tests should have individual test methods w/ descriptive names.
Avoids painful debugging of random changes to API output. Having a test method name that describes what the test is testing for narrows the focus and makes it easier to find the reason for test failures.
@mattab Please give me your thoughts.
The text was updated successfully, but these errors were encountered: