Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reducing test failures due to non-bugs (eg, order of API output) #6983

Closed
4 tasks
diosmosis opened this issue Jan 12, 2015 · 4 comments
Closed
4 tasks

Reducing test failures due to non-bugs (eg, order of API output) #6983

diosmosis opened this issue Jan 12, 2015 · 4 comments
Labels
not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. RFC Indicates the issue is a request for comments where the author is looking for feedback. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.

Comments

@diosmosis
Copy link
Member

Refs #6544

Tests (mainly system) can fail for many reasons that aren't actual bugs. This creates pointless work that should be avoided. This ticket contains a list of possible changes that can remove these bugs:

  • Do not test API rendering in system tests.

    Avoids test failures that occur when formatting bugs are fixed or formats are changed.

    All tests should either compare against PHP arrays or DataTables themselves instead of rendering to XML/CSV/JSON and comparing against an expected file. The expected data can still be stored in files, we just shouldn't compare the rendered output.

  • When comparing API output, do not take column ordering into account.

    Avoids test failures that occur when API output reorders columns.

  • Remove all PHP test code and put into composer managed component

    Avoids test failures that occur when tests are run against older Piwik versions.

    Since the test related infrastructure will be managed by composer and not git, checking out a tag won't make new test code unavailable.

  • Instead of testing all available API methods, System tests should have individual test methods w/ descriptive names.

    Avoids painful debugging of random changes to API output. Having a test method name that describes what the test is testing for narrows the focus and makes it easier to find the reason for test failures.

@mattab Please give me your thoughts.

@diosmosis diosmosis added Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. RFC Indicates the issue is a request for comments where the author is looking for feedback. labels Jan 12, 2015
@tsteur
Copy link
Member

tsteur commented Jan 13, 2015

Order of API output might be important when using sort parameter, or in general the order is quite important or not? What is not important is the order within an "object/entity", meaning the properties. Maybe you meant that?

@mattab
Copy link
Member

mattab commented Jan 13, 2015

+1, in general the order of columns is important and should be tested in our system tests. better we make columns order fixed in core

@diosmosis
Copy link
Member Author

What is not important is the order within an "object/entity", meaning the properties. Maybe you meant that?

I meant column ordering as in DataTable row column ordering.

better we make columns order fixed in core

Do you think the ordering of columns should be tested for all API methods?

diosmosis pushed a commit that referenced this issue Jan 17, 2015
…order to avoid failures due to differences in API output w/ different Piwik versions.
@mattab
Copy link
Member

mattab commented Feb 19, 2015

We've made good progress at keeping the build greens!

Do you think the ordering of columns should be tested for all API methods?

in system tests I think it's ok to test for columns order

Closing the RFC, we can create follow up issue where needed

@mattab mattab closed this as completed Feb 19, 2015
@mattab mattab added the not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. label Feb 19, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. RFC Indicates the issue is a request for comments where the author is looking for feedback. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Development

No branches or pull requests

3 participants