Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make our test runner a bit smarter #6633

Open
4 of 8 tasks
tsteur opened this issue Nov 11, 2014 · 2 comments
Open
4 of 8 tasks

Make our test runner a bit smarter #6633

tsteur opened this issue Nov 11, 2014 · 2 comments
Labels
c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.

Comments

@tsteur
Copy link
Member

tsteur commented Nov 11, 2014

Currently, it is possible to execute tests by "testsuite", "group" and "file/dir":

./console tests:run --testsuite unit --file tests/PHPUnit/file.php groupName

The argument is currently interpreted as a group. I'd suggest to make this argument a bit smarter.

  • If argument is unit, integration, system, plugins or core interpret it as "--testsuite". This allows us to run a testsuite like ./console tests:run unit
  • If argument is a plugin name run all tests within this plugin eg ./console tests:run Insights. It would be nice if lower case was possible as well.
  • If arguments ends with .php search for a while having this name in tests folder eg ./console tests:run Widgets.php. To be defined if multiple files have this name. Can we specify multiple files in PHPUnit maybe? --> NO. Maybe instead let the user choose which one should be executed (eg enter number: [1] foo/bar.php, [2] bar/bar.php)
  • If arguments starts with tests/PHPUnit, plugins/ or if file_exists interpret it as file / directory
  • If nothing matches do not execute anything? Or shall we interpret it as --group? I'd prefer the first one as it could be possible that there is eg a group having the given name but you thought you were executing plugin tests or whatever. Eg if there is a typo etc.
  • It is to be defined how we handle multiple parameters. Eg would ./console tests:run unit Insights run all unit tests of the insights plugin? Could be nice
  • document internal change in Changelog or write in dev chat
  • remove no longer needed groups from tests (eg the ones having a plugin name, 'core', 'plugins', etc)

If someone specifies a --testsuite option and a argument named unit we should maybe interpret the argument as group? Same for --file etc.

This will allow us to remove some current groups. Executing groups is not so useful currently as they are not always set and very inconsistent. Eg not all tests in plugins have the group name set. This way we kinda gain "dynamical groups" that actually work.

As this adds some kind of magic and someone might still want to execute a group "unit" or "Insights" we should add another option --group so people can still avoid the magic and it will be more consistent with --testsuite and --file and with PHPUnit command line parameters.

@tsteur tsteur added the c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. label Nov 11, 2014
@mnapoli
Copy link
Contributor

mnapoli commented Nov 11, 2014

Related, but I didn't want to derail the topic: #6634

Except that, +1 with your suggestion, running tests is a little challenge everytime ;)

tsteur added a commit that referenced this issue Nov 11, 2014
…ards comptible as we will fallback to groups if nothing matches
@mattab
Copy link
Member

mattab commented Nov 11, 2014

All ideas are excellent - we will all appreciate!

document teal change in Changelog or write in dev chat

+1 for documenting in changelog maybe by pasting/reusing the help text of the command.

@mattab mattab added this to the Short term milestone Nov 11, 2014
tsteur added a commit that referenced this issue Nov 11, 2014
@mattab mattab added the Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. label Dec 1, 2014
@mattab mattab modified the milestone: Short term Apr 7, 2015
@mattab mattab added the c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. label Apr 8, 2015
@mattab mattab added this to the Mid term milestone Apr 8, 2015
@mattab mattab modified the milestones: Long term, Mid term Dec 5, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Development

No branches or pull requests

3 participants