Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run Screenshot tests on PHP 5.3 #6414

Closed
mattab opened this issue Oct 10, 2014 · 7 comments
Closed

Run Screenshot tests on PHP 5.3 #6414

mattab opened this issue Oct 10, 2014 · 7 comments
Assignees
@tsteur
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone
2.9.0

Comments

@mattab
Copy link
Member

mattab commented Oct 10, 2014

The goal of this issue is to run the Screenshot test suites of 200 screenshots tests, so that they run on PHP 5.3 and 5.5.

Recently we had an issue on PHP 5.3 that could have been discovered by a UI test if they were running on 5.3. Currently they only run on 5.5 though... so we missed it.

Proposed tasks

  • Modify travis.yml so that UI tests run on 5.3 and 5.5 (possibly also 5.4 ?)
  • Modify core + pro Plugins UI tests travis so they run on 5.3 and 5.5.
@mattab mattab added Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. labels Oct 10, 2014
@mattab mattab added this to the Short term milestone Oct 10, 2014
@sgiehl
Copy link
Member

sgiehl commented Oct 10, 2014

+1 for that. Maybe we could run the tests on other php versions some kind of time triggered (e.g. every 3-6 hours), so they won't run after each commit. Guess that should be enough to identify such errors.

@mattab mattab changed the title Run Screenshot tests on all PHP versions Run Screenshot tests on PHP 5.3 Oct 12, 2014
@tsteur
Copy link
Member

tsteur commented Oct 12, 2014

I think the UI tests are there to test UI stuff (JS, CSS, HTML) so testing many different PHP versions shouldn't matter so much. Ideally, all the other stuff is covered by unit / integration / system tests. Unfortunately, this is not 100% the case eg for some Controller code. Would personally run them only on PHP 5.3. If needed also on the latest meaning 5.6. Nothing else.

@mattab
Copy link
Member Author

mattab commented Oct 17, 2014

In an effort to keep jobs small, maybe we only run UI tests on 5.3.3 then!

@mnapoli
Copy link
Contributor

mnapoli commented Oct 19, 2014

maybe we only run UI tests on 5.3.3 then

+1

@tsteur
Copy link
Member

tsteur commented Nov 10, 2014

Shall we still run them on 5.3.3? Currently they are running on 5.5. Not sure what the latest thoughts on this are...

@mattab
Copy link
Member Author

mattab commented Nov 10, 2014

👍 to run on lowest PHP version 5.3.3!

@tsteur
Copy link
Member

tsteur commented Nov 10, 2014

Just FYI: It seems to make the UI tests slower from 18-20 minutes to 25-28 minutes but think it is still good to use PHP 5.3.3 to run the tests. It is worth it as we will catch more errors this way.

@tsteur tsteur closed this as completed Nov 10, 2014
@tsteur tsteur self-assigned this Nov 10, 2014
@tsteur tsteur modified the milestones: Piwik 2.9.0, Short term Nov 10, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tsteur tsteur

Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Milestone
2.9.0
Development

No branches or pull requests
4 participants
@tsteur @mattab @mnapoli @sgiehl