New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
QA: Enable Widgets HTML compare integration tests #2908
Comments
There are a few pending bugs with the widget compare mode.
It should be stripped out since jenkins/we all have different piwik urls.
I tried this patch but that wasn't enough to fix this issue so it is still pending:
which is a date in human readable form which will change every day. It should be stripped out before comparing html. |
Anthon initial report:
|
The Widgets HTML integration tests were disabled in [5764] |
(In [5785]) Refs #1465 Completely disabling widget testing since it fails for mysqli for some unknown reasons (to be investigated later) eg. See log in http://qa.piwik.org:8080/jenkins/job/Piwik/2838/consoleFull |
(In [6014]) Refs #2908 |
Note: When the HTML changes, this should not "fail" the build but simply appear as a NOTICE message. Not sure how we can manage that, but it's important not to fail the build. |
A much better way than comparing processed vs. expected from files is to check for expected data in the HTML. This is what's common practice for Ruby tests, e.g. https://github.com/piwik/piwik-ruby-tracking/blob/master/spec/views/piwik_tracking_tag.html.erb_spec.rb These two tests just check for the URL and the correct idSite in the tracking code. This is much more flexible and prevents copy/paste to get a test passing. |
I think we need 2 levels:
|
@matt Regarding 'will look for differences in the "text output" as you suggest', what specifically will be looked for? |
I'm not sure the best practises in the field, but I guess doing a "striptags" and looking at the "text" content (without html/js/css) could be a good basic test ? |
The issue is what to look for. Visits/actions correlate directly with individual reports, but not all reports together. So, if we look for data, we'd have to specify data for each individual report, which is like what is done now, but all in PHP instead of in 'expected' files. I'm not sure what else there is to look for. |
I think we just strip_tags and look at all the content. This is like the expected file test, but it will greatly increase coverage and also test the Controller stack + Views/visualization + templates. |
strip_tags is not a good idea. HTML can change quickly and data be represented differently. Using strip_tags will fail once you do a simple HTML reorder. |
Testing for HTML is not necessarily a good idea. Instead we would like to generate screenshots and compare screenshots! |
We would like to enable a new set of integration tests which call all known widgets, record output HTML in processed/ and then compare to expected/*.html files.
This will help us detect regressions in the Javascript/HTML rather than at the API level which we currently relying on.
This ticket will keep track of this work.
The text was updated successfully, but these errors were encountered: