@robocoder opened this Issue on February 6th 2012 Contributor

We would like to enable a new set of integration tests which call all known widgets, record output HTML in processed/ and then compare to expected/*.html files.

This will help us detect regressions in the Javascript/HTML rather than at the API level which we currently relying on.

This ticket will keep track of this work.

@mattab commented on February 8th 2012 Member

There are a few pending bugs with the widget compare mode.

  • There are dates in HTML outputs, so that changes every day
    piwik.startDateString = "2010-01-02";   piwik.endDateString = "2010-01-31"; piwik.minDateYear = 2010;   piwik.minDateMonth = parseInt("01", 10);    piwik.minDateDay = parseInt("02", 10);  piwik.maxDateYear = 2012;   piwik.maxDateMonth = parseInt("02", 10);    piwik.maxDateDay = parseInt("05", 10);  piwik.language = "en";</script><!--[if lt IE 9]>
  • The Piwik URL is found in the output
    piwik.piwik_url = "http://localhost/trunk/tests/";

It should be stripped out since jenkins/we all have different piwik urls.

  • The cache buster also changes
<link rel="stylesheet" type="text/css" href="themes/default/ieonly.css?cb=d269fbd1ca401f064e0d1a07049472c3" />
<span class="topBarElem"><a id="topmenu-widgetize" href="index.php?apiTestingLevel=none&amp;widgetTestingLevel=compare_output&amp;module=Widgetize&amp;action=index">Widgets</a></span> | 

I tried this patch but that wasn't enough to fix this issue so it is still pending:

Index: Integration.php
--- Integration.php (revision 5774)
+++ Integration.php (working copy)
@@ -92,6 +92,7 @@
    function setUp() 
+       $_GET = $_REQUEST = array();

        // Make sure translations are loaded to check messages in English 
@@ -119,6 +120,7 @@
    function tearDown() 
+       $_GET = $_REQUEST = array();

        // re-enable tag cloud shuffling
  • some tests are bound to fail, since they report a relative time since the data was created. For example, SEO widget contains:

<td><img style='vertical-align:middle;margin-right:6px;' src='plugins/SEO/images/whois.png' border='0' alt="Domain Age"> Domain Age
<div style='margin-left:15px'>
4&nbsp;years&nbsp;202&nbsp;days                                                 </div>  

which is a date in human readable form which will change every day. It should be stripped out before comparing html.

@mattab commented on February 8th 2012 Member

Anthon initial report:

  • When the integration tests run a second time (PDO_MYSQL then MYSQLI) on the CI server, an error is shown in the console log. Something missing from the setup/teardown?
[exec] <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
     [exec]    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
     [exec] <html>
     [exec] <head>
     [exec]     <title>Piwik &rsaquo; Error</title>
     [exec]     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
     [exec]     <link rel="shortcut icon" href="plugins/CoreHome/templates/images/favicon.ico" />
     [exec]     <link rel="stylesheet" type="text/css" href="themes/default/simple_structure.css" />
     [exec] </head>
     [exec] <body>
     [exec] <div id="contentsimple">
     [exec]     <div id="title"><img title='Piwik' alt="Piwik" src='themes/default/images/logo-header.png' style='margin-left:10px' /><span id="subh1"> # <a href='http://piwik.org/'>web analytics</a></span></div>
     [exec] <p>The parameter 'idGoal' doesn't have a correct type, and a default value wasn't provided.</p>
     [exec]                 <p><a href="index.php">Go to Piwik</a><br/>
     [exec]                 <a href="index.php?module=Login">Login</a></p>
     [exec]                 <font color="<a href='/888888'>#888888</a>">Backtrace:<br /><pre><a href='/0'>#0</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/plugins/Goals/Controller.php(57): Piwik_Common::getRequestVar('idGoal', NULL, 'string')
     [exec] <a href='/1'>#1</a> [internal function]: Piwik_Goals_Controller->widgetGoalReport()
     [exec] <a href='/2'>#2</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/core/FrontController.php(138): call_user_func_array(Array, Array)
     [exec] <a href='/3'>#3</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/core/FrontController.php(159): Piwik_FrontController->dispatch(NULL, NULL, NULL)
     [exec] <a href='/4'>#4</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/integration/Integration.php(765): Piwik_FrontController->fetchDispatch()
     [exec] <a href='/5'>#5</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/integration/Integration.php(1228): Test_Integration->callWidgetsCompareOutput('test_TwoVisitor...', 'all', Array, false, false)
     [exec] <a href='/6'>#6</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/integration/Integration.php(1112): Test_Integration_Facade->runControllerTests()
     [exec] <a href='/7'>#7</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/simpletest/invoker.php(68): Test_Integration_Facade->test_RunAllTests()
     [exec] <a href='/8'>#8</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/simpletest/invoker.php(126): SimpleInvoker->invoke('test_RunAllTest...')
     [exec] <a href='/9'>#9</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/simpletest/errors.php(49): SimpleInvokerDecorator->invoke('test_RunAllTest...')
     [exec] <a href='/10'>#10</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/simpletest/invoker.php(126): SimpleErrorTrappingInvoker->invoke('test_RunAllTest...')
     [exec] <a href='/11'>#11</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/simpletest/exceptions.php(43): SimpleInvokerDecorator->invoke('test_RunAllTest...')
     [exec] <a href='/12'>#12</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/simpletest/test_case.php(143): SimpleExceptionTrappingInvoker->invoke('test_RunAllTest...')
     [exec] <a href='/13'>#13</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/simpletest/test_case.php(595): SimpleTestCase->run(Object(HtmlTimerReporter))
     [exec] <a href='/14'>#14</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/simpletest/test_case.php(598): TestSuite->run(Object(HtmlTimerReporter))
     [exec] <a href='/15'>#15</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/TestRunner.php(111): TestSuite->run(Object(HtmlTimerReporter))
     [exec] <a href='/16'>#16</a> /home/www/data/root/jenkins.private/jobs/Piwik/workspace/build/tests/all_tests.php(19): TestRunner->run()
     [exec] <a href='/17'>#17</a> {main}</pre></font> <ul>
     [exec]                         <li><a target="_blank" href="?module=Proxy&action=redirect&url=http://piwik.org">Piwik.org homepage</a></li>
     [exec]                         <li><a target="_blank" href="?module=Proxy&action=redirect&url=http://piwik.org/faq/">Piwik Frequently Asked Questions</a></li>
     [exec]                         <li><a target="_blank" href="?module=Proxy&action=redirect&url=http://piwik.org/docs/">Piwik Documentation</a></li>
     [exec]                         <li><a target="_blank" href="?module=Proxy&action=redirect&url=http://forum.piwik.org/">Piwik Forums</a></li>
     [exec]                         <li><a target="_blank" href="?module=Proxy&action=redirect&url=http://demo.piwik.org">Piwik Online Demo</a></li>
     [exec]                         </ul></div>
     [exec] </body>
     [exec] </html>
@mattab commented on February 8th 2012 Member

The Widgets HTML integration tests were disabled in [5764]

@mattab commented on February 8th 2012 Member

(In [5785]) Refs #1465 Completely disabling widget testing since it fails for mysqli for some unknown reasons (to be investigated later)
Refs #2908

eg. See log in http://qa.piwik.org:8080/jenkins/job/Piwik/2838/consoleFull

@mattab commented on March 9th 2012 Member

(In [6014]) Refs #2908
remove .html widgets output until tests pass

@diosmosis commented on March 11th 2013 Member

In de5af594fdbb32744017fdd125dc3d957bc629b6: Refs #2908, refactored tests so database setup (adding sites, tracking visits) are separated from API tests. Put setup code into fixtures and reused code as much as possible.

@mattab commented on March 11th 2013 Member

Note: When the HTML changes, this should not "fail" the build but simply appear as a NOTICE message. Not sure how we can manage that, but it's important not to fail the build.

@halfdan commented on March 11th 2013 Member

A much better way than comparing processed vs. expected from files is to check for expected data in the HTML. This is what's common practice for Ruby tests, e.g. https://github.com/piwik/piwik-ruby-tracking/blob/master/spec/views/piwik_tracking_tag.html.erb_spec.rb

These two tests just check for the URL and the correct idSite in the tracking code. This is much more flexible and prevents copy/paste to get a test passing.

@mattab commented on March 11th 2013 Member

I think we need 2 levels:

  • Widget HTML tests that test for exact HTML output. This can be done by developers by running phpunit --group Widgets.
    • This is not included in default build therefore do not fail the CI build.
    • this is useful for example, when we do a big refactoring, we can easily check that nothing has been broken. Similarly useful if eg. we upgrade the template engine to latest version.
  • a new test that will, given a test fixture, generate the HTML of all widgets, and:
    • will look for differences in the "text output" as you suggest
    • will also look for notices/warning/errors and other error cases e.g. Returned output < 300 characters
      When text is different or there is a notice/error, the build fails.
@diosmosis commented on March 14th 2013 Member

@matt Regarding 'will look for differences in the "text output" as you suggest', what specifically will be looked for?

@mattab commented on March 14th 2013 Member

I'm not sure the best practises in the field, but I guess doing a "striptags" and looking at the "text" content (without html/js/css) could be a good basic test ?

@diosmosis commented on March 14th 2013 Member

The issue is what to look for. Visits/actions correlate directly with individual reports, but not all reports together. So, if we look for data, we'd have to specify data for each individual report, which is like what is done now, but all in PHP instead of in 'expected' files. I'm not sure what else there is to look for.

@mattab commented on March 17th 2013 Member

I think we just strip_tags and look at all the content. This is like the expected file test, but it will greatly increase coverage and also test the Controller stack + Views/visualization + templates.

@halfdan commented on March 17th 2013 Member

strip_tags is not a good idea. HTML can change quickly and data be represented differently. Using strip_tags will fail once you do a simple HTML reorder.

@mattab commented on August 9th 2013 Member

Testing for HTML is not necessarily a good idea. Instead we would like to generate screenshots and compare screenshots!

This Issue was closed on August 9th 2013
Powered by GitHub Issue Mirror