Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move UI unit tests away from PHP code and run directly in phantomjs w/ jasmine testing framework #4739

Closed
diosmosis opened this issue Feb 22, 2014 · 27 comments
Labels
Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone

Comments

@diosmosis
Copy link
Member

Converting UI tests to JavaScript will confer many benefits including:

  • Clearer test output (& no travis timing issues).
  • Faster tests (we can setup just once for all tests & don't have to execute phantomjs multiple times).
  • Clearer test code (via Jasmine's BDD style output).

Steps to complete:

  • Move all of test environment setup to Fixture class. (done in uiunittests branch)
  • Extend mocha & chai to handle screenshot compare use-case.
  • Create JS script that will run a PHP program to setup the database and run each UI test suite.
  • Modify existing UI tests.
  • Modify travis config to run via phantomjs.

Reason for using Jasmine: phantomjs uses Jasmine internally for its tests so we know it'll work. mocha + chai can be used as an alternative.

CC-ing tsteur as per matt's request.

Update: Using mocha + chai is actually easier to get running and easier to extend.

@tsteur
Copy link
Member

tsteur commented Feb 23, 2014

+1 for MochaJS see #4691

@diosmosis
Copy link
Member Author

In 71dee21: Refs #4739, adding mocha + chai to tests/lib dir.

@diosmosis
Copy link
Member Author

In 39c8961: Refs #4739, remove unneeded persist_fixture_data.

@diosmosis
Copy link
Member Author

In f4e39c9: Refs #4739, #4189, remove persist_fixture_data config option code, placed OmniFixture into a namespace, removed PHP screenshot testcase classes and added fixture for use w/ JavaScript screenshot tests.

@diosmosis
Copy link
Member Author

In f2faa23: Refs #4739, make sure recursive directory links are created even when --persist-fixture-data is used.

@tsteur
Copy link
Member

tsteur commented Feb 24, 2014

@capedfuzz it would be cool if we could reuse some parts of this to write JavaScript unit tests for all the angular code

BTW: probably not really useful but just mention it... I built something similar a year ago to run tests using PhantomJS + Jasmine 2.0 but would now use MochaJS. Reckon there is not really anything new for you but just wanted to mention it. There is a testrunner https://github.com/tsteur/titanium-alloy-jasmine-testing/blob/master/tests.js and a loader https://github.com/tsteur/titanium-alloy-jasmine-testing/blob/master/specs/jasmine/utils/loader.js who finds all the test files in a specs directory: https://github.com/piwik/piwik-mobile-2/tree/master/specs

@diosmosis
Copy link
Member Author

@tsteur If the tests for the angularjs code will run in a webpage, then you won't need any code I've written. The screenshot tests run directly in phantomjs, not in a webpage.

@diosmosis
Copy link
Member Author

In bba1420: Refs #4739, allow JavaScript screenshot tests to override Piwik behavior by storing values in tmp/ file, add hook so plugins can setup their own test environment, reset user password on test fixture setup.

@diosmosis
Copy link
Member Author

In bf92e42: Refs #4739, add console command to run UI tests.

@diosmosis
Copy link
Member Author

In 35586df: Refs #4739, pass groups to new RunUITests console command.

@diosmosis
Copy link
Member Author

In 8ba7f01: Refs #4739, move screenshot testing library code to core repository (from UI repository).

@diosmosis
Copy link
Member Author

In 4ad2b98: Refs #4739, make sure UI screenshots are stored in plugin directories if tests are for plugins, cleaned up path output in console and add .gitignore to DBStats plugin.

@diosmosis
Copy link
Member Author

In 8d72a9f: Refs #4739, fixing bug in processed screenshot dir path creation.

@diosmosis
Copy link
Member Author

In ba6a60a: Refs #4739, add information about new UI tests to screenshot testing README.

@diosmosis
Copy link
Member Author

In f073d10: Refs #4739, modify SyncUITestScreenshots command to work with new screenshot testing code and display location hint in diffviewer output if expected file is stored within plugin and not the UI tests repo.

@diosmosis
Copy link
Member Author

In 28a2f9b: Refs #4739, add travis only command line options to UI test runner: --store-in-ui-tests-repo, --use-github-expected.

@diosmosis
Copy link
Member Author

@tsteur FYI, I did some thinking and I think the screenshot testing code could be used for angularjs tests too. For example:


describe("Something", function () {
    var webpage,
        url = config.piwikUrl + "?module=CoreHome&action=blankPage"; // page loads all JavaScript but no contents

    before(function (done) {
        webpage = require('webpage').create();
        webpage.load(url, done);
    });

    after(function () {
        webpage.close();
    });

    it("...", function () {
        var result = webpage.evaluate(function () {
            // do test here ...
        });

        expect(result).to.be.equal('whatever');
    });
});

I'm not sure how much extra work it will be to get it to work or if there are any big benefits, but I think it's possible.

@diosmosis
Copy link
Member Author

In bb5e2b3: Refs #4739, remove old capture.js file, add section about screenshot testing config to README and remove unused constant from Config.php.

@diosmosis
Copy link
Member Author

In f44b952: Refs #4739, move segment setup code to UITestFixture and make sure it is run on every test run.

@diosmosis
Copy link
Member Author

In 9c9a91f: Refs #4739, remove zeitgeist query parameter handling in TestingEnvironment.

@diosmosis
Copy link
Member Author

In c9f13e1: Refs #4739, add dragDrop utility function to screenshot test page renderer and capture alert()'s during screenshot tests.

@diosmosis
Copy link
Member Author

In d789162: Refs #4739, display page logs when a screenshot test times out.

@diosmosis
Copy link
Member Author

In 0a98302: Refs #4739, in the screenshot test diffviewer, display the location hint using the processed file path if the expected file does not exist yet.

@diosmosis
Copy link
Member Author

In a43cae6: Refs #4739, pass along errors to callback in testEnvironment Piwik API/Controller calls.

@diosmosis
Copy link
Member Author

In 202671a: Refs #4739, only pad widgets in screenshot tests when the element is a direct child of a .widget element.

@mattab
Copy link
Member

mattab commented Mar 11, 2014

awesome work Benaka there! is this ticket closed?

@mattab
Copy link
Member

mattab commented Mar 20, 2014

In e4b1c9c: Refs #4739 Run the UI tests build as early as possible so we don't have to wait 30 minutes to see screenshots

@diosmosis diosmosis added this to the 2.2.1 - Piwik 2.2.1 milestone Jul 8, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…nfig option code, placed OmniFixture into a namespace, removed PHP screenshot testcase classes and added fixture for use w/ JavaScript screenshot tests.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
… even when --persist-fixture-data is used.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…iwik behavior by storing values in tmp/ file, add hook so plugins can setup their own test environment, reset user password on test fixture setup.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…irectories if tests are for plugins, cleaned up path output in console and add .gitignore to DBStats plugin.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…th new screenshot testing code and display location hint in diffviewer output if expected file is stored within plugin and not the UI tests repo.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
… runner: --store-in-ui-tests-repo, --use-github-expected.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…creenshot testing config to README and remove unused constant from Config.php.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…t page renderer and capture alert()'s during screenshot tests.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…location hint using the processed file path if the expected file does not exist yet.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…lement is a direct child of a .widget element.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…e don't have to wait 30 minutes to see screenshots
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Development

No branches or pull requests

3 participants