bamboo.openx.org server appears to be no longer available; should we close this or look into setting up our own (eg using xinc)?

vipsoft, we should consider running our own bamboo, or maybe Hudson as it seems better. We could have 1 or 2 boxes by our sponsor Free to support it - if you want to have a try I can give you the server details

I looked at xing, interesting to build this in php! the project hasn't seen an update for a while though.. I used Hudson in the past, and people were very happy with it; !http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson

I thought we could use xinc to avoid using a java server, but since webtest is java-based, we might as well use bamboo or hudson.

Note: the current Bamboo is at http://bamboo.openx.org:8085/browse/PIWIK-TRUNK but seems to fail due to some errors. We would like to setup our own integration build server at eg. tests.piwik.org

I asked atlassian and they offered a free license of Bamboo. However, it might be better for us to pay for hosted continuous integration server (bamboo or hudson). If anyone knows such a hosted service, let us know!

System requirements are:

• multiple PHP CGI versions (e.g., 5.1.3, 5.2.0, 5.2.6, 5.2-SVN, 5.3.1, 5.3-SVN)
• Tomcat (or other Java server) to support bamboo or hudson; or Apache if we go with xinc
• Java runtime, ant, JDK for WebTest
• MySQL (and in the future, other databases)

Is Free still offering a box (or two)? Maybe we can set this up ourselves.

Anthon,
Yes the servers are still available. I can give you the details by email if you'd like to give it a try.

Should test Piwik with:

• safe_mode disabled and no open_basedir restriction
• safe_mode enabled
• open_basedir restriction
• safe_mode enabled and open_basedir restriction

To test the one click update process, a script could download and install the latest release (latest.zip). Then instead of api.piwik.org/1.0/getLatestVersion, it could check api.piwik.org/1.0/getLatestSvnRevision, and update to the latest svn snapshot, e.g., !http://dev.piwik.org/trac/changeset/{latest revision}/trunk?old_path=%2F&format=zip. (Unlike the build script, Trac packages it in the "trunk" folder instead of "piwik".) It gets a little trickier because the piwik_option table stores the version number in UpdateCheck_LatestVersion, not the SVN version.

It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP.

Should also include testing with E_STRICT because some environments don't permit the error_reporting level to be overridden by .htaccess or scripts.

(In [2269]) refs #818 - unit tests with both pdo_mysql and mysqli adapters

(In [2270]) refs #818 - fix typo in target

(In [2271]) refs #818 - bamboo.openx.org doesn't have mysqli extension ... comment out for now

Replying to vipsoft:

It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP.

Cool. Hudson and Bamboo support remote agents, or on the cloud (Amazon EC2) -- what Atlassian calls "elastic agents". Neither provide Amazon Machine Images for Windows, nor support for Microsoft Azure.

Minor issue with Hudson on Jetty: http://issues.hudson-ci.org/browse/HUDSON-6760

(In [2318]) refs #818 - build directory missing a folder

Note to self: It appears there's a bug in HtmlUnit 2.7 that causes WebTest R_1804 to fail the web test after sign in. Sizzle.filter (in jquery.js 1.4.2) is throwing an exception (Syntax error, unrecognized expression: ul li) for this selector in superfish-modified.js, "li:has(ul li)".

Anthon, I have a few questions

• Will Hudson run on Mysql, mysqli all tests?
• Is there a way to trigger the JS tests for piwik.js, as well as unit tests and webtests?
• Would it be better if we migrate to PhpUnit if we want to have the unit tests ran easily (right now Bamboo doesn't parse Simpletest output)

I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release?

Thanks

Replying to matt:

Anthon, I have a few questions

• Will Hudson run on Mysql, mysqli all tests?

Eventually. I commented out the mysqli unit test because bamboo.openx.org (or rather its elastic agent) doesn't have the mysqli extension installed.

• Is there a way to trigger the JS tests for piwik.js, as well as unit tests and webtests?

We could use webtest (via HtmlUnit browser simulation). Need to investigate further.

• Would it be better if we migrate to PhpUnit if we want to have the unit tests ran easily (right now Bamboo doesn't parse Simpletest output)

I might be able to wrap it in a webtest testcase.

I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release?

As of last nite, I have it running locally but the webtest is failing on the second-to-last step. I tried using Webtest 3.0 and the same version of webtest as on bamboo.openx.org (r 1783) but both choked, so I'm now running the latest snapshot and trying to locate the source of the failure in HtmlUnit. Right now, it's not looking hopeful for end-of-the-month.

Re: piwik.js. It looks like we can use Selenium for native browser testing. (There's a Hudson plugin for this.)

(In [2414]) refs #818 - phpdoc 1.4.3 expects this to be an absolute path

Looks like #660 works around the problem in HtmlUnit.

I'm going to start moving my build setup over to dev6.

Awesome.

The continuous build server running Unit tests as well as webtests will be a life saver!! (not to note the possibility of running mysqli and later postgresql...)

Another option for native browser testing of piwik.js is to run our own Testswarm instance. http://wiki.github.com/jeresig/testswarm/

It looks like bamboo.openx.org is down, so there's greater urgency to get this setup on dev6.

In r2429, Matt indicates that Config.test.php fails when run standalone. We should definitely consider iterating through the unit tests, in addition to all_tests.php, as this can catch some side-effects (due to globals).

Fixed the config.test.php in r2433

(In [2445]) refs #818 - add webtest wrapper around unit tests

Hudson is now running at http://qa.piwik.org:8080/ with some the unit tests are failing. (See other tickets.)

I'll open a new ticket for improvements.

Nice work! The feeds are broken though:

http://qa.piwik.org:8080/hudson/rssFailed

The items links to iproxy.ca:8080.

(In [2485]) Refs #818 - upcase words in installation steps; use "JavaScript" consistently; fixes WebTest (installation)

Artifacts can be access through the "Workspace" link, once you're logged in, e.g.,

• http://qa.piwik.org:8080/hudson/job/Piwik/ws/default/tests/results/

(There's a configuration option to archive artifacts somewhere...)

Yes, the plan is to use Piwik_QA to tweet build results.

Is it possible to make Artifacts public? I can't login and would very much like to see the current build status and possible errors.

http://trac-hacks.org/wiki/HudsonTracPlugin would be nice for trac to see the build history from hudson in the trac timeline.

re: public artifacts are possible, but we'll have to review the artifacts to ensure no passwords and internal paths are disclosed.

re: hudson trac plugin. Good idea. We don't currently use the Bamboo Trac plugin that's currently installed.

I've rebuilt php with --enable-mbstring, so the mb_convert_encoding calls work (requires for .csv and .tsv output).

The webtest wrapper on the unit tests is currently failing because the output isn't proper html, eg text before the <html> and after the </html>.

http://qa.piwik.org:8080/hudson/job/Piwik/ws/default/tests/results/002_UnitTests/001_response_invoke.xml/*view*/

(In [2488]) refs #818 - clean up output

(In [2489]) refs #818

Current status of our Hudson server:

• php 5.2.13 and pdo_mysql
• unit tests via php-cli (excludes tests/integration) passes
• webtest install passes
• webtest unit tests (includes tests/integration) passes
• phpdocs generated

Are the phpdocs artifacts available for download? It would be cool to link the http://piwik.org/docs/tracking-api/ doc page to the PiwikTracker API page (to the doc from latest successful build maybe?).

I made some configuration changes to the job, but they're not having any effect. It appears that webtest results are treated as build artifacts, and artifacts are not published if the build fails.

I'll try scripting a workaround.

• webtest results (pass/fail) copied to http://qa.piwik.org/webtest/
• Piwik documentation copied to http://qa.piwik.org/phpdocs/ (if generated)

In [2502], exclude integration tests if REQUEST_UR or HTTP_HOST not defined; this means we can now fully test with php-cli if these CGI environment variables are defined.

(In [2503]) refs #818 - PATH_INFO and SCRIPT_NAME are also accepted

Great for artifacts. Should the URL be prefixed by the job ID? (or do we only keep artifacts for last build?)

When you say, "if generated", are the latest successful build doc pages always going to be there (ie. failing builds don't delete old docs). Can I safely link to http://qa.piwik.org/phpdocs/Piwik/PiwikTracker.html ?

Webtest results are only for the latest build.

Yes, should be safe to link to the docs. It's only overwritten if the build succeeds.

I linked the page from the bottom of: http://piwik.org/docs/tracking-api/