Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build a demo application that tracks all possible data for manually tracking testing #7066

Closed
tsteur opened this issue Jan 23, 2015 · 12 comments
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. RFC Indicates the issue is a request for comments where the author is looking for feedback.
Milestone

Comments

@tsteur
Copy link
Member

tsteur commented Jan 23, 2015

I'm currently working on #7056 and looking through the code trying to understand what that bug causes. After a while I noticed it is not that easy and I actually need to reproduce this issue. As this problem is related to Ecommerce and UserId (Login/Logout) it is not as easy to test as one has to call several tracking calls manually and one has to be careful to set correct data etc.

Usually best way would be trying to create an integration test that shows the problem so we also make sure for the future that we won't have it again. But sometimes it is hard to test this way correctly and it takes much more time when someone just wants to have a look whether there is a bug or not etc. Sometimes one has to write the integration test afterwards as one sometimes has no clue how the reproduce that problem... Also sometimes the problem is not in PHP but in JS, eg Content-tracking has a lot of DOM logic etc...

While we can already track Piwik itself (config enable_measure_piwik_usage_in_idsite) it would be nice to have another separate website that anyone can use to reproduce such problems and to generate visits if needed etc. It could be just HTML pages if possible or a PHP based dummy application with a fake shop, with login/logout (userId), content-tracking, etc. Kinda like a "Kitchen Sink" demo.

For example for content-tracking I adjusted the Marketplace in my local instance to track the plugins using content-tracking. Next time I have to test this again I need to do this again and again...

Maybe this application could be even used for tracker tests somehow but probably not.

@tsteur tsteur added Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. Help wanted Beginner friendly issues or issues where we'd highly appreciate community's help and involvement. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. labels Jan 23, 2015
@tsteur tsteur added this to the Mid term milestone Jan 23, 2015
@tsteur tsteur changed the title Build a demo application that tracks all possible data for manual tracking testing Build a demo application that tracks all possible data for manually tracking testing Jan 23, 2015
@mattab
Copy link
Member

mattab commented Feb 10, 2015

For example for content-tracking I adjusted the Marketplace in my local instance to track the plugins using content-tracking. Next time I have to test this again I need to do this again and again...

That's a nice idea! could we push this to Production marketplace as it would give us some interesting content tracking data?

@tsteur
Copy link
Member Author

tsteur commented Feb 10, 2015

We could, maybe a new issue for this?

@mattab
Copy link
Member

mattab commented Feb 10, 2015

(done.)

It's a good idea to create a fake app that tracks all kinds of data, events, etc. Although somehow I feel it would be nice to do this in Piwik app itselfe since it's easy enough to click 10-20 times to generate data. We could generate events, content tracking, user id, most features, and on each page view to Piwik it would track data for most features. Of course you mention "Ecommerce" which would not be easy to fake in Piwik but if we wanted we could send fake ecommerce transactions eg. when the user visits Piwik ecommerce reports. I'd prefer to work on measuring Piwik because it's so important and this issue would be a good excuse to get started on this :-)

@tsteur
Copy link
Member Author

tsteur commented Feb 10, 2015

I would rather want to see a separate app since it will allow you easier to test such stuff. Eg you see an issue and you need to exactly reproduce the steps... This will be hard with the Piwik UI since you won't be able to do it and you wouldn't be transparent when Piwik tracks what...

As a temporary solution it would be maybe ok but it wouldn't actually really solve my problem (debugging, adding new features to tracker etc)

@tsteur
Copy link
Member Author

tsteur commented Feb 10, 2015

Tracking some more features in Piwik itself apart from this issue sounds good though

@tsteur
Copy link
Member Author

tsteur commented Jan 21, 2016

Ideally, at some point, we would track all possible features in such an application, and also use it for automated real live tests in various browsers. This would be really valuable.

This application would be good for automated tests, but also for manual tests and for learning how to integrate tracking into a website. We would have info buttons / sidebar that show how to integrate a just used feature, how to do the same with JavaScript tracker, iOS tracker, Android tracker, via HTTP Api etc. Even though it sounds small I believe there is huge value.

Ideally we would also host it at some point on a Piwik domain and people could test it and see the requests on demo.piwik.org

If someone wants to work on such feature, please ping me. It's really interesting thing to build from scratch. Could use some mobile frameworks for this like ionic 2, angularjs2, or other fancy stuff :)

Repository for this will be https://github.com/piwik/tracking-kitchen-sink

Example for another KitchenSink: http://dev.sencha.com/extjs/5.0.0/examples/kitchensink/

@hpvd
Copy link

hpvd commented Jan 21, 2016

sounds pretty interesting and valuable!

@mattab mattab modified the milestones: Long term, Mid term Dec 5, 2016
@tsteur tsteur added RFC Indicates the issue is a request for comments where the author is looking for feedback. and removed Help wanted Beginner friendly issues or issues where we'd highly appreciate community's help and involvement. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. labels Feb 23, 2021
@tsteur
Copy link
Member Author

tsteur commented Feb 24, 2021

@diosmosis @sgiehl @flamisz

moving the thoughts from slack into this issue now. We could have a folder say tests/demo (can easily rename any time) where we have a lose collection of useful individual HTML pages (could partially also be PHP) that are needed/useful to reproduce or test certain features.

For example:

  • tests/demo/optout-with-tracker-on-same-page.html which would embed the opt out screen and have the tracking code set up to the same domain (as this would try to additionally do the opt out using iframe messages)
  • tests/demo/optout-without-tracker-on-same-page.html which would embed the opt out screen but not have the tracking code integrated
  • tests/demo/tracking/ecommerce.html which could have a few buttons to simulate "add to cart", "cart update", "order"
  • etc.

All tracking related test pages we would put in a subfolder tests/demos/tracking. I think we would use these pages mostly for tracking related things but it might be also useful for other things (eg optout iframe, we could also have one for tracking consent and one for cookie consent and custom opt out).

I think there wouldn't be much of restrictions on what to put there or not. The idea would be whenever we need to test/reproduce something, we'd check if an example is there and if not, then we think about whether it may be useful to add one so others can reuse it in the future if we find ourselves building such a page anyway.

As idSite we would always use 1. I think it wouldn't work much for widgets since they would mostly need a token I suppose.

Regarding keeping them up to date: Since we mostly keep BC I suppose they shouldn't regress too much and if they do become outdated, that may be fine as I suppose we'd notice this when trying to use one of those examples.

I suppose while we build new features or fix bugs (mostly tracking related) we build such pages often for ourselves (I've built many of them over the years but often find myself changing the same html pages and throwing away what I had created previously) and then we would instead share these pages with the team. There would be no obligation to add such a page as part of building a feature although it would be great to do in some cases as it would be mostly needed anyway for the author of a new feature as well as the tester.

Any thoughts/ideas? Ideally we can also just start doing this and see how it goes over time and tweak things.

@tsteur tsteur modified the milestones: Backlog (Help wanted), 4.2.0, 4.3.0 Feb 24, 2021
@tsteur
Copy link
Member Author

tsteur commented Feb 24, 2021

We just discussed the idea that the support team might need this as well so we're thinking of a new public repository where we create PHP files instead of HTML files and have eg idSite, hostname and token etc configurable.

@tsteur
Copy link
Member Author

tsteur commented Feb 26, 2021

FYI I have created https://github.com/matomo-org/test-examples for now. The initial structure and how everything works is just an initial start and we can change anything.

@tsteur
Copy link
Member Author

tsteur commented Feb 26, 2021

FYI created https://developer.matomo.org/guides/reproducing-issues as a start for a new doc and also added https://developer.matomo.org/guides/contributing-to-piwik-core#creating-a-pull-requestsharing-test-example-pages for now.

@diosmosis @sgiehl @flamisz @mattab any other thoughts on this or can we close this issue maybe? We could add another check to the pull request template just to remind us to think about it but it wouldn't be a requirement as most of the times this would not be possible.

@mattab
Copy link
Member

mattab commented Feb 28, 2021

This looks great! 😎

it would be helpful if the test examples also add the basic tracking (pageviews, events, content tracking, outlink, download tracking, goal id 1, custom dimension 1) but it can be added later of course and it'd be fine for me to close this

@tsteur tsteur closed this as completed Feb 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. RFC Indicates the issue is a request for comments where the author is looking for feedback.
Projects
None yet
Development

No branches or pull requests

3 participants