See discussion in matomo-org/developer-documentation#560

So far we have a so called OmniFixture which is generated by executing all our different test fixtures. This is so far done that way to have all kind of different test data in the UI tests.

In matomo-org/developer-documentation#560 we were discussing how often to update this OmniFixture to keep things efficient or if we maybe want to change the way this OmniFixture works completely.

Problem

When changing one of these fixtures and then updating the OmniFixture, one or many UI tests will fail because the metrics (numbers) in the tests may be different etc. For every single screenshot you then need to investigate if the change is actually expected and still accurate. This requires checking what the raw data is in the DB to verify if the new metric numbers in the changed screenshots are still correct.

This can easily take many hours every time and is tedious if we want to ensure that the tests are still accurate and that the shown results are expected. Meaning each screenshot will take quite a bit of time to investigate. Sometimes that will be easier depending on what fixture changes, sometimes it'll take a lot longer.

When there's only a simple change in a fixture then it can be faster to update the OmniFixture right away to keep it updated. If a changed OmniFixture impacts many screenshots for example because a fixture tracks more data, then it can take much more time to update these screenshots every time. That's unless we say the data in the UI tests is irrelevant and it's ok if they are wrong as they are tested by system tests and we only look at the functionality that the screenshots tests such as opening a subtable, flattening, ...

The more developers we have, the more often our test fixtures will change and the more maintenance work we have with this OmniFixture and the more of our time this can consume. Even to the point where someone might rather not change a test fixture to not having to also update the OmniFixture as it's tedious etc.