Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix exporting main metrics with a period of weeks and export format CSV #17442

Merged
merged 2 commits into from Jul 20, 2021

Conversation

ziegenberg
Copy link
Contributor

@ziegenberg ziegenberg commented Apr 9, 2021

Description:

FIXES #17441.

Escape the first column in CSV exports because it might contain a date span, and the date span will contain a comma.

If a test to prevent regressions is desired, I would need help with implementing that.

Note: As soon PHP 8 is the required minimum escaping could be limited with str_contains().

Review

  • Functional review done
  • Potential edge cases thought about (behavior of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
  • Usability review was done (is anything maybe unclear or think about anything that would cause people to reach out to support)
  • Security review done see checklist
  • Code review done
  • Tests were added if useful/possible
  • Reviewed for breaking changes
  • Developer changelog updated if needed
  • Documentation added if needed
  • Existing documentation updated if needed

@ziegenberg
Copy link
Contributor Author

Ready for review.

@ziegenberg
Copy link
Contributor Author

I did not run the tests locally and they might need to be updated.

@sgiehl
Copy link
Member

sgiehl commented Apr 9, 2021

Wondering if it might make sense to use a method like this to build a correctly escaped CSV string

function convertArrayToCSV($data) {
    $fh = fopen('php://temp', 'rw');
    fputcsv($fh, array_keys(current($data)));

    foreach ($data as $row) {
        fputcsv($fh, $row);
    }
    rewind($fh);
    $csv = stream_get_contents($fh);
    fclose($fh);

    return $csv;
}

Haven't tested it though, but could maybe prevent other issues with CSV.

@sgiehl sgiehl added the Needs Review PRs that need a code review label Apr 21, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2021

This issue is in "needs review" but there has been no activity for 7 days. ping @tsteur @sgiehl @diosmosis @flamisz

@github-actions github-actions bot added the Stale The label used by the Close Stale Issues action label Jun 4, 2021
@github-actions
Copy link
Contributor

This PR was last updated more than one month ago, maybe it's time to close it. Please check if there is anything we still can do or close this PR. ping @matomo-org/core-reviewers

@github-actions github-actions bot added the Stale for long The label used by the Close Stale Issues action label Jul 17, 2021
@ziegenberg
Copy link
Contributor Author

@sgiehl There is also buildCsvString() which is only called from renderDataTable(). So creating an extra csv-rendering function for renderDataTableMap() seems a bit redundant.

May it be feasible to combine both?

@sgiehl
Copy link
Member

sgiehl commented Jul 19, 2021

Sorry. Seems I didn't have a closer look before 🙈
I guess it might simply be enough to use the formatValue method, which should automatically escape the string if needed.

@github-actions github-actions bot removed Stale The label used by the Close Stale Issues action Stale for long The label used by the Close Stale Issues action labels Jul 20, 2021
@ziegenberg
Copy link
Contributor Author

ziegenberg commented Jul 20, 2021

I rebased onto current 4.x-dev and modified it to use formatValue(). It does seem to do the trick! :D

@sgiehl
Copy link
Member

sgiehl commented Jul 20, 2021

Thanks for updating the PR. Looks good now. Will wait until the tests finished and if all relevant tests are passing will merge it.

@ziegenberg
Copy link
Contributor Author

should we add some unit tests for regression testing?

@sgiehl
Copy link
Member

sgiehl commented Jul 20, 2021

should we add some unit tests for regression testing?

That would be nice. Maybe we could add one here: https://github.com/matomo-org/matomo/blob/4.x-dev/tests/PHPUnit/System/CsvExportTest.php#L35-L39

@ziegenberg
Copy link
Contributor Author

I guess CsvExportTest::$fixture = new TwoVisitsWithContents(); here would need to be extended as $dateTime = self::$fixture->dateTime; is currently set to a single date and not a time span?

@sgiehl
Copy link
Member

sgiehl commented Jul 20, 2021

In the test array linked above you can simply add something like this:

            array($apiToCall, array('idSite'                 => $idSite,
                                    'date'                   => Date::factory($dateTime)->toString() .','. Date::factory($dateTime)->addDay(21)->toString(),
                                    'period'                 => 'week',
                                    'format'                 => 'csv',
                                    'testSuffix'             => '_multi')),

Running the tests should then produce 2 new result files in processed folder, that can simply be moved to expected files.

… weeks and export format CSV

Signed-off-by: Daniel Ziegenberg <daniel@ziegenberg.at>
@ziegenberg
Copy link
Contributor Author

I guess, this is as good as it gets for the moment. Tests are added. I'm waiting for Travis opinion on my work. Locally the tests run through.

-> ready for review

@sgiehl
Copy link
Member

sgiehl commented Jul 20, 2021

@ziegenberg Thanks for that. With those tests we actually have all we need. Tests also seem to pass 👍

@sgiehl sgiehl added this to the 4.4.0 milestone Jul 20, 2021
@sgiehl sgiehl added not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. and removed Needs Review PRs that need a code review labels Jul 20, 2021
@sgiehl sgiehl merged commit 4d21dd2 into matomo-org:4.x-dev Jul 20, 2021
@ziegenberg ziegenberg deleted the 17441 branch July 20, 2021 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Exporting main metrics with a period of "week" as CSV produces broken exports
2 participants