Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest S3 directly #17622

Closed
vsoch opened this issue May 25, 2021 · 4 comments
Closed

Ingest S3 directly #17622

vsoch opened this issue May 25, 2021 · 4 comments
Labels
answered For when a question was asked and we referred to forum or answered it.

Comments

@vsoch
Copy link

vsoch commented May 25, 2021

Hi matomo! I am wondering if there is a best practice for ingesting S3 logs directly, ideally from S3 and not needing to sync them to the same matomo server first and then using the script? Thank you!

@vsoch vsoch added the Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. label May 25, 2021
@sgiehl
Copy link
Member

sgiehl commented May 26, 2021

Hi @vsoch. Thanks for creating the issue. I assume you are talking about our log-importer?
Actually it should be possible to run the script an any server that supports python 3. You can send the extracted tracking request to any Matomo server that is reachable using the --url option. It might also be needed to set the --token-auth param manually in this case, at the log impoter might not be able to determine the token_auth automatically when running on a different server.

@vsoch
Copy link
Author

vsoch commented May 26, 2021

Thank you for the speedy response! So what would be best practice to consistently upload new logs from S3 - running a server or something like lambda alongside a Kubernetes deployment to run the log importer? Something else?

@diosmosis
Copy link
Member

Hi @vsoch, we don't have an established best practice for this specific use case. A lambda probably wouldn't work since there's a hard 15 minute run time limit (if I recall correctly). Can you launch a kubernetes job for this? Eg, when a log file is uploaded to S3 (if that is how you are using S3), launching a kubernetes job to download and import it. There are a lot of ways to accomplish this, it really depends on what you want and how your architecture is set up.

@vsoch
Copy link
Author

vsoch commented May 26, 2021

Ah, gotcha! Thank you for this discussion - we also have in mind to do a kubernetes job, and wanted to check if there was a suggested best practice first. I can come back here and comment after we get it working. But safe to close the issue, thanks again for your help!

@vsoch vsoch closed this as completed May 26, 2021
@tsteur tsteur added answered For when a question was asked and we referred to forum or answered it. and removed Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. labels May 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Development

No branches or pull requests

4 participants