Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Github issues mirror for Data ownership and better searchability, on issues.piwik.org #5299

Closed
mattab opened this issue Jun 5, 2014 · 6 comments
Assignees
Labels
Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Jun 5, 2014

Follows up #5273 where we migrated Piwik issues from Trac to Github issues. This is following up our previous migration from SVN to git.

The possibly only downside of moving to Github issues is that we lose the control over the tickets. This has many up-sides (github is faster, more reliable, less frustrating to manage) but we also lose traffic from Search Engine, and possibly Github could go down.

Goal: We wish to build a Mirror of all our issues content.

Note:

@mattab
Copy link
Member Author

mattab commented Jun 10, 2014

From Github, a bit more info:

Thanks for your feedback. Currently, as you noticed, the API allows you to make a backup of all the issues and comments in your repositories. The rate limits are in place to keep the API fast and reliable for everyone, which has always been a priority for as, as is making this data available to you.

At the moment, we don't offer a way to download all issues and comments as a ZIP archive. I'm happy to pass the suggestion to the team, but I can't promise an ETA for when this might be available.

In the meantime, using caching and conditional requests should allow you to re-fetch issues and comments for each archive cycle without draining your API quota:

https://developer.github.com/v3/#conditional-requests

@mattab mattab added this to the 2.x - The Great Piwik 2.x Backlog milestone Jul 8, 2014
@mattab
Copy link
Member Author

mattab commented Jul 9, 2014

In our latest blog post I wrote: At Piwik we care a lot about Data ownership. For this reason we need to have an up to date copy of all our tickets and comments out of github.com servers. Our next step will be to create and release as open source a tool to let anyone create a Mirror of their Github issues.

@tsteur
Copy link
Member

tsteur commented Jul 11, 2014

@tsteur
Copy link
Member

tsteur commented Jul 11, 2014

A first version is running on http://issues.piwik.org/ . A cronjob is setup to sync issues once per day (can take up to an hour or even more). We're getting an email to developer@piwik.org in case something goes wrong during the cronjob.

It's a bit "slow" meaning it takes about 400ms on the server side, sometimes up to 1second. Can you enable APC on this server? It takes not even 100ms on my server...

Need to see how stable the importer is in the next days and whether everything works...

@tsteur
Copy link
Member

tsteur commented Jul 18, 2014

Seems to work and google is already indexing. Will close it for now. Future issues should be created in https://github.com/piwik/github-issues-mirror/issues

@mattab
Copy link
Member Author

mattab commented Aug 3, 2014

Amazing Thomas! well done on this ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Development

No branches or pull requests

2 participants