Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

during core:update display SQL queries as they are executed #7276

Closed
futureweb opened this issue Feb 23, 2015 · 7 comments
Closed

during core:update display SQL queries as they are executed #7276

futureweb opened this issue Feb 23, 2015 · 7 comments
Assignees
@diosmosis
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone
2.12.0

Comments

@futureweb
Copy link
Contributor

Hi,

we got a pretty large Piwik DB - and DB Updates on Piwik Updates often takes several hours to complete ... It would be VERY HELPFUL if the CLI Update Script (console core:update) would show some information on progress ...
I know it's not possible to output estimated time left ... but you could output something like

"14 out of 147 Queries finished"
"15 out of 147 Queries finished"
"16 out of 147 Queries finished"

Would be a good feeling to see there is some progress while waiting a few Hours for the Script to complete ... ;-)

Thank you, bye from Austria
Andreas Schnederle-Wagner
@gaumondp
Copy link

+1 , I almost kill an upgrade a month ago because I wasn't sure it was still running...

Hopefully I opend a second SSH session and used "htop" and saw the script running.

Going 2.7 to 2.11 for a 14 GB database took 36 minutes (8 GB, 2 CPU MySQL-only server) CPU at 40% while updating.

How large is your DB @futureweb ?

@mattab mattab added the Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. label Feb 23, 2015
@mattab mattab added this to the Short term milestone Feb 23, 2015
@mattab
Copy link
Member

mattab commented Feb 23, 2015

Hi guys, thanks for the suggestion. It's a must-have and we will implement it in the next few months.

@mattab mattab modified the milestones: Piwik 2.12.0, Short term Feb 23, 2015
@diosmosis diosmosis self-assigned this Feb 23, 2015
@futureweb
Copy link
Contributor Author

Hey Matthieu,

sounds good - thx for picking up my idea so fast! ;-)
@gaumondp: DB is about 32GB ... and update from 2.10 --> 2.11.1 took something between 1 and 2 Hours ... so I'm afraid everytime a DB Update is coming towards me ...

Andreas

diosmosis pushed a commit that referenced this issue Feb 25, 2015
Refs #7276, initial backwards compatible refactor of updater classes …
fd305b4 
…(moved CoreUpdater static functions to core/Updater class, remove use of static methods in Updater, don't use static method setUpdater in ColumnsUpdater).
diosmosis pushed a commit that referenced this issue Feb 25, 2015
Refs #7276, finish updater refactoring; use instance methods of Updat…
bc28d90 
…er in Columns\Updater and document methods in Updater.
diosmosis pushed a commit that referenced this issue Feb 25, 2015
Refs #7276, fill out UpdateListener, use UpdateListener in core/Updat…
9ac8e1d 
…er update process and document UpdateListener.
diosmosis pushed a commit that referenced this issue Feb 25, 2015
Refs #7276, add integration test for core:update command, fix strict …
5a830fb 
…notice errors caused by change to Updates.php and add some more functionality to ConsoleCommandTestCase base class.
@samgabriel
Copy link

Hi guys, Can you please stop updating the piwik_log_visits and piwik_log_link_visit_actions These tables are huge on any normal high traffic site. It would be much better to create a new table and do a 1:1 mapping. The cost on inserting into multiple tables cannot justify being down for 8 hours while the update is completing which is what is happening in our case now.
Doing an update from 2.4.0 to 2.11.0 so far has taken about 8 hours for the alter column statement. Yesterday we were updating from 2.2 to 2.3 which also took about 8 hours to update the piwik_log_action table.

@mattab
Copy link
Member

mattab commented Feb 26, 2015

@samgabriel currently such DB schema changes are blocked, see issue #6953

diosmosis pushed a commit that referenced this issue Feb 26, 2015
Refs #7276, recreate CLI update logic originally in web controller in…
ed3917d 
… core:update command.
@mattab mattab changed the title Update Script enhancement during core:update display SQL queries as they are executed Mar 2, 2015
@mattab
Copy link
Member

mattab commented Mar 2, 2015

Btw I think it already works to do core:update --verbose to see which SQL queries are being executed

@diosmosis
Copy link
Member

That doesn't tell you the progress, and it shows every SQL statement Piwik uses, including statements that are irrelevant to the update. Nor is the output particularly easy to read.

@diosmosis diosmosis mentioned this issue Mar 6, 2015
Merged
2 tasks
@mattab mattab mentioned this issue Mar 6, 2015
Closed
diosmosis added a commit that referenced this issue Mar 17, 2015
@diosmosis
Merge pull request #7333 from piwik/7276_update_command_progress
f3d61fc 
Fixes #7276 display update progress in core:archive command. Includes Updater refactor: deprecate static methods in Updates in favor of instance methods that are passed an Updater instance, use instance methods in Updater, move methods in CoreUpdater that perform updates to Updater class, add observer class that can be used within Updater to hook into single updater run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@diosmosis diosmosis

Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Milestone
2.12.0
Development

No branches or pull requests
5 participants
@diosmosis @mattab @futureweb @samgabriel @gaumondp