Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #7276 display update progress in core:archive command (includes Updater refactor) #7333

Merged
merged 27 commits into from Mar 17, 2015

Conversation

diosmosis
Copy link
Member

Non-intrusively injects logic into the update process so the core:archive command will print out the current query being executed and the amount of queries left to run. Refactored Updater class + Columns\Updater class to use instance methods where possible, changed the methods of Updates.php base so Updater instance is passed to the descendants (preserved old method for BC). Added UpdateObserver class that can be used to inject logic into the update process. And moved static methods in CoreUpdater.php to Updater class.

Fixes #7276

TODO:

  • revise UpdaterTest.php integration/unit tests (improve when possible)
  • figure out why translations not working in command

Command output looks like:

    *** CoreUpdater_UpdateTitle ***

    CoreUpdater_DatabaseUpgradeRequired

    CoreUpdater_YourDatabaseIsOutOfDate

    CoreUpdater_PiwikWillBeUpgradedFromVersionXToVersionY

    CoreUpdater_TheFollowingPluginsWillBeUpgradedX

    CoreUpdater_TheFollowingDimensionsWillBeUpgradedX

    *** Note: this is a Dry Run ***

    [redacted]

    *** End of Dry Run ***

�[33mA database upgrade is required. Execute update? (y/N) �[39m
Starting the database upgrade process now. This may take a while, so please be patient.

    *** CoreUpdater_UpdateTitle ***

    CoreUpdater_DatabaseUpgradeRequired

    CoreUpdater_YourDatabaseIsOutOfDate

    CoreUpdater_PiwikWillBeUpgradedFromVersionXToVersionY

    CoreUpdater_TheFollowingPluginsWillBeUpgradedX

    CoreUpdater_TheFollowingDimensionsWillBeUpgradedX

    CoreUpdater_TheUpgradeProcessMayTakeAWhilePleaseBePatient

  Executing �[33mALTER TABLE `piwik_log_visit` ADD COLUMN `config_browser_engine` VARCHAR(10) NOT NULL�[39m... Done. �[32m[1 / 81]�[39m
  Executing �[33mUPDATE piwik_log_visit SET `config_browser_engine` = IF (`config_browser_name` IN ('OP'), 'Presto', IF (`config_browser_name` IN ('SF','CH','OW','AR','EP','FL','WO','AB','IR','CS','FD','HA','MI','GE','DF','BB','BP','TI','CF','RK','B2','NF'), 'WebKit', IF (`config_browser_name` IN ('KO'), 'KHTML', IF (`config_browser_name` IN ('NS','PX','FF','FB','CA','GA','KM','MO','SM','CO','FE','KP','KZ','TB'), 'Gecko', IF (`config_browser_name` IN ('IE'), 'Trident', '')))))�[39m... Done. �[32m[2 / 81]�[39m
  Executing �[33mUPDATE IGNORE piwik_archive_blob_2015_01 SET name='DevicesDetection_browserEngines' WHERE name = 'UserSettings_browserType'�[39m... Done. �[32m[3 / 81]�[39m
  Executing �[33mUPDATE IGNORE piwik_archive_blob_2015_02 SET name='DevicesDetection_browserEngines' WHERE name = 'UserSettings_browserType'�[39m... Done. �[32m[4 / 81]�[39m
  Executing �[33mUPDATE IGNORE piwik_archive_blob_2015_03 SET name='DevicesDetection_browserEngines' WHERE name = 'UserSettings_browserType'�[39m... Done. �[32m[5 / 81]�[39m
  Executing �[33mCREATE TABLE `piwik_sequence` (
                `name` VARCHAR(120) NOT NULL,
                `value` BIGINT(20) UNSIGNED NOT NULL,
                PRIMARY KEY(`name`)
        ) ENGINE=InnoDB DEFAULT CHARSET=utf8�[39m... Done. �[32m[6 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2010_01', 502)�[39m... Done. �[32m[7 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2010_12', 508)�[39m... Done. �[32m[8 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_01', 576)�[39m... Done. �[32m[9 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_02', 566)�[39m... Done. �[32m[10 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_03', 572)�[39m... Done. �[32m[11 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_04', 570)�[39m... Done. �[32m[12 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_05', 574)�[39m... Done. �[32m[13 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_06', 570)�[39m... Done. �[32m[14 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_07', 572)�[39m... Done. �[32m[15 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_08', 574)�[39m... Done. �[32m[16 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_09', 570)�[39m... Done. �[32m[17 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_10', 574)�[39m... Done. �[32m[18 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_11', 570)�[39m... Done. �[32m[19 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2011_12', 572)�[39m... Done. �[32m[20 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_01', 576)�[39m... Done. �[32m[21 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_02', 568)�[39m... Done. �[32m[22 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_03', 572)�[39m... Done. �[32m[23 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_04', 572)�[39m... Done. �[32m[24 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_05', 572)�[39m... Done. �[32m[25 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_06', 570)�[39m... Done. �[32m[26 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_07', 574)�[39m... Done. �[32m[27 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_08', 572)�[39m... Done. �[32m[28 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_09', 570)�[39m... Done. �[32m[29 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_10', 574)�[39m... Done. �[32m[30 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_11', 570)�[39m... Done. �[32m[31 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2012_12', 574)�[39m... Done. �[32m[32 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_01', 574)�[39m... Done. �[32m[33 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_02', 566)�[39m... Done. �[32m[34 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_03', 572)�[39m... Done. �[32m[35 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_04', 572)�[39m... Done. �[32m[36 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_05', 572)�[39m... Done. �[32m[37 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_06', 570)�[39m... Done. �[32m[38 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_07', 574)�[39m... Done. �[32m[39 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_08', 572)�[39m... Done. �[32m[40 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_09', 572)�[39m... Done. �[32m[41 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_10', 572)�[39m... Done. �[32m[42 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_11', 570)�[39m... Done. �[32m[43 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2013_12', 574)�[39m... Done. �[32m[44 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_01', 574)�[39m... Done. �[32m[45 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_02', 566)�[39m... Done. �[32m[46 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_03', 574)�[39m... Done. �[32m[47 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_04', 570)�[39m... Done. �[32m[48 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_05', 572)�[39m... Done. �[32m[49 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_06', 572)�[39m... Done. �[32m[50 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_07', 572)�[39m... Done. �[32m[51 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_08', 572)�[39m... Done. �[32m[52 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_09', 572)�[39m... Done. �[32m[53 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_10', 572)�[39m... Done. �[32m[54 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_11', 570)�[39m... Done. �[32m[55 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2014_12', 574)�[39m... Done. �[32m[56 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2015_01', 574)�[39m... Done. �[32m[57 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2015_02', 566)�[39m... Done. �[32m[58 / 81]�[39m
  Executing �[33mINSERT INTO piwik_sequence (name, value) VALUES ('piwik_archive_numeric_2015_03', 514)�[39m... Done. �[32m[59 / 81]�[39m
  Executing �[33m# ATTENTION: This update script will execute some more SQL queries than that below as it is necessary to rebuilt some archives #�[39m... Done. �[32m[60 / 81]�[39m
  Executing �[33mUPDATE piwik_report SET reports = REPLACE(reports, 'UserSettings_getBrowserVersion', 'DevicesDetection_getBrowserVersions')�[39m... Done. �[32m[61 / 81]�[39m
  Executing �[33mUPDATE piwik_report SET reports = REPLACE(reports, 'UserSettings_getBrowser', 'DevicesDetection_getBrowsers')�[39m... Done. �[32m[62 / 81]�[39m
  Executing �[33mUPDATE piwik_report SET reports = REPLACE(reports, 'UserSettings_getOSFamily', 'DevicesDetection_getOsFamilies')�[39m... Done. �[32m[63 / 81]�[39m
  Executing �[33mUPDATE piwik_report SET reports = REPLACE(reports, 'UserSettings_getOS', 'DevicesDetection_getOsVersions')�[39m... Done. �[32m[64 / 81]�[39m
  Executing �[33mUPDATE piwik_report SET reports = REPLACE(reports, 'UserSettings_getMobileVsDesktop', 'DevicesDetection_getType')�[39m... Done. �[32m[65 / 81]�[39m
  Executing �[33mUPDATE piwik_report SET reports = REPLACE(reports, 'UserSettings_getBrowserType', 'DevicesDetection_getBrowserEngines')�[39m... Done. �[32m[66 / 81]�[39m
  Executing �[33mUPDATE piwik_report SET reports = REPLACE(reports, 'UserSettings_getWideScreen', 'UserSettings_getScreenType')�[39m... Done. �[32m[67 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_01 SET name = 'Resolution_resolution' WHERE name = 'UserSettings_resolution'�[39m... Done. �[32m[68 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_01 SET name = 'Resolution_configuration' WHERE name = 'UserSettings_configuration'�[39m... Done. �[32m[69 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_02 SET name = 'Resolution_resolution' WHERE name = 'UserSettings_resolution'�[39m... Done. �[32m[70 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_02 SET name = 'Resolution_configuration' WHERE name = 'UserSettings_configuration'�[39m... Done. �[32m[71 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_03 SET name = 'Resolution_resolution' WHERE name = 'UserSettings_resolution'�[39m... Done. �[32m[72 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_03 SET name = 'Resolution_configuration' WHERE name = 'UserSettings_configuration'�[39m... Done. �[32m[73 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_01 SET name = 'DevicePlugins_plugin' WHERE name = 'UserSettings_plugin'�[39m... Done. �[32m[74 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_02 SET name = 'DevicePlugins_plugin' WHERE name = 'UserSettings_plugin'�[39m... Done. �[32m[75 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_03 SET name = 'DevicePlugins_plugin' WHERE name = 'UserSettings_plugin'�[39m... Done. �[32m[76 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_01 SET name = 'UserLanguage_language' WHERE name = 'UserSettings_language'�[39m... Done. �[32m[77 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_02 SET name = 'UserLanguage_language' WHERE name = 'UserSettings_language'�[39m... Done. �[32m[78 / 81]�[39m
  Executing �[33mUPDATE piwik_archive_blob_2015_03 SET name = 'UserLanguage_language' WHERE name = 'UserSettings_language'�[39m... Done. �[32m[79 / 81]�[39m
  Executing �[33mALTER TABLE `piwik_log_visit`
                CHANGE `config_os_version` `config_os_version`  VARCHAR( 100 ) DEFAULT NULL,
                CHANGE `config_device_type` `config_device_type`  VARCHAR( 100 ) DEFAULT NULL�[39m... Done. �[32m[80 / 81]�[39m
  Executing �[33mALTER TABLE `piwik_log_link_visit_action` MODIFY COLUMN `idaction_content_interaction` INTEGER(10) UNSIGNED DEFAULT NULL, MODIFY COLUMN `idaction_content_name` INTEGER(10) UNSIGNED DEFAULT NULL, MODIFY COLUMN `idaction_content_piece` INTEGER(10) UNSIGNED DEFAULT NULL, MODIFY COLUMN `idaction_content_target` INTEGER(10) UNSIGNED DEFAULT NULL, MODIFY COLUMN `bandwidth` BIGINT(15) UNSIGNED DEFAULT NULL�[39m... Done. �[32m[81 / 81]�[39m

�[32m****************************************�[39m
  Piwik has been successfully updated!  
�[32m****************************************�[39m

diosmosis added 8 commits February 24, 2015 19:36
…(moved CoreUpdater static functions to core/Updater class, remove use of static methods in Updater, don't use static method setUpdater in ColumnsUpdater).
…er in Columns\Updater and document methods in Updater.
…er update process and document UpdateListener.
…notice errors caused by change to Updates.php and add some more functionality to ConsoleCommandTestCase base class.
… to show progress of update. Also remove CLI output code from CoreUpdater\Controller::runUpdaterAndExit.
@diosmosis diosmosis added Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. Needs Review PRs that need a code review labels Mar 3, 2015
@diosmosis diosmosis added this to the Piwik 2.12.0 milestone Mar 3, 2015
@mnapoli
Copy link
Contributor

mnapoli commented Mar 3, 2015

Instead of reimplementing an event listener (for the "updater observer") would it make sense to reuse the event dispatcher instead?

@diosmosis
Copy link
Member Author

Do you mean create a new EventDispatcher for the Updater instance and post events to the member dispatcher? Or use Piwik::postEvent/Piwik::addAction?

@mnapoli
Copy link
Contributor

mnapoli commented Mar 4, 2015

I meant use Piwik::postEvent with either Piwik::addAction or the regular way (getListHooksRegistered())? I don't see a reason to create a separate event dispatcher.

@diosmosis
Copy link
Member Author

Piwik::postEvent will post to every plugin, and the "events" executed by the command should only be executed for a single Updater run (ie during the execution of the command, and nowhere else). Also, Piwik::addAction additions cannot be removed, and the Updater is not a singleton service (at the moment) so posting global events from it would be unseemly.

Note: Though this pull request has an 'observer' which seems akin to event listener, the intent is not to add an event dispatcher. The purpose of an event dispatcher is to provide extensibility, ie, the ability to create extensions. The purpose of the observer in this pull request is to allow Aspect Oriented Programming, so cross-cutting concerns can be added/removed non-intrusively on a case-by-case basis.

@mattab mattab added the not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. label Mar 6, 2015
@mattab
Copy link
Member

mattab commented Mar 6, 2015

(adding not-in-changelog label)

diosmosis added 4 commits March 10, 2015 00:35
…php methods instance methods, create Update instances via DI, make Columns\Updater use instance methods instead of static, and add integration test for Columns\Updater.
@diosmosis diosmosis removed the Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. label Mar 10, 2015
@diosmosis
Copy link
Member Author

@piwik/core-team I'm done w/ this change. Since it includes modifications to the Updates.php + descendents API, it should be reviewed. No BC breaking changes.

@@ -237,7 +236,7 @@ public static function wasDimensionMovedFromCoreToPlugin($name, $version)
'log_visit.config_realplayer' => 'TINYINT(1) NOT NULL',
'log_visit.config_device_type' => 'TINYINT( 100 ) NULL DEFAULT NULL',
'log_visit.visitor_localtime' => 'TIME NOT NULL',
'log_visit.location_region' => 'char(2) DEFAULT NULL1',
'log_visit.location_region' => 'char(2) DEFAULT NULL1', // TODO: some of these have "NULL1", is this a bug?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tsteur can you confirm whether it is expected that some of these have NULL1 ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, confirmed this already a few months ago. It's expected

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you remember why they are set to NULL1? Would be useful in a comment in the code.

@mattab
Copy link
Member

mattab commented Mar 12, 2015

Nice PR! I put some feedback in line.

also, I found a bug which is related, but i guess it's not a regression of this Pull request. But because it's related it makes sense to solve it in here.

How to reproduce a critical error (i guess there are way simpler technique but I did it this way)

update failed but returns ok

Issues to address

  • there are some \n\n displayed in the CLI output -> these should be new lines
  • the CLI exits with success code, but it should return with error code (this is important)
$ echo $?
0

Expected to get: 1 (error)

  • the text says "Piwik has been successfully updated!" but it should say an error instead
  • let's add a test case for this "Failed update process" case to ensure it doesn't regress in the future (check output code is 1 and that success message is not displayed)

@diosmosis
Copy link
Member Author

@mattab Addressed your comments (except for NULL1, will look for something to put as a comment later).

@mattab
Copy link
Member

mattab commented Mar 16, 2015

Looks good to me

(needs a rebase + you can merge it)

diosmosis added a commit that referenced this pull request Mar 17, 2015
Fixes #7276 display update progress in core:archive command. Includes Updater refactor: deprecate static methods in Updates in favor of instance methods that are passed an Updater instance, use instance methods in Updater, move methods in CoreUpdater that perform updates to Updater class, add observer class that can be used within Updater to hook into single updater run.
@diosmosis diosmosis merged commit f3d61fc into master Mar 17, 2015
@diosmosis diosmosis deleted the 7276_update_command_progress branch March 17, 2015 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs Review PRs that need a code review not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants