Correction: add step 2.

Steps to Reproduce (for Bugs)

1. Get a database from the 3.13.5 release.
2. Download matomo 4.2.1 piwik.zip file, unzip into /var/www/piwik.
3. Copy some obsolete files into the /var/www/piwik/plugins directory.

Hi @satphil Thanks for creating the issue. I'm quite sure we already had a similar report in the past, but wasn't able to find it on the fly.
The colum location_region actually should have already been there before Matomo 4. Do you maybe have an backup from before the update where you could check if that column was there before or not?

Hi @satphil, thanks for creating this issue!

I can confirm the faq on the three step upgrade omits the requirement to remove no longer used files. In the OneClickUpdater we explicitly do this (for the reasons described in the issue description): https://github.com/matomo-org/matomo/blob/4.x-dev/plugins/CoreUpdater/Model.php#L26-L36, the manual upgrade should mirror this.

I'm not sure what the best solution here is since we don't keep a list of files that were removed but diff between what is in the archive and the current install, but it might require a new core command. Or some pre tested bash functions. But the process will become more complicated since it's no longer just a "FTP and upload".

" It is not needed to delete your old Matomo files before transferring the new ones. The new files should overwrite all the old files with the same names."

The initial files should be removed before uploading the new files and the guide should be updated ideally. There has been a similar issue before (I think eg #17181). Otherwise there might be still some dimensions that should be removed etc.

This is very disturbing.

Please either:

• make core:update clean obsolete files (core application + shipped plugins)
• update documentation to state this needs to be done manually
  Also consider updating https://matomo.org/changelog/matomo-4-0-0/ to make an explicit statement about it.

While the former would involve way more work but would eventually be way more rewarding (keeping the promise of simplicity), the latter could be done swiftly.

@Berbe we are already planning to improve it for the next release.

I went to upgrade recently and was wondering why everything was failing. Only after hunting down this issue was I able to find that I needed to remove all the old files first. The documentation still says to just copy and override existing files. This should really be at least changed in the documentation.

@Berbe 2021-04-21
Please either:

• [...]
• update documentation to state this needs to be done manually
  Also consider updating https://matomo.org/changelog/matomo-4-0-0/ to make an explicit statement about it.

[...] the latter could be done swiftly.

@sgiehl 2021-04-21
@Berbe we are already planning to improve it for the next release.

@mattab removed this from the 4.3.0 milestone on 26 May
@mattab added this to the 4.4.0 milestone on 26 May
@tsteur removed this from the 4.4.0 milestone on 23 Jul
@tsteur added this to the 4.7.0 milestone on 23 Jul

@tsteur would you mind to summarize what we should do here? Is it only about updating the documentation? Or shall we also implement a possibility to automatically remove those files in the UI or maybe within core:update?

The first step would be definitely to update the guide.

Not sure what else we can do to solve this issue in the future so it won't happen again? Can we do something that eg when files are left there, the updates still work?

Maybe we can detect files that shouldn't be there and ignore them? Or does it maybe make sense before the DB upgrade to show again if some files need to be deleted?

Or maybe we could detect when this error happens and give users clear instructions on what to do solve this issue? Like remove these files and try again?

I'm assuming there are various ways but I'm not too deep into it right now to know how to best solve the issue without taking too much time and while providing a good user experience.

The matomo software implicitly depends on the presence or absence of files, so it is imperative that obsolete files are removed during upgrades. Commands like rsync have switches like "--delete-after - receiver deletes extraneous files from dest dirs after transfer, not during".
One approach, mentioned in my original bug report: save the config/config.ini.php file, delete the old piwik directory, copy over the contents of the new piwik.zip file and then return the saved config/config.ini.php file.

Deleting the directory and then copying back the config.ini.php file works well, but could in some cases could potentially change existing directory permissions/owner and delete custom .htaccess rules.

An alternative could be to provide a command to delete all files in the Matomo installation directory except the config.ini.php file and any .htaccess files (if they exist):

$find . ! -name .htaccess ! -name config.ini.php -type f -delete

Proposed documentation change:

It is not needed to delete your old Matomo files before transferring the new ones. The new files should overwrite all the old files with the same names

It is recommended to delete your old Matomo files before transferring the new ones, be sure to preserve config/config.ini.php and any .htaccess files.

# go into the the directory that contains your existing "matomo/" installation 

# download latest version
$ wget https://builds.matomo.org/matomo.zip

# remove old files
$find . ! -name .htaccess ! -name config.ini.php -type f -delete

# extract the new release files
$ unzip -o matomo.zip # extract it to "matomo/" directory

# delete the archive and read below the next step
$ rm matomo.zip 

@bx80 Feedback: could we update instructions to account for:

• when Matomo setup in multi tenant mode as documented in https://developer.matomo.org/guides/multi-tenants then would these instruction also work? (I think it might keep the directories & files eg. $matomoDir/misc/user/subdomain_a.mydomain.com/config.ini.php
• common.config.ini.php file that can be used to refactor settings in multi tenant https://matomo.org/faq/new-to-piwik/faq_137/#recommended-matomo-settings-via-common-config-ini-php or https://matomo.org/faq/on-premise/how-to-deploy-same-code-base-on-all-the-servers-in-multi-server-environment/

@bx80 in addition to config.ini.php and .htaccess files we should also keep:

• misc directory, as it might contain a geoip database, that would be lost otherwise
• config.php file as it might contain some custom di configuration
• Any plugin that is not part of the Matomo package

Updated to ignore everything in the misc/ directory, common.config.ini.php and config.php.

Only deleting plugins provided by the release isn't as straight forward as we don't have a naming system or directory structure to identify them. I can make the main delete command skip the plugins directory and then add a second command to just delete files in a fixed list of plugins directories. This does make for a large command to paste and will need to be updated in the future if we add new plugins to the release, but it will preserve custom plugins. Open to any better ideas!

# remove old files, excluding all plugins
find . ! -name .htaccess ! -name config.ini.php ! -name config.php ! -name common.config.ini.php ! -path "./misc/*" \ 
! -path "./plugins/*" -type f -delete

# remove old files from plugins that are part of the main matomo release
find plugins/API plugins/Actions plugins/Annotations plugins/BulkTracking \
plugins/Contents plugins/CoreAdminHome plugins/CoreConsole plugins/CoreHome \
plugins/CorePluginsAdmin plugins/CoreUpdater plugins/CoreVisualizations \
plugins/CoreVue plugins/CustomDimensions plugins/CustomJsTracker \
plugins/DBStats plugins/Dashboard plugins/DevicePlugins \
plugins/DevicesDetection plugins/Diagnostics \
plugins/Ecommerce plugins/Events plugins/Feedback plugins/GeoIp2 plugins/Goals \
plugins/Heartbeat plugins/ImageGraph plugins/Insights plugins/Installation \
plugins/Intl plugins/IntranetMeasurable plugins/LanguagesManager plugins/Live \
plugins/Login plugins/Marketplace plugins/MobileAppMeasureable \
plugins/MobileMessages plugins/Monolog plugins/Morpheus plugins/MultiSites \
plugins/Overlay plugins/PagePerformance plugins/PrivacyManager \
plugins/ProfessionalServices plugins/Proxy plugins/Referrers \
plugins/Resolution plugins/RssWidget plugins/SEO plugins/ScheduledReports \
plugins/SegmentEditor plugins/SitesManager plugins/TagManager plugins/Tour \
plugins/Transitions plugins/TwoFactorAuth plugins/UserCountry \
plugins/UserCountryMap plugins/UserId plugins/UserLanguage \
plugins/UsersManager plugins/VisitFrequency plugins/VisitTime \
plugins/VisitorInterest plugins/VisitsSummary plugins/WebsiteMeasurable \
plugins/Widgetize -type f -delete

The only other way i can think of, is that we could provide a command that would output this text and dynamically generate it so that we don't have to maintain the page up to date in the future. Would it maybe make sense to have a console command to generate these shell commands?

Or they could run diagnostics:run command after updating files and then delete any extra file that shouldn't be there? Maybe it's the easiest and works across OS etc?

Like after updating files you need to run the ./console diagnostics:run command to identify which files need to be deleted....

Thanks for the ideas, a console command sounds like a much nicer approach than copying shell commands 👍

Getting the list of unexpected files from the diagnostics FileIntegrity class would be good as it will always be up to date based on the release manifest. However there is a twenty line cut off to the list of unexpected files shown on the diagnostic report and the output is formatted which would make manually deleting multiple files from the command line a chore.

To improve the experience I've linked a PR which adds two extra options to the diagnostic command:

./console diagnostics:run --show-unexpected-files which will just output a raw list of unexpected file names, this could be piped to a file, less or directly to the OS delete command (eg. ./console diagnostics:run --show-unexpected-file | xargs rm -f)

./console diagnostics:run --delete-unexpected-files which will attempt to delete the unexpected files using the Filesystem::deleteFileIfExists method.

The documentation can then be updated to something like:
Run the ./console diagnostics:run --show-unexpected-files command and review the list of unexpected files.
Run the ./console diagnostics:run --delete-unexpected-files to delete the files.

Once the linked PR has passed review and has been released, the manual three step update FAQ can be updated as follows:

### 2) Replace the Matomo files with the latest version

You can download the latest Matomo release from [builds.matomo.org/matomo.zip][3]. After downloading it, 
you can extract it to your computer.

If you upload these files using FTP to your Matomo server, please make sure all files are transferred in 
binary mode. 

If you have SSH access to your server you can download the latest version and extract it directly from the 
server hosting Matomo:

    # go into the the directory that contains your existing "matomo/" installation 
    $ wget https://builds.matomo.org/matomo.zip # download latest version
    $ unzip -o matomo.zip # extract it to "matomo/" directory
    $ rm matomo.zip # delete the archive
    $ ./console diagnostics:run --show-unexpected-files # check for any obsolete files, Matomo 4.13.2+ only
    $ ./console diagnostics:run --delete-unexpected-files # delete any obsolete files, Matomo 4.13.2+ only
    # read below for the next step

Thanks for the feedback @satphil, a different filesystem hierarchy would certainly make things easier for this issue 👍

Unfortunately at this point I think the rework involved would significantly outweigh the eventual benefits. Code, build scripts, test scripts and documentation would all have to be changed and retested. Rearranging the directory structure on all existing Matomo installations could then break security rules, permissions and scripts, most of which would require manual intervention to fix.

It might be less disruptive to split user provided plugins from Matomo release provided plugins at some point, e.g. matomo/plugins vs matomo/plugins_release but would still be a lot of work.

thank you all for the work on this. we were upgrading from 3.13.5 and had issues with the files we had left hanging around.

Executing ./console --matomo-domain="xxxx" plugin:uninstall "ExampleTheme"... PHP Fatal error:  Access level to Piwik\Plugins\CustomVariables\Columns\CustomVariableName::configureSegments() must be public (as in class Piwik\Columns\Dimension) in /xxx/plugins/CustomVariables/Columns/CustomVariableName.php on line 15

cheers!

The guide at https://matomo.org/docs/update/#the-manual-three-step-update still reads as:

It is not needed to delete your old Matomo files before transferring the new ones. The new files should overwrite all the old files with the same names.

Thanks for heads up @kosarko, I've now updated the guide 👍