An example package.json:

{
  "name": "piwik",
  "version": "1.0.0",
  "description": "the leading free/libre analytics platform",
  "main": "piwik.js",
  "directories": {
    "test": "tests"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/piwik/piwik.git"
  },
  "author": "Piwik.org <hello@piwik.org>",
  "license": "GPL-3.0+",
  "bugs": {
    "url": "https://github.com/piwik/piwik/issues"
  },
  "dependencies": {
    "jquery-ui": "1.10.4",
    "jquery": "~2.2.0",
    "angular": "~1.6.0",
    "angular-sanitize": "~1.6.0",
    "angular-animate": "~1.6.0",
    "angular-cookies": "~1.6.0",
    "angular-mocks": "~1.6.0",
    "ng-dialog": "~1.3.0",
    "html5shiv": "~3.7.0",
    "mousetrap": "~1.4.0",
    "sprintf-js": "~1.0.0",
    "jscrollpane": "~2.0.0",
    "jquery-mousewheel": "~3.1.12",
    "jquery-placeholder": "~2.0.7",
    "jquery.dotdotdot": "~1.7.2",
    "jquery.scrollto": "~1.4.13",
    "chroma-js": "~0.6.0",
    "visibilityjs": "~1.2.1"
  },
  "keywords": [
    "piwik",
    "web",
    "analytics"
  ],
  "homepage": "http://piwik.org"
}

➜  ~/public_html/piwik git:(npm) ✗ npm outdated 
Package             Current  Wanted  Latest  Location
chroma-js             0.6.3   0.6.3   1.3.4  piwik
jquery                2.2.4   2.2.4   3.2.1  piwik
jquery-placeholder    2.0.7   2.0.7   2.3.1  piwik
jquery-ui            1.10.4  1.10.4  1.12.1  piwik
jquery.dotdotdot      1.7.4   1.7.4   3.0.1  piwik
jquery.scrollto      1.4.14  1.4.14   2.1.2  piwik
mousetrap             1.4.6   1.4.6   1.6.1  piwik
ng-dialog             1.3.0   1.3.0   1.4.0  piwik
sprintf-js            1.0.3   1.0.3   1.1.1  piwik

In my opinion keeping all third-party-libraries checked into the main repository is a bad idea.

FYI the reason we put all JS files in the repository is so that people can easily use Piwik from GIT: https://piwik.org/faq/how-to-install/faq_18271/
Sure, we could change it and ask people to run the NPM command, but then it would add some new requirements for them, plus it would break any existing Piwik deployed from Git (which there are hundreds).

It makes it hard to see which libraries are outdated and have bugs, which may be already fixed upstream

There are services that read the bower.json and composer files and report any outdated library, so far we have tried this tool: https://gemnasium.com/piwik/piwik (currently doesn't load but may be temporary issue).
There are quite a few other such services; https://alternativeto.net/software/requires-io/ - I would rather use the services to solve this particular challenge of being notified when there are new version of libraries.

If it helps to maintain and easily update the libs, I wouldn't mind using npm or similar to maintain all (non php) dependencies. We still could commit those libraries to git, so it can be directly used.

I agree that this would be a breaking change and probably should be made with Piwik 4.
I am also definitly for using a service like gemnasium. But they would work even better with npm as they are just a nice visualisation of npm outdated (or bower list and composer outdated at the moment)

But I am still quite against checking the resulting node_modules folder into the main git repository. I think it will result in many large commits changing thousands of lines every time angular (or another library) has a bugfix release.

I'd really recommend to do it the same way as PHP dependecies in composer (run npm install on tests and build script, do updates similar to #11780, still remove unwanted directories in the build script

Another idea would be adding the libraries in /tests/libs as dev-dependecies in the package.json.
This way they could be managed the same way, but can be excluded from the final build with npm install --only=production

UPDATE: I feel like this folder needs a cleanup equally desperately: The java-folder holds 7-9 year old java files without a license or context. And screenshot-testing has an empty node_modules folder

npm is famous for sending a crazy amount of http requests over the network (not sure if this recently improved but it was terrible), so I wouldn't trust this platform, when it comes to quickly release a security release, or forcing users to use npm on their server to use Piwik from git. Composer is a bit different because it's leaner and PHP based, which our users already support for sure.

+1 definitely to switch to npm vs bower (if you conclude it's better), but would still check in the files in git

UPDATE: I feel like this folder needs a cleanup equally desperately:

Yes, you're most welcome to remove anything un-used there 👍 (we iterated a lot in tests over the last 4 years and definitely a few things could be cleaned up)

While npm is still not the fastest package manager it definitly improved. And one factor is that we are mostly using flat dependecies (e.g. jquery) instead of fetching (server-side) node-modules which hundreds of dependencies each.
When disabling every cache, the package.json from above takes 2.3 seconds to install.

There are services that read the bower.json and composer files and report any outdated library, so far we have tried this tool: https://gemnasium.com/piwik/piwik (currently doesn't load but may be temporary issue).

Seems like github now also offers this functionality out of the box (even though it seems a bit like work in progress)

https://github.com/piwik/piwik/network/dependencies

(More Github News)

(slightly out of topic, this article about NPM is a really interesting read: I’m harvesting credit card numbers and passwords from your site. Here’s how.)

important note: let's not upgrade Jquery library. Because Matomo for Wordpress​​​ uses the Jquery provided by WP which is old. Only when WordPress upgrades their Jquery then we can upgrade ours. Sounds funny but unfortunately that's the case for now.

refs https://core.trac.wordpress.org/ticket/37110

important note: let's not upgrade Jquery library. Because Matomo for Wordpress​​​ uses the Jquery provided by WP

@mattab Could you expand on that please? I fail to understand what wordpress is doing here.

What could be the reason to ship vulnerable and outdated code to your users?

npm is famous for sending a crazy amount of http requests over the network (not sure if this recently improved but it was terrible), so I wouldn't trust this platform when it comes to quickly release a security release

Again, I fail to understand your point of view here. How is shipping libraries that are never updated a good thing? Look into this directory: https://github.com/matomo-org/matomo/tree/4.x-dev/libs/bower_components The last update time is in YEARS. Years for a javascript library is a lot.

If you don't like npm you can use yarn, but staying like that doesn't seem like a good idea.

forcing users to use npm on their server to use Piwik

Well, in 2020 you can be pretty sure that users that don't use Docker will have node/npm installed, so asking them to do npm i after a git pull is really not a big deal. When you do a web project, you have javascript dependencies, that's how it is. And it's best to use the tools designed for managing them instead of staying blocked in the past.

+1 definitely to switch to npm vs bower (if you conclude it's better), but would still check in the files in git

No! Why do that? This is what a politician would do! Only half-measure instead of eradicating the problem entirely!

Use npm, then use webpack or parcel to bundle the dependencies into nice bundles (not git tracked!) and voilà !

I would strongly recommand to tackle this issue. I think we can all agree that vendored dependencies should not be kept in the repo. And having a package-lock.json (or yarn.lock) also allows github and other tools to scan for vulnerabilities in the versions.

The change of name from piwik to matomo is a great opportunity to introduce such changes and improve dramatically the codebase.

Anyway, I'm just giving my opinion on the matter and at the end of the day you do what you think is best. I wanted to maybe try and contribute to this project, but when I saw the .bowerrc and the libs folder I got scared. Then I ran a static code analyzer for PHP and got discouraged by the huge number of issues in the codebase (we're talking 1526 errors and 5167 other issues found just for core/).

Finally, my tone might seem harsh and I'm sorry if you read it like that, I'm just trying to participate to the discussion and improve this project! :)

Cheers,
~Nico

@NicolasCARPi
Keep in mind that you are arguing with a three year old comment. In the meantime the work to use npm in Matomo has been ongoing for quite a while and will be finished with Matomo 4.0 #14082
So Matomo 4 will use the latest libraries.

No! Why do that?

This one I agree with you. In my opinion we should not be committing the files, but running npm in the build script of the matomo.zip just like we do with composer. Requiring all Matomo developers to have npm and node isn't really that much to require (and all php software projects already do it)

asking them to do npm i after a git pull is really not a big deal

And here I totally disagree. I don't have any statistics to prove this, but I am very sure that at leat 90% of all Matomo instances run on servers without access to node/npm or even a php-cli. Because they are running on PHP shared hosters or other kind of servers without any access apart from FTP. And while I also would prefer if every Matomo user knew what a git pull is or were familiar with the command line, this isn't the world we live in and I have to accept that Matomo wouldn't be so hugely popular right now if it wasn't that easy to set up for everyone.

So we can't run any nodejs on the Matomo instance itself, but shipping libraries installed with npm in the matomo.zip should be possible.

One thing that also makes things harder (especially considering webpack) is that every Matomo plugin can add custom JS and CSS (and even LESS overwriting variables) which makes using webpack as a build step for the matomo.zip hard.
It isn't impossible (Nextcloud is doing exactly this: running webpack during build of core and all plugins and then shipping the built zip files), but it is really non-trivial and they have a lot more people working on it.

I got scared.

I get this and to feel better I check that no library used has a reported security issue affecting Matomo.

@mattab Could you expand on that please? I fail to understand what wordpress is doing here.

This is another issue. There is a official Matomo for Wordpress plugin that "simply" ships Matomo inside a wordpress plugin.
But Wordpress requires for all plugins that they don't include any js libraries that are already included in Wordpress core (which seems reasonable in case of a vulnerability).

Unfortunately Wordpress is stuck in an even larger backwards compatibility loop which forces them to ship an ancient jquery version but changing this would mean breaking millions of websites and plugins, so they won't do it anytime soon.

I don't have a solution for this as staying with an ancient version isn't really a solution and using a normal version means that the plugin can't be published.

One last thing: In case you know an actual security issue that affects Matomo because of some outdated library, please contact us: https://matomo.org/security/

Thank you @Findus23 for your reply, I'm glad the work towards NPM has already started! :)

Shipping an already complete .zip archive is indeed a viable solution for ease of use.

~Nico

PS:

Keep in mind that you are arguing with a three year old comment

That doesn't make it invalid in my book ;)