👍 I've got a "not needed right now" every time I bring that up but I'm sure it could help people debug problems (and help us understand their problems).

If it's easy to do then definitely +1 as it seems important to be able to view logs from the UI! please move to short term if can be done rather quickly 👍

edit: it would be great to let a user view the last core:archive output runs, as these are often useful to have (and hard to get!) when debugging archiving issues.

Adding tentatively to 2.15.1 - let's think of the scope for Log viewer for MVP. Having a log viewer in Piwik LTS would be really high value!

Will it be shipped with Piwik or a standalone plugin on the marketplace? I'm not sure if https://github.com/piwik/plugin-PiwikDebugger already has this feature but could add it there.

It's important to know whether we will ship it with Piwik etc to select the right library for it in case we can reuse one. Some are not really secure or might be hard to integrate re permissions etc with Piwik.

it's useful for people and support teams to have better troubleshooting mechanism directly in product so I would suggest to ship it with core.

I would have suggested to put it on the Marketplace as most will never need it and it's usually quickly installed if needed but if it's supposed to be core then ok

It's not supposed to, up to us whether to put it in core or not. if we try to keep core as small as possible it makes sense to put it outside on Marketplace

From my user perspective I say I would not mind having 200 KB of PHP code added to Core if it helps users already in despair...

• Is it universally needed ?
• Does it affect performance having it in Core ?
• Is the size of integrating it into Core is too heavy ?
• Does it needs to be tighten with the Core release itself ?
• Does having it into Core makes it more difficult to maintain or makes it impossible or too slow to evolve ?

Those are the questions we should ask before adding a plugin to Core. At this moment AFAIK, not everyone needs it but my guts feeling says otherwise. Builtin debug/diagnostic if light and has no impact on performance seems a good thing.

IMO, a log viewer would provide better diagnostic capabilities for core and every plugin that uses the logger. It's existence provides better resilience in case of failure in numerous plugins, w/o having to change any of those plugins (eg, it would make debugging LoginLdap much easier). It should therefore be included in core (as a plugin, of course).

That said, providing a pretty view for logs in the UI is not the point of this issue, the point is to provide tools to help both users debug their own installs and for sysadmins / us debug other peoples' installs. So being able to view or download logs for a specific time range and from a specific plugin would be useful, whereas being able to see logs in the Piwik UI alone would not be so useful.

Is it universally needed ?

Not really, most users will not need it and it will add "clutter" to the Piwik UI for many users. If one needs it, I'd say it's easy to explain to go to "Marketplace" and click "install" next to that plugin.

FYI: If it's shipped with core but not enabled by default, one has to click on "Plugins" and "activate" next to that plugin so it's not really easier. If it's enabled by default it adds clutter to the UI and needs more performance.

Does it affect performance having it in Core ?

Yes, maybe not much, but it does on every request (plugin loading, event manager, etc). Also we might need to change logging behaviour to log to files by default if we ship it with core, this can affect performance additionally as we're not using eg syslog.

Is the size of integrating it into Core is too heavy ?

Depends on the library we choose etc.

Does it needs to be tighten with the Core release itself ?

If we put it in core we cannot release new versions independently. Meaning after the 2.15.0 we will also not be able to add new features for most likely > 6 or 12 months for it.

Does having it into Core makes it more difficult to maintain or makes it impossible or too slow to evolve ?

No

Having logging/diagnostic capabilities in Piwik is important and useful. However, I think that it doesn't have to be in core directly. We developers tend to need these kinda things a lot and therefore maybe think it should be in core etc but I think it doesn't have to be for > 80% of the users. It's maybe different for companies that give support for Piwik etc. In these cases it can be still installed directly. I think the advantages of having it in the marketplace outweigh the advantages of having it in core but my world won't collapse if it's in core. Goal should be to have a small core and more things independent deployable so that we can fix bugs and add new features independently.

@tsteur , I was not "really" asking the question or even expecting answers but it's cool nonetheless. :)

Anyway I still think it's a good practice Core team ask itself those kind of questions every time someone comes up with suggestion and features requests. I know you already do this, I was just trying to be the devil's advocate and tried to fill your/ @mattab shoes.

Long live to Free Analytics !

Answered the questions as I thought about those kinda things upfront as you mentioned and wanted to explain myself why I think it doesn't have to be in core ;)

Cheers!

Also: Are we only talking about logs written by Piwik or optionally also logs from webserver like Apache, Nginx, ... etc?

Do we have some specific requirements for MVP?

Also: Are we only talking about logs written by Piwik or optionally also logs from webserver like Apache, Nginx, ... etc?

Only logs created by Piwik / monolog

Do we have some specific requirements for MVP?

not quite, but we keep is very simple - main requirement is: let super user browse app logs (that are stored in DB or in files, see faq) (maybe ability to filter by log level and/or date could be useful, eg. "show me only errors", "show me logs for today only")

OK, so far I found only http://pimpmylog.com/ (released under GPL as well) that seems to be usable but need to check whether we can build a custom authentication plugin/method for it

https://github.com/Syonix/monolog-viewer doesn't support multiple users

I had a log for some logger repositories:

• pimpmylog.com is rather built for webserver logs and seems to be built for files only. Searching/Showing logs for a specific data range is not possible. Couldn't find how to write an adapter for database etc. To display piwik log is possible though see screenshot (themable). I reckon it is maybe possibility to connect to Piwik database for authentication but not 100% sure.
  
• https://github.com/Gundars/HogLog only prints plain logs
• https://github.com/dperrymorrow/Fire-Log was not possible to reuse as built for CodeIgniter
• https://github.com/mikemand/logviewer (built for Laravel, there are more apps that were built for Laravel)
• https://github.com/Syonix/monolog-viewer only supports files so far but can be extended for different adapters from what I saw, code is messy but licensed under GPL so could adjust eg authentication. Requires PHP 5.4.

Releasing it as a plugin would allow us to require PHP 5.4 but I presume it should maybe work under PHP 5.3 too?

Maybe we have to build a custom solution unless someone has maybe an idea?

Releasing it as a plugin would allow us to require PHP 5.4 but I presume it should maybe work under PHP 5.3 too?

IMO it's fine if the plugin requires PHP 5.4

Had a more detailed look at code in https://github.com/Syonix/monolog-viewer and it's too simple. Eg it loads the whole file in memory etc and seems to not provide paging etc meaning it will take a load of memory and be slow etc.

I think we can maybe implement a simple solution ourselves for now even though I really would prefer to use something existing. If somebody knows anything please comment and we'll look into it.

If we develop it in a separate plugin we can add new features at any time and start with an MVP solution to see what we actually need over time

Search for dates will be probably hard to implement (or rather slow) since we have to loop through many things but will see.

In some cases to get all possible logging features it might be useful to write eg a loggly plugin etc. There are many great services out there if one wants to send this kinda data there. Haven't found any similar self-hosted solutions on https://github.com/Kickball/awesome-selfhosted

FYI: I'm working on a simple solution in https://github.com/piwik/plugin-LogViewer

As mentioned developed plugin in new repository https://github.com/piwik/plugin-LogViewer . Screenshots can be seen at https://github.com/piwik/plugin-LogViewer/tree/master/tests/UI/expected-ui-screenshots . It is possible to click eg on a requestId and the UI would automatically show all log messages belonging to that request. If one clicks on a date it shows all log messages of that date (same if one clicks on a severity or tag).

I wrote an adapter for file and database. One can always select them no matter if they are currently configured or not. Sometimes one only wants to enable it for a while and later browse through the logs. I implemented it memory friendly to make sure it works with huge files but it is slower as we're basically reading line by line. It's quite fast but if we do not find a result and have to iterate over all lines it becomes a bit slow.

I kept things simple for now. If we leave it as a standalone plugin we can add features and change things anytime and release new versions when needed.

If we leave it as a standalone plugin I'd add it as a submodule to Piwik. As mentioned I'd prefer to have it as a standalone plugin, eg because it's useless by default since we only log to screen anyway and people would need to change config first.

Could someone have a look and try it? @diosmosis @mattab

One bug: multi-line log entries (ie, exceptions w/ stack traces) are displayed incorrectly:

Also the columns might look better w/ vertical-align: top.

Another bug: if I filter by INFO, the exception stack traces disappear:

Same thing happens when I search for a string that is not in the stack trace, but is in the message.

Some suggestions (so you only have to add it if you want to ;)):

• Something that displays the current logger config would be useful I think. ie, seeing the log_level and log_writers settings might be nice. Some quick instructions (or a link to a FAQ) on setting up logging would also be useful IMO.
• A note that says you can click on the columns to filter would be useful too. It was easy to see when I saw the cursor over the cells, but some people might not realize they are clickable?
• A feature to download the currently displayed log files might be useful (perhaps archived). Would aid in sending logs to maintainers / developers.
• A refresh link/button might be useful? I think it would also be useful if the new log lines that appear after a refresh were highlighted in some way to differentiate from the old ones. (use case would be, eg, debugging loginldap: 1) open log viewer in new tab, 2) try to login w/ LDAP login, 3) refresh log viewer and see new logs).
• This is probably not achievable easily, but it would be great if we could change the log level just for the LogViewer page. Ie, selecting DEBUG in the LogViewer would display DEBUG logs, but they wouldn't get written to the database or file (so they would have to be written to a temporary location and then removed later).
• This is more of a change to core, but we should probably log the current DI environment as well (ie, tracker, cli, web (doesn't exist yet)). This way, we could exclude tracker logs from the LogViewer easily (even if the tag is for a plugin because the log was in the plugin's dimension or something). I will probably create a 3.0 issue for this.

If you think any of these are good ideas, let me know, I'll add them as issues in the LogViewer repo. Eventually, they can be added.

If we leave it as a standalone plugin I'd add it as a submodule to Piwik.

+1 for adding as a submodule. I think having good diagnostics isn't a choice we should leave for users to make for themselves.

Good feedback thx! I tried to keep it simple for now so won't add all of them but maybe some that are implemented quickly.

Re multi line logs: I didn't think of that. I don't think multi-line logs are a good idea. I think the proper solution would be to actually create a log entry for each line when logging or to remove the line breaks. It would be hard to handle them correctly and it can lead to side effects eg when multiple processes log see eg http://stackoverflow.com/questions/4669590/displaying-a-log-per-line-for-a-multiline-text

Should we rather remove the line breaks or log one message per line? I'd go with the second for now but can change it any time

+1 for adding as a submodule. I think having good diagnostics isn't a choice we should leave for users to make for themselves.

It wouldn't be installed for users by default (unless one installs via git). As mentioned > 90% will most likely never need it, developers will use console tools anyway (most of them). Its rather interesting for when debugging where you do not have access to and for those cases we can ask users to install it. Piwik PRO can bundle it with their plugins and install it by default for clients.

Issued PR, pretty much all issues/bugs you mentioned were due to multi line messages.

Will now have a look re suggestions

I added a simple refresh button, a simple export feature (very simple just exporting as TSV) and a simple info icon that shows plain log config on hover and FAQ on click. Created issues for all other features and some more ideas (matomo-org/plugin-LogViewer#2 and matomo-org/plugin-LogViewer#5).

The suggestions are all very useful and awesome, would even love to work on it but feel like it's not needed right now. It may be better to use it for a while and to add those features for a while. Otherwise building a whole logging viewer tool now :) For this some services like Loggly are great :)

As mentioned > 90% will most likely never need it, developers will use console tools anyway (most of them).

If you mean users that use Piwik to view stats for a few websites, but not to achieve any critical organisational function (whether the organisation is a business, sole proprietorship, non-profit, whatever), then they probably won't use it. And even if they did, I doubt they'd know how. These aren't the users core Piwik should be targeting, just some of the users the open source Piwik is supporting.

I think the core Piwik platform should primarily target users that gain something from using Piwik while making it as easy as possible for hobbyists (or whoever) to use Piwik. So I think diagnostic tools should be available for everyone, but their use shouldn't be mandatory. Having a page in the admin UI that is useful to the users who need analytics is worth it, even if the majority of all users don't bother to use it. Especially since users that don't use it will just ignore the page. What harm could the extra page do?

I'm not too worried about developers using the plugin or re PRO support, we/they can usually get this information w/o needing the LogViewer plugin. To me, this is about creating a platform that can take care of itself (ie, allows self-service).

For this some services like Loggly are great :)

Unfortunately, as a SaaS, it's may not be something Piwik users may want to use. Unless there's a self-hosted option...

great set of features for a simple yet usable MVP. For sure many will enjoy this plugin and speed up troubleshooting. Well done @tsteur

👍 for tests coverage: got unit, integration, system and UI tests!

Next steps:

• Publicise the LogViewer plugin Publicise the LogViewer plugin #8834
• See other issues in the LogViewer issue tracker