I was wondering if my plugins were up to date and how to update them.

Until one plugin did have an update available, i saw no info on that.
The plugin shows up on the top of the plugins list then.

To give an always clear status indicator, i'd suggest an update-button on the plugins page, similar to the one for core updates.
Or, integrate into the core button, so the status text would read like eg.

You are using Piwik version z.x.y
All plugins are up to date (3 need manual check)