Unfortunately my local server doesn't support umlauts in the domain so I cannot fully reproduce. I still debugged Matomo though and how we read the trusted hosts configuration and there everything should be all fine.

In case you can access your Matomo instance on the server, can you for a one time test change this line?
https://github.com/matomo-org/matomo/blob/3.8.0-b5/core/Url.php#L211
from $host = @$_SERVER['HTTP_HOST']; to $host = @$_SERVER['HTTP_HOST'];var_export($host);exit;?
And then reload the page... Make sure to undo the change afterwards. Could you send us the output of that?

That's interesting because for me it is working fine.
I added 127.0.0.15 äöü-test.matomo to /etc/hosts and curl couldn't connect:

$ curl http://äöü-test.matomo -v
* Could not resolve host: xn---test-fra0kzb.matomo
* Closing connection 0
curl: (6) Could not resolve host: xn---test-fra0kzb.matomo

So I know that xn---test-fra0kzb.matomo was the punycode and added it to /etc/hosts and DNS was working fine.
I also added the punycode (not the umlaut domain) to server_name in the nginx config.
Afterwards Matomo loaded correctly and suggested adding the punycode to the config.

(unrelated, but after 3.7.0. that screen looks a bit ugly)

After adding it Matomo works as it should.

@schousda I'll close the issue for now as it seems to work, but feel free to comment and reopen if you still experience the issue and ideally give us the output of the HOST entry mentioned above.