Matomo can be tricked to record spoofed X-Forwarded-For IPs #17202

timdream · 2021-02-07T20:26:42Z

Problem

Considering the current implementation:

Lines 99 to 125 in a31fd86

    
               /** 
        
                * Returns the last IP address in a comma separated list, subject to an optional exclusion list. 
        
                * 
        
                * @param string $csv Comma separated list of elements. 
        
                * @param array $excludedIps Optional list of excluded IP addresses (or IP address ranges). 
        
                * @return string Last (non-excluded) IP address in the list or an empty string if all given IPs are excluded. 
        
                */ 
        
               public static function getFirstIpFromList($csv, $excludedIps = null) 
        
               { 
        
                   $p = strrpos($csv, ','); 
        
                   if ($p !== false) { 
        
                       $elements = explode(',', $csv); 
        
                       foreach ($elements as $ipString) { 
        
                           $element = trim(Common::sanitizeInputValue($ipString)); 
        
                           if(empty($element)) { 
        
                               continue; 
        
                           } 
        
                           $ip = \Matomo\Network\IP::fromStringIP(IPUtils::sanitizeIp($element)); 
        
                           if (empty($excludedIps) || (!in_array($element, $excludedIps) && !$ip->isInRanges($excludedIps))) { 
        
                               return $element; 
        
                           } 
        
                       } 
        
                       return ''; 
        
                   } 
        
                   return trim(Common::sanitizeInputValue($csv)); 
        
               }

With the following setup:

[client] -> [reverse proxy 1] -> [reverse proxy 2] -> [matomo Nginx]

Since each proxy appends the IP of the incoming connection according to RFC7239, X-Forwarded-For will be this by the time the connection reaches Nginx (1):

X-Forwarded-For: (client), (proxy 1)

Matomo currently extracts the first IP from the list, since #10404. This is correct until you realized the "client" can pretended to be a proxy as well, by including its own X-Forwarded-For header, like this:

curl -i -H"X-Forwarded-For: 8.8.8.8" "http://example.com/matomo/matomo.php?..."

When this happens, proxy 1 will be faithfully preserve the header passed in and append the client IP after it (2):

X-Forwarded-For: 8.8.8.8, (client), (proxy 1)

And Matomo will extract 8.8.8.8 as the client IP.

Solutions

Solution 1: Trust the "transparent proxies"

If Matomo decided the first "client" IP can always be trusted, we can still extract the first IP address, and this issue can be closed as "works as intended." But it will in risk of having the analytics data being polluted. Brutal force login detection can be tricked too.

Solution 2: Extract the last IP address

This would involve reverting #10404 and go back to extract the last IP address. Since the original proxy_ips[] configuration is still intact, when facing a header like (2), above, a correctly configured Matomo will be able to exclude the "proxy 1" IP address.

Notes

There can be a solution 1.1 where Mamoto will rely on proxy 1 (the first public-facing proxy) to remove the spoofed header. However this is not always possible (not possible for Apache AFAIK)
It is unclear to me what other client IP headers (HTTP_CF_CONNECTING_IP, HTTP_CLIENT_IP, etc) behaves and if Matomo have to do things differently between these and the X-Forwarded-For header.
IP::getNonProxyIpFromHeader retrieves final proxy instead of client #7060 is a staled discussion related to this, but the discussion talks about the header processing in the context before Wrong IP extracted from HTTP_X_FORWARDED_FOR when there is more than one #10342.
Better documentation of Proxy setup #16379 have since promoted proxy_ips[] pretty well in the documentation. We may consider doubling down on the documentation effort when we pick either solution for this issue.
Lastly, one tiny nitpick: Extract the first IP from HTTP_X_FORWARDED_FOR and HTTP_CLIENT_IP and HTTP_CF_CONNECTING_IP and HTTP_X_FORWARDED_HOST when there is more than one IP #10404 updated the name of the function IP::getFirstIpFromList but it didn't change the word "last" in the function comment right above it :)

Thanks!

The text was updated successfully, but these errors were encountered:

diosmosis · 2021-02-07T23:18:49Z

Hi @timdream, thanks for creating this issue. Is this behavior in Matomo causing a problem for you specifically? Do you know of a client that does this and is causing a problem?

timdream · 2021-02-08T01:27:52Z

@diosmosis It did not cause a problem for me! I am just trying to document what I've found that may be problematic in terms of spoofing and attack vector. I did not spot such attack/spoofing taking place on my own instance.

tsteur · 2021-02-08T22:42:46Z

Thanks @timdream would this be kind of a duplicate or related to #7060 ?

timdream · 2021-02-09T02:15:23Z

@tsteur Yes the issue is related (see note), but I wouldn't say it is a duplicate because the research I made here contains new information on the current implementation.

tsteur · 2021-02-10T00:02:06Z

There can be a solution 1.1 where Mamoto will rely on proxy 1 (the first public-facing proxy) to remove the spoofed header. However this is not always possible (not possible for Apache AFAIK)

that was my first thought would be the preferred solution as AFAIK in #10404 it says

The RFC at https://tools.ietf.org/html/rfc7239 makes it clear that we should extract the first IP from the header HTTP_X_FORWARDED_FOR

So changing it to last IP could potentially break things again etc.

Another workaround might be to use other headers depending on what can be configured that can only be set by the public facing proxy? I don't know if that's the same problem for Apache.

Mostly seeing this so far as a documentation issue to mention it in the docs for setting up the proxy if it's not mentioned yet.

tsteur · 2021-06-29T20:15:20Z

BTW if you have a WAF then you may be able to configure to block any request having a X_FORWARDED_FOR request set.

tsteur · 2021-07-15T04:59:28Z

We are adding a new config setting for this in #17765

Maybe in Matomo 5 we make it default to read the last entry. However, depending on the setup the last IP might be an internal IP and it could break things.

To be evaluated with Matomo 5 whether we'll change the default.

MrIsak · 2022-02-25T10:15:30Z

BTW if you have a WAF then you may be able to configure to block any request having a X_FORWARDED_FOR request set.

You are definitely on to something. AFAIK is it more or less best practice to replace the XFF(X-Forwarded-For) header on all traffic passing the first proxy that handles incoming traffic, with the real client source IP. Since you can never trust a client.
But this is not up to the application to deal with IMO.

If the XFF header is present, and the "proxy_client_headers[] = HTTP_X_FORWARDED_FOR" is set,
the application must be able to trust the chain of IP's. And use the first as client.

tsteur · 2022-05-10T02:55:07Z

We have prepared this feature in https://github.com/matomo-org/matomo/pull/17765/files

Ideally we enable this security feature by default to have the IP detection more secure ([General] proxy_ip_read_last_in_list=1).

Then we update the guide in https://matomo.org/faq/how-to-install/faq_98/ to mention people may need to disable this feature if the first IP should be used.

jane-twizel · 2022-07-08T02:34:39Z

Will need to be communicated in the developer changelog as well.

tsteur added the c: Security For issues that make Matomo more secure. Please report issues through HackerOne and not in Github. label Feb 9, 2021

tsteur added this to the 5.0.0 milestone Jul 15, 2021

tsteur mentioned this issue Jul 15, 2021

add config to read the last IP address in the list of proxies rather than the first #17765

Merged

10 tasks

mattab added the 5.0.0 label Jan 4, 2023

bx80 mentioned this issue Feb 9, 2023

Increase security of X-Forwarded-For header usage by using last ip by default #20341

Merged

11 tasks

innocraft-automation closed this as completed Feb 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Matomo can be tricked to record spoofed X-Forwarded-For IPs #17202

Matomo can be tricked to record spoofed X-Forwarded-For IPs #17202

timdream commented Feb 7, 2021

diosmosis commented Feb 7, 2021

timdream commented Feb 8, 2021

tsteur commented Feb 8, 2021

timdream commented Feb 9, 2021

tsteur commented Feb 10, 2021

tsteur commented Jun 29, 2021

tsteur commented Jul 15, 2021

MrIsak commented Feb 25, 2022

tsteur commented May 10, 2022

jane-twizel commented Jul 8, 2022

Matomo can be tricked to record spoofed X-Forwarded-For IPs #17202

Matomo can be tricked to record spoofed X-Forwarded-For IPs #17202

Comments

timdream commented Feb 7, 2021

Problem

Solutions

Solution 1: Trust the "transparent proxies"

Solution 2: Extract the last IP address

Notes

diosmosis commented Feb 7, 2021

timdream commented Feb 8, 2021

tsteur commented Feb 8, 2021

timdream commented Feb 9, 2021

tsteur commented Feb 10, 2021

tsteur commented Jun 29, 2021

tsteur commented Jul 15, 2021

MrIsak commented Feb 25, 2022

tsteur commented May 10, 2022

jane-twizel commented Jul 8, 2022