Probe cannot connect from behind IDS/IPS with HTTPS inspection
Hello list, I just observed strange behaviour of one Atlas probe. It was unable to connect to the control server from a corporate network. I suspect that some IDS/IPS appliance detects and drops probe attempt to use port 443 for non-TLS traffic*. I've tested it by trying to SSH to control server from within that network and it failed. However, normal SSH to port 22 works fine. It would be nice if the probe would be able to try more ways to reach the control server, eg. first attempt on port 443, second on port 22, third on some arbitrary high port number. I know that it would be better to put the probe in front of the firewall/IPS, but this is much more complicated in many corporate networks. *) At least it doesn't do MitM on HTTPs traffic. I've checked that too. Cheers, Ondřej Caletka
On 2014.05.21. 9:26, Ondřej Caletka wrote:
Hello list,
I just observed strange behaviour of one Atlas probe. It was unable to connect to the control server from a corporate network. I suspect that some IDS/IPS appliance detects and drops probe attempt to use port 443 for non-TLS traffic*. I've tested it by trying to SSH to control server from within that network and it failed. However, normal SSH to port 22 works fine.
It would be nice if the probe would be able to try more ways to reach the control server, eg. first attempt on port 443, second on port 22, third on some arbitrary high port number. I know that it would be better to put the probe in front of the firewall/IPS, but this is much more complicated in many corporate networks.
*) At least it doesn't do MitM on HTTPs traffic. I've checked that too.
Cheers, Ondřej Caletka
Hello, The probe and the infrastructure it connects to are doing mutual authentication (every actor has its own key pair for that) over SSH, meaning that if there's an appliance meddling with the traffic, the connection will fail. That's of course by design. We *could* try to fall back to other ports, but reality is that in the vast majority of the cases 443 works fine (*), and where it doesn't, other ports would very likely be blocked as well. It's difficult to justify the added complexity for the marginal benefit. Regards, Robert (*) As far as we know... We are aware of cases where it doesn't.
On 21.05.14 9:48 , Robert Kisteleki wrote:
On 2014.05.21. 9:26, Ondřej Caletka wrote:
Hello list,
I just observed strange behaviour of one Atlas probe. It was unable to connect to the control server from a corporate network. I suspect that some IDS/IPS appliance detects and drops probe attempt to use port 443 for non-TLS traffic*. I've tested it by trying to SSH to control server from within that network and it failed. However, normal SSH to port 22 works fine.
It would be nice if the probe would be able to try more ways to reach the control server, eg. first attempt on port 443, second on port 22, third on some arbitrary high port number. I know that it would be better to put the probe in front of the firewall/IPS, but this is much more complicated in many corporate networks.
*) At least it doesn't do MitM on HTTPs traffic. I've checked that too.
Cheers, Ondřej Caletka
Hello,
The probe and the infrastructure it connects to are doing mutual authentication (every actor has its own key pair for that) over SSH, meaning that if there's an appliance meddling with the traffic, the connection will fail. That's of course by design.
We *could* try to fall back to other ports, but reality is that in the vast majority of the cases 443 works fine (*), and where it doesn't, other ports would very likely be blocked as well. It's difficult to justify the added complexity for the marginal benefit.
Regards, Robert
(*) As far as we know... We are aware of cases where it doesn't.
It seems to me that just trying port 22 if no contact can be made on 443 should be added to the requested features list. The reason being that probes could then work in places where policy allows 22. However I agree that the priority should be low considering the small number of cases this would fix and the likelihood that measurements would be affected by middle boxes as well. Daniel
Dne 21.5.2014 13:45, Daniel Karrenberg napsal(a):
It seems to me that just trying port 22 if no contact can be made on 443 should be added to the requested features list. The reason being that probes could then work in places where policy allows 22. However I agree that the priority should be low considering the small number of cases this would fix and the likelihood that measurements would be affected by middle boxes as well.
I agree completely. The only question is, what are atlas probes supposed to measure, Internet infrastructure or "eyeball" perspective of the Internet? From what I see, the objective is somewhere in the middle with probes installed both in datacentres (measuring infrastructure) as well as at users homes (measuring eyeballs). If measuring the users experience is legitimate use case, then it shouldn't be problem to have some middleboxes on the way. -- Ondřej
On 2014/05/21 16:37 , Ondřej Caletka wrote:
Dne 21.5.2014 13:45, Daniel Karrenberg napsal(a):
It seems to me that just trying port 22 if no contact can be made on 443 should be added to the requested features list. The reason being that probes could then work in places where policy allows 22. However I agree that the priority should be low considering the small number of cases this would fix and the likelihood that measurements would be affected by middle boxes as well.
I agree completely. The only question is, what are atlas probes supposed to measure, Internet infrastructure or "eyeball" perspective of the Internet? From what I see, the objective is somewhere in the middle with probes installed both in datacentres (measuring infrastructure) as well as at users homes (measuring eyeballs). If measuring the users experience is legitimate use case, then it shouldn't be problem to have some middleboxes on the way.
My experience is that probes at users' homes just work. There are a few gotchas but most work. Probes that cause trouble are usually where somebody explicitly tried to configure something in the network, firewalls, etc. I'm curious what this firewall is trying to do. If it allows unrestricted outbound connectivity over ssh, but not ssh on port 443. What is that rule trying to protect? Philip
Dne 21.5.2014 16:58, Philip Homburg napsal(a):
I'm curious what this firewall is trying to do. If it allows unrestricted outbound connectivity over ssh, but not ssh on port 443. What is that rule trying to protect?
I don't think anyone ever knows. It's like with the DNS inspection modules for Cisco not allowing UDP/53 packets bigger than 512 bytes (effectively killing EDNS0 and DNSSEC) but allowing unrestricted traffic on any other UDP port :) -- Ondřej
On 21/05/14 16:58, Philip Homburg wrote:
I'm curious what this firewall is trying to do. If it allows unrestricted outbound connectivity over ssh, but not ssh on port 443. What is that rule trying to protect? Its trying to prevent exactly this. A lot of exploits/malware dial home using the same assumption you do. i.e. 443 will be open. however most of the don't use ssl. With this in mind many IDS will drop anything talking on 443 that isn't ssl/tls. They also do the same protocol analysis on other ports so trying port 80 in this case is also likely to be drooped, as the traffic is not http.
On 2014/05/21 17:13 , john wrote:
On 21/05/14 16:58, Philip Homburg wrote:
I'm curious what this firewall is trying to do. If it allows unrestricted outbound connectivity over ssh, but not ssh on port 443. What is that rule trying to protect? Its trying to prevent exactly this. A lot of exploits/malware dial home using the same assumption you do. i.e. 443 will be open. however most of the don't use ssl. With this in mind many IDS will drop anything talking on 443 that isn't ssl/tls. They also do the same protocol analysis on other ports so trying port 80 in this case is also likely to be drooped, as the traffic is not http.
But in those setups, port 22 is usually closed. And if you really want to prevent malware from dialing home, you would have to close all other ports as well. Typically, I find port 22 closed and then port 443 wide open. So the availability of port 22 is what surprises me. Philip
On 21/05/14 17:20, Philip Homburg wrote:
On 2014/05/21 17:13 , john wrote:
On 21/05/14 16:58, Philip Homburg wrote:
I'm curious what this firewall is trying to do. If it allows unrestricted outbound connectivity over ssh, but not ssh on port 443. What is that rule trying to protect? Its trying to prevent exactly this. A lot of exploits/malware dial home using the same assumption you do. i.e. 443 will be open. however most of the don't use ssl. With this in mind many IDS will drop anything talking on 443 that isn't ssl/tls. They also do the same protocol analysis on other ports so trying port 80 in this case is also likely to be drooped, as the traffic is not http.
But in those setups, port 22 is usually closed. And if you really want to prevent malware from dialing home, you would have to close all other ports as well.
Typically, I find port 22 closed and then port 443 wide open. So the availability of port 22 is what surprises me. I agree i suspect if one has gone to this much trouble then high ports will also be closed and port 22 will be restricted to know hosts/subnets
On 2014.05.21. 17:37, john wrote:
On 21/05/14 17:20, Philip Homburg wrote:
On 2014/05/21 17:13 , john wrote:
On 21/05/14 16:58, Philip Homburg wrote:
I'm curious what this firewall is trying to do. If it allows unrestricted outbound connectivity over ssh, but not ssh on port 443. What is that rule trying to protect? Its trying to prevent exactly this. A lot of exploits/malware dial home using the same assumption you do. i.e. 443 will be open. however most of the don't use ssl. With this in mind many IDS will drop anything talking on 443 that isn't ssl/tls. They also do the same protocol analysis on other ports so trying port 80 in this case is also likely to be drooped, as the traffic is not http.
But in those setups, port 22 is usually closed. And if you really want to prevent malware from dialing home, you would have to close all other ports as well.
Typically, I find port 22 closed and then port 443 wide open. So the availability of port 22 is what surprises me. I agree i suspect if one has gone to this much trouble then high ports will also be closed and port 22 will be restricted to know hosts/subnets
Which brings us back to the original point: if 443 is closed / filtered, the chances are that so is 22. (As it's almost always an explicit user action to fiddle with these.) We can spend resources researching how often this happens, what we can do about it, implementing fallback mechanisms and workarounds, use different ports or even protocols. But at the end of the day, we'd likely save only a fraction of the struggling probes. Robert
On 21.05.14 16:58 , Philip Homburg wrote:
...
I'm curious what this firewall is trying to do. If it allows unrestricted outbound connectivity over ssh, but not ssh on port 443. What is that rule trying to protect?
That is all an interesting discussion but not all that useful. Our decision to use 443 comes from our experience/expectation that 443 is more permeable than 22. However the protocol we are running "belongs" on 22 and it is somewhat silly to not use that port when it is in fact usable. So I think it is a reasonable request to use 22 when available. Again: the relative priority of this is another question that depends on the number of cases where 22 works and 443 does not. Currently I would not expect this to happen often. But that may change as middle box silliness increases. So I suggest again to put it on the list of requests with low priority. Daniel
participants (6)
-
Daniel Karrenberg -
Daniel Karrenberg -
john -
Ondřej Caletka -
Philip Homburg -
Robert Kisteleki