During a recent penetration test for a client we at Cygenta noticed a few odd responses from some of the scans we were doing with a tool that is usually fantastically reliable, NMap. For those of you that don't know, NMap is used for finding and reporting on open network ports e.g. a website is normally found on port 80 and the https version on port 443. Computers offer up 65,535 different ports that can be assigned to different network services and the first 1,024 are generally pretty standard. For example most FTP servers will default to port 21 and SSH will normally be on port 22. This doesn't stop anyone from trying to hide a service by changing the port: you could, for example, run your SSH server on port 80 or even port 65,500 and the random scans that people do over the internet might just bypass you.
Someone on twitter recently mentioned that by moving their SSH server to a high number port, the number of attempted bruteforce attacks dropped to zero! This is what is called security through obscurity and it is not a great way to secure your services. Your users will need to know the port: if Amazon ran their website not on the default port 443 but instead, say port 7764, everytime someone wanted to go there they would have to write in the browser www.amazon.com:7764 which isn't convenient and somewhat defeats the purpose of domain names.
NMap allows us as penetration testers to scan all of the ports very quickly on an IP or URI/Domain and when it does this it will also return what service it thinks is running on it. Here is an example:
In this case, port 80 is open, the service is HTTP and there is a version too - Google Web Server (GWS)
With the background of what NMap does, I am sure you can appreciate the time and effort this tool saves us, but now to the meat of this blog post. When things go wrong, unless you are experienced you might not notice. This is why it's important never to rely on one tool for the answer. Let me step you through what happened, how we resolved it and most importantly why we did things the way we did.
Obviously we have changed names and IP's to keep our client confidential. For brevity we will focus on just the one host that gave us issues, rather than the large scope we had. We started by looking at the website itself, a standard website nothing interesting to see at all. NMap scans were kicked off and returned results pretty quickly, nothing we were not expecting, for example ports 80 and 443 both open. Except the service was wrong, it wasn't showing as HTTP/HTTPS it was showing something very different!
As you can see the service is not shown as HTTP but as RTSP, a real time streaming protocal normally used for things such as CCTV streams. Clearly NMap was getting confused.
Now we want to make it clear that we had already looked at the website and that it was definitely not a RTSP feed, but just a plain old HTTP site. So what was going on? Why was NMap so confused and yet so certain of itself?
At this point some of you are shouting about firing up wireshark or tshark and looking at the traffic being sent between the server and us, but as we already know by what's being returned the issue appears to be with NMap and how it is parsing the results. So we had to go deeper into understanding how NMap does things.
First when NMap performs a scan and it receives a response it does what is known as a 'soft match' this is essentially a REGEX match against the response vs a set of strings it has in it's code. If there is another match later on then it's considered a 'hard match'. Lets look at the match directive within NMap source:
match Directive Syntax: match <service> <pattern> [<versioninfo>]
From here we can look at a few match patterns that NMap has internally...
match ftp m/^220.*Welcome to .*Pure-?FTPd (\d\S+\s*)/ p/Pure-FTPd/ v/$1/ cpe:/a:pureftpd:pure-ftpd:$1/
match ssh m/^SSH-([\d.]+)-OpenSSH[_-]([\w.]+)\r?\n/i p/OpenSSH/ v/$2/ i/protocol $1/ cpe:/a:openbsd:openssh:$2/
match mysql m|^\x10\0\0\x01\xff\x13\x04Bad handshake$| p/MySQL/ cpe:/a:mysql:mysql/
match chargen m|@ABCDEFGHIJKLMNOPQRSTUVWXYZ| match uucp m|^login: login: login: $| p/NetBSD uucpd/ o/NetBSD/ cpe:/o:netbsd:netbsd/a
match printer m|^([\w-_.]+): lpd: Illegal service request\n$| p/lpd/ h/$1/
match afs m|^[\d\D]{28}\s*(OpenAFS)([\d\.]{3}[^\s\0]*)\0| p/$1/ v/$2/
The match directive tells NMap how to recognize services based on responses to the string sent by the previous Probedirective. A single Probe line may be followed by dozens or hundreds of match statements. If the given pattern matches, an optional version specifier builds the application name, version number, and additional info for Nmap to report. The arguments to this directive follow: <service>
This is simply the service name that the pattern matches. Examples would be ssh, smtp, http, or snmp. As a special case, you can prefix the service name with ssl/, as in ssl/vmware-auth. In that case, the service would be stored as vmware-auth tunneled by SSL. This is useful for services which can be fully recognized without the overhead of making an SSL connection.
For instance, the data returned from the X11Probe is very unlikely to match any regular expressions crafted for the GetRequest probe. On the other hand, it is likely that results returned from a Probe such as RTSPRequest might match a regular expression crafted for GetRequest since the two protocols being tested for are closely related. So the RTSPRequest probe has a fallback to GetRequest matches.
Put simply, the returned response from this webserver is too similar to the RTSP response so NMap cannot diferentiate the two and it comes down to who matches first. As you can see, it's facinating to see how some of the tools we use on a day-to-day basis actually work under the hood. It should go without saying that you need to understand the tools you are working with otherwise they can lead you astray.
We hope you found this write-up as facinating as we found digging into the issue, if so please stay in touch with us and subscribe to the blog.