Using Nmap results to help harden Linux systems
System security is not a one-and-done task. Rather, there are numerous layers to an organization's approach to security. Some of those layers are physical security to the datacenters, regular patching and maintenance of the infrastructure, continuing user awareness education, and scanning systems for issues. This article discusses how to use the nmap and nc commands to scan a system so that you can determine the appropriate next steps. I use a few systems in my examples here. The system that does the scanning is my local Red Hat Enterprise Linux (RHEL) 8.3 computer, opendemo.usersys.redhat.com is the Red Hat Satellite 6.8 system used because it has several open ports, and I have various target systems.
[ You might also like: Sysadmin security: 8 Linux lockdown controls ]
Basic scans
To see the ports in use on my Satellite server, I SSH to the server and then use netstat:
[root@opendemo ~]# netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1443/mongod
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1197/redis-server 1
tcp 0 0 0.0.0.0:5646 0.0.0.0:* LISTEN 1132/qdrouterd
tcp 0 0 127.0.0.1:8751 0.0.0.0:* LISTEN 1194/python
tcp 0 0 0.0.0.0:5647 0.0.0.0:* LISTEN 1132/qdrouterd
tcp 0 0 127.0.0.1:19090 0.0.0.0:* LISTEN 1237/cockpit-ws
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1175/sshd
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1242/postmaster
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1396/master
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 1138/ruby
tcp 0 0 127.0.0.1:45285 0.0.0.0:* LISTEN 28650/Passenger Rack
tcp 0 0 127.0.0.1:5671 0.0.0.0:* LISTEN 1140/qpidd
tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN 1240/ruby
tcp 0 0 127.0.0.1:5672 0.0.0.0:* LISTEN 1140/qpidd
tcp6 0 0 :::8140 :::* LISTEN 2101/java
tcp6 0 0 127.0.0.1:61613 :::* LISTEN 1135/java
tcp6 0 0 :::5646 :::* LISTEN 1132/qdrouterd
tcp6 0 0 :::5647 :::* LISTEN 1132/qdrouterd
tcp6 0 0 :::80 :::* LISTEN 1131/httpd
tcp6 0 0 :::22 :::* LISTEN 1175/sshd
tcp6 0 0 ::1:5432 :::* LISTEN 1242/postmaster
tcp6 0 0 :::3128 :::* LISTEN 1258/(squid-1)
tcp6 0 0 ::1:25 :::* LISTEN 1396/master
tcp6 0 0 127.0.0.1:8443 :::* LISTEN 1135/java
tcp6 0 0 :::443 :::* LISTEN 1131/httpd
tcp6 0 0 :::9090 :::* LISTEN 1138/ruby
tcp6 0 0 127.0.0.1:8005 :::* LISTEN 1135/java
tcp6 0 0 ::1:5671 :::* LISTEN 1140/qpidd
tcp6 0 0 :::8008 :::* LISTEN 1240/ruby
tcp6 0 0 ::1:5672 :::* LISTEN 1140/qpidd
tcp6 0 0 :::5000 :::* LISTEN 1131/httpd
[root@opendemo ~]#
However, some of those are limited to the localhost, 127.0.0.1. To see which ports are publicly visible, I start by using a default nmap scan from my local system:
[pgervase@pgervase ~]$ nmap opendemo.usersys.redhat.com
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 20:28 EST
Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240)
Host is up (0.041s latency).
Not shown: 993 closed ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
3128/tcp open squid-http
5000/tcp open upnp
8008/tcp open http
9090/tcp open zeus-admin
Nmap done: 1 IP address (1 host up) scanned in 3.81 seconds
[pgervase@pgervase ~]$
This output shows that my local system can see fewer public ports than what I could see when I was SSH'd into the server. Some of those non-public ports are 25, which is used by master, and 8005, 8140, 8443, and 61613, which are used by java. Looking at ps output and grepping for the PID of master from that netstat output, master is the postfix mailer:
[root@opendemo ~]# ps auxww | grep 1396
root 1396 0.0 0.0 89740 2188 ? Ss Jan05 0:00 /usr/libexec/postfix/master -w
root 29665 0.0 0.0 112816 968 pts/0 R+ 20:32 0:00 grep --color=auto 1396
[root@opendemo ~]#
That (master) is running locally so that mail can get sent to internal addresses but is not listening for incoming email, nor is it sending anything to any other host.
The other ports mentioned were for java. When you look at the netstat output, two different java processes are responsible for those ports:
[root@opendemo ~]# netstat -tlpn | grep java
tcp6 0 0 :::8140 :::* LISTEN 2101/java
tcp6 0 0 127.0.0.1:61613 :::* LISTEN 1135/java
tcp6 0 0 127.0.0.1:8443 :::* LISTEN 1135/java
tcp6 0 0 127.0.0.1:8005 :::* LISTEN 1135/java
[root@opendemo ~]#
When you look at ps output for PID 1135, it’s used by tomcat:
[root@opendemo ~]# ps auxww | grep 1135
tomcat 1135 0.3 3.5 12409252 2165668 ? Ssl Jan05 9:25 /usr/lib/jvm/jre/bin/java -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
root 31507 0.0 0.0 112816 968 pts/0 S+ 20:53 0:00 grep --color=auto 1135
[root@opendemo ~]#
When I look in the/usr/share/tomcat/conf/server.xml file, it has content such as:
<Server port="8005" shutdown="SHUTDOWN">
...
<Connector port="8443"
address="localhost"
protocol="HTTP/1.1"
SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="want"
sslProtocols="TLSv1.2"
sslEnabledProtocols="TLSv1.2"
....
This shows how the ports that will get used are defined in the config file.
When I look at the other java process I mentioned, PID 2101 for port 8140, I see this is used by puppet:
[root@opendemo ~]# ps auxww | grep 2101
puppet 2101 0.2 2.5 9787492 1545188 ? Sl Jan05 7:14 /usr/bin/java -Xms2G -Xmx2G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -XX:OnOutOfMemoryError="kill -9 %p" -XX:ErrorFile=/var/log/puppetlabs/puppetserver/puppetserver_err_pid%p.log -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter
root 31696 0.0 0.0 112816 968 pts/0 S+ 20:55 0:00 grep --color=auto 2101
[root@opendemo ~]#
Based on the netstat output, port 8140 should be visible to the public, but nmap from my local system didn't report it in its results. Here again, is the netstat output from the Satellite server:
[root@opendemo ~]# netstat -tunap| grep 8140
tcp6 0 0 :::8140 :::* LISTEN 2101/java
[root@opendemo ~]#
and the nmap from my local server:
[pgervase@pgervase ~]$ nmap opendemo.usersys.redhat.com | grep 8140
[pgervase@pgervase ~]$
However, I can force nmap to check on a specific port or range of ports:
[pgervase@pgervase ~]$ nmap -p 8140 opendemo.usersys.redhat.com
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 21:07 EST
Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240)
Host is up (0.039s latency).
PORT STATE SERVICE
8140/tcp open puppet
Nmap done: 1 IP address (1 host up) scanned in 0.39 seconds
[pgervase@pgervase ~]$ nmap -p 8000-9000 opendemo.usersys.redhat.com
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 21:07 EST
Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240)
Host is up (0.040s latency).
Not shown: 999 closed ports
PORT STATE SERVICE
8008/tcp open http
8140/tcp open puppet
Nmap done: 1 IP address (1 host up) scanned in 2.12 seconds
[pgervase@pgervase ~]$
By forcing nmap to check those ports, I was able to see the :8140 port which a basic nmap scan did not report. This shows that a default nmap scan without extra arguments might be good enough for a first look at the system but might miss ports that are actually open.
This information is important in security testing so that sysadmins can identify potential vulnerabilities. From the nmap output scan, run locally on my system, you saw the ports that were publicly open. Prior versions of Satellite had tomcat configured so that some of those ports were public when that was not necessary. To read over some of the discussion for that issue, you can read the Bugzilla where this was resolved.
Verify certificates
Another issue that nmap can help with is to verify the certs being used on those various ports. Using nmap, you saw the open ports. Using those ports, you can use OpenSSL to see the cert used on the port. A number of those port are using self-signed certificates. To use nmap and OpenSSL together to check the ports on a remote system, you could do something like:
$ for port in `nmap -p 1-5000 opendemo.usersys.redhat.com | grep " open " | cut -d "/" -f 1`
> do echo checking on port: $port
> echo | openssl s_client -showcerts -connect opendemo.usersys.redhat.com:$port
> done &> opendemo.certs.txt.`date +%Y%m%d`
In my opendemo.certs.txt.20210127 file, it would have content like:
checking on port: 443
depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU = SomeOrgUnit, CN = opendemo.usersys.redhat.com
verify return:1
depth=0 C = US, ST = North Carolina, O = Katello, OU = SomeOrgUnit, CN = opendemo.usersys.redhat.com
verify return:1
CONNECTED(00000003)
….
SSL handshake has read 3476 bytes and written 463 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Use that output file to verify that the certs in use are the correct TLS version.
If you use nc (or ncat), you might see more information than presented in the web UI. For this example, I used nc to connect to a web server:
$ nc 10.19.47.242 80
asdf
HTTP/1.1 400 Bad Request
Date: Sat, 09 Jan 2021 01:25:40 GMT
Server: Apache/2.4.37 (Red Hat Enterprise Linux)
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
From that output, I can see the version of Apache that was installed. With that information, an attacker could learn what exploits the server was vulnerable to. Because of this, a web server should limit the amount of information that displayed:
[pgervase@pgervase ~]$ nc opendemo.usersys.redhat.com 443
GET / HTTP/1.1
HTTP/1.1 400 Bad Request
Date: Fri, 08 Jan 2021 02:33:08 GMT
Server: Apache
Content-Length: 362
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</p>
</body></html>
[pgervase@pgervase ~]$
Note that in this output, there is no version information for Apache.
In this next example, I use nc to connect to port 21 on my client system, which I can see is open:
[pgervase@pgervase ~]$ nmap 10.19.47.242
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-08 21:02 EST
Nmap scan report for 10.19.47.242
Host is up (0.039s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
80/tcp open http
111/tcp open rpcbind
Nmap done: 1 IP address (1 host up) scanned in 0.83 seconds
[pgervase@pgervase ~]$ nc 10.19.47.242 21
220 (vsFTPd 3.0.3)
That 3.0.3 version is confirmed when I SSH to the system and use the rpm command:
[root@vulnerable ~]# rpm -q vsftpd
vsftpd-3.0.3-32.el8.x86_64
[root@vulnerable ~]# rpm -qi vsftpd
Name : vsftpd
Version : 3.0.3
Release : 32.el8
<snipped>
Again, just like with learning the Apache version on the device, being able to do reconnaissance in your environment so that you know what a potential attacker can learn about your systems is important.
Scanning from Kali
In the next section, I show some results from scanning a system from a Kali server. In this example, I know that the target server has distccd running on port 3632, but, like earlier, nmap does not detect that port by default, and so I had to check for it specifically:
Now that you know distccd is open, you can use nmap's built-in capabilities to determine where it could potentially be exploited:
If you'd used only a plain nmap scan, you'd have missed that exploitable vulnerability. In my example, I ran uname -a on the remote system, but I could have run any command.
Identifying services
One final way to use nmap is with the -sV option, which probes the open ports and determines the service or version information. For this example, I changed the port that Apache runs on from 80 to 90 and then restarted the service. Below you can see the difference between a plain nmap scan and then using the -sV option, which correctly determined the service as httpd rather than dnsix:
[root@pgervase ~]# nmap 10.19.47.242
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-09 19:57 EST
Nmap scan report for 10.19.47.242
Host is up (0.043s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
90/tcp open dnsix
111/tcp open rpcbind
Nmap done: 1 IP address (1 host up) scanned in 1.80 seconds
[root@pgervase ~]# nmap -sV 10.19.47.242
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-09 19:52 EST
Nmap scan report for 10.19.47.242
Host is up (0.040s latency).
Not shown: 996 closed ports
PORT STATE SERVICE VERSION
21/tcp open ftp vsftpd 3.0.3
22/tcp open ssh OpenSSH 8.0 (protocol 2.0)
90/tcp open http Apache httpd 2.4.37 ((Red Hat Enterprise Linux))
111/tcp open rpcbind 2-4 (RPC #100000)
Service Info: OS: Unix
[ Want to learn more about security? Check out the IT security and compliance checklist. ]
Wrap up
Now that you've been able to get a detailed report of what's running on your systems, what do you do next? The first thing is to be sure that there are no unexpected ports open. For this, verify with the applications team, security teams, and your coworkers might be appropriate. Next is to ensure that the exposed services are properly secured. This means taking steps such as making sure that all software is updated, updated ciphers are supported, insecure protocols are not in use, and default passwords for the services have been changed.
This article is an introduction to investigating your servers. Use nc and nmap to verify which ports are open and use the ps command to trace back the processes using those ports. I've also provided an example of how you can use nmap with the --script argument to test your systems. To continue on this learning path, one possible next step is to research using nmap as an attack engine by investigating the default scripts in /usr/share/nmap/scripts/.
Peter Gervase
I am a Senior Principal Security Architect at Verizon. Before that, I worked at Red Hat in various roles such as consulting and in the Solutions Architect where I specialized in Smart Management, Ansible, and OpenShift. More about me
Try Red Hat Enterprise Linux
Download it at no charge from the Red Hat Developer program.
Related Content
