<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi all,<div><br></div><div>I am currently investigating a rather strange issue where every couple of minute a new port is being listened on. To trace this issue I created a quick shell script to dump me all processes and their open ports to a file:</div><div><br></div><div><div>#!/bin/bash</div><div>echo "==============================================" >> /tmp/traceports.log</div><div>date >> /tmp/traceports.log</div><div>echo "==============================================" >> /tmp/traceports.log</div><div>/usr/sbin/lsof -i -P >> /tmp/traceports.log</div><div>echo "==============================================" >> /tmp/traceports.log</div><div>netstat -tulpn >> /tmp/traceports.log</div><div>echo "==============================================" >> /tmp/traceports.log</div><div>echo "Done" >> /tmp/traceports.log</div></div><div><br></div><div>At on point for example port 36281 was listened on. However the output from the log file didn't provide much insight into the source or purpose. LSOF didn't even list this port / process while NETSTAT indicated that the process was "-" (tcp 0 0 0.0.0.0:36281 0.0.0.0:* LISTEN - )</div><div><br></div><div>Any idea how I can investigate this further and isolate the process or application which is listening on these ports.</div><div><br></div><div>Thanks in advance for any pointers.</div><div>Alex</div><div><br></div><div><div><font face="Courier New">==============================================</font></div><div><font face="Courier New">Tue Jun 4 04:40:02 UTC 2013</font></div><div><font face="Courier New">==============================================</font></div><div><font face="Courier New">COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME</font></div><div><font face="Courier New">master 1119 root 12u IPv4 11310 0t0 TCP localhost:25 (LISTEN)</font></div><div><font face="Courier New">master 1119 root 13u IPv6 11312 0t0 TCP localhost:25 (LISTEN)</font></div><div><font face="Courier New">rpcbind 3402 rpc 8u IPv4 33084 0t0 UDP *:111 </font></div><div><font face="Courier New">rpcbind 3402 rpc 9u IPv4 33088 0t0 UDP *:609 </font></div><div><font face="Courier New">rpcbind 3402 rpc 10u IPv4 33089 0t0 TCP *:111 (LISTEN)</font></div><div><font face="Courier New">rpcbind 3402 rpc 11u IPv6 33091 0t0 UDP *:111 </font></div><div><font face="Courier New">rpcbind 3402 rpc 12u IPv6 33093 0t0 UDP *:609 </font></div><div><font face="Courier New">rpcbind 3402 rpc 13u IPv6 33094 0t0 TCP *:111 (LISTEN)</font></div><div><font face="Courier New">rpc.statd 4050 rpcuser 5u IPv4 42097 0t0 UDP *:834 </font></div><div><font face="Courier New">rpc.statd 4050 rpcuser 8u IPv4 42104 0t0 UDP *:38979 </font></div><div><font face="Courier New">rpc.statd 4050 rpcuser 9u IPv4 42108 0t0 TCP *:48234 (LISTEN)</font></div><div><font face="Courier New">rpc.statd 4050 rpcuser 10u IPv6 42112 0t0 UDP *:42649 </font></div><div><font face="Courier New">rpc.statd 4050 rpcuser 11u IPv6 42116 0t0 TCP *:34077 (LISTEN)</font></div><div><font face="Courier New">sshd 4315 root 3u IPv4 45281 0t0 TCP *:22 (LISTEN)</font></div><div><font face="Courier New">sshd 4315 root 4u IPv6 45283 0t0 TCP *:22 (LISTEN)</font></div><div><font face="Courier New">ntpd 4447 ntp 16u IPv4 45597 0t0 UDP *:123 </font></div><div><font face="Courier New">ntpd 4447 ntp 17u IPv6 45598 0t0 UDP *:123 </font></div><div><font face="Courier New">ntpd 4447 ntp 18u IPv6 45602 0t0 UDP localhost:123 </font></div><div><font face="Courier New">ntpd 4447 ntp 19u IPv6 45603 0t0 UDP [fe80::5477:49ff:fe7d:d451]:123 </font></div><div><font face="Courier New">ntpd 4447 ntp 20u IPv4 45604 0t0 UDP localhost:123 </font></div><div><font face="Courier New">ntpd 4447 ntp 21u IPv4 45605 0t0 UDP th-dc03-con01.one.local:123 </font></div><div><font face="Courier New">osad 4487 root 3u IPv4 45703 0t0 TCP th-dc03-con01.one.local:52769->th-dc03-space01.one.local:5222 (ESTABLISHED)</font></div><div><font face="Courier New">sshd 7216 root 3u IPv4 20723320 0t0 TCP th-dc03-con01.one.local:22->10.104.97.54:34982 (ESTABLISHED)</font></div><div><font face="Courier New">sshd 7218 ahartner 3u IPv4 20723320 0t0 TCP th-dc03-con01.one.local:22->10.104.97.54:34982 (ESTABLISHED)</font></div><div><font face="Courier New">zabbix_ag 19024 zabbix 4u IPv4 1169196 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19024 zabbix 8u IPv6 1169197 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19025 zabbix 4u IPv4 1169196 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19025 zabbix 8u IPv6 1169197 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19026 zabbix 4u IPv4 1169196 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19026 zabbix 8u IPv6 1169197 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19027 zabbix 4u IPv4 1169196 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19027 zabbix 8u IPv6 1169197 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19028 zabbix 4u IPv4 1169196 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19028 zabbix 8u IPv6 1169197 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19029 zabbix 4u IPv4 1169196 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">zabbix_ag 19029 zabbix 8u IPv6 1169197 0t0 TCP *:10050 (LISTEN)</font></div><div><font face="Courier New">java 20573 lpm 10u IPv6 15562206 0t0 TCP *:48082 (LISTEN)</font></div><div><font face="Courier New">java 20573 lpm 11u IPv6 15562353 0t0 TCP *:48081 (LISTEN)</font></div><div><font face="Courier New">java 20573 lpm 12u IPv6 16327122 0t0 TCP localhost:48081->localhost:52568 (ESTABLISHED)</font></div><div><font face="Courier New">java 20573 lpm 14u IPv6 15562358 0t0 TCP *:5435 (LISTEN)</font></div><div><font face="Courier New">java 20573 lpm 20u IPv6 16327138 0t0 TCP localhost:48081->localhost:52569 (ESTABLISHED)</font></div><div><font face="Courier New">java 20573 lpm 21u IPv6 16327149 0t0 TCP localhost:48081->localhost:52570 (ESTABLISHED)</font></div><div><font face="Courier New">java 20582 lpm 40u IPv6 15562469 0t0 TCP *:8080 (LISTEN)</font></div><div><font face="Courier New">java 20582 lpm 41u IPv6 15562476 0t0 TCP *:8443 (LISTEN)</font></div><div><font face="Courier New">java 20582 lpm 42u IPv6 16327137 0t0 TCP localhost:52569->localhost:48081 (ESTABLISHED)</font></div><div><font face="Courier New">java 20582 lpm 87u IPv6 16327121 0t0 TCP localhost:52568->localhost:48081 (ESTABLISHED)</font></div><div><font face="Courier New">java 20582 lpm 88u IPv6 16327148 0t0 TCP localhost:52570->localhost:48081 (ESTABLISHED)</font></div><div><font face="Courier New">java 20582 lpm 92u IPv6 17443300 0t0 TCP th-dc03-con01.one.local:33654->10.103.45.97:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 96u IPv6 17443330 0t0 TCP th-dc03-con01.one.local:35346->10.103.45.105:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 99u IPv6 17443309 0t0 TCP th-dc03-con01.one.local:36762->10.103.45.102:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 101u IPv6 17443310 0t0 TCP th-dc03-con01.one.local:36763->10.103.45.102:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 103u IPv6 17443312 0t0 TCP th-dc03-con01.one.local:51976->10.103.45.99:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 105u IPv6 17443314 0t0 TCP th-dc03-con01.one.local:51977->10.103.45.99:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 109u IPv6 17443320 0t0 TCP th-dc03-con01.one.local:52449->10.103.45.103:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 114u IPv6 17443325 0t0 TCP th-dc03-con01.one.local:44779->10.103.45.104:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 115u IPv6 17443326 0t0 TCP th-dc03-con01.one.local:44780->10.103.45.104:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">java 20582 lpm 117u IPv6 17443328 0t0 TCP th-dc03-con01.one.local:35345->10.103.45.105:8443 (CLOSE_WAIT)</font></div><div><font face="Courier New">.vasd 22532 daemon 14u IPv4 20755165 0t0 TCP th-dc03-con01.one.local:49161->th-dc03-ad02.one.local:389 (ESTABLISHED)</font></div><div><font face="Courier New">ossec-age 29917 ossec 16u IPv4 1505336 0t0 UDP th-dc03-con01.one.local:48001->th-dc03-hids01.one.local:1514 </font></div><div><font face="Courier New">==============================================</font></div><div><font face="Courier New">Active Internet connections (only servers)</font></div><div><font face="Courier New">Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name </font></div><div><font face="Courier New">tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3402/rpcbind </font></div><div><font face="Courier New">tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 4315/sshd </font></div><div><font face="Courier New"><b>tcp 0 0 0.0.0.0:36281 0.0.0.0:* LISTEN - </b> </font></div><div><font face="Courier New">tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1119/master </font></div><div><font face="Courier New">tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 19024/zabbix_agentd </font></div><div><font face="Courier New">tcp 0 0 0.0.0.0:48234 0.0.0.0:* LISTEN 4050/rpc.statd </font></div><div><font face="Courier New">tcp 0 0 :::111 :::* LISTEN 3402/rpcbind </font></div><div><font face="Courier New">tcp 0 0 :::8080 :::* LISTEN 20582/java </font></div><div><font face="Courier New">tcp 0 0 :::48081 :::* LISTEN 20573/java </font></div><div><font face="Courier New">tcp 0 0 :::48082 :::* LISTEN 20573/java </font></div><div><font face="Courier New">tcp 0 0 :::22 :::* LISTEN 4315/sshd </font></div><div><font face="Courier New">tcp 0 0 ::1:25 :::* LISTEN 1119/master </font></div><div><font face="Courier New">tcp 0 0 :::35675 :::* LISTEN - </font></div><div><font face="Courier New">tcp 0 0 :::8443 :::* LISTEN 20582/java </font></div><div><font face="Courier New">tcp 0 0 :::5435 :::* LISTEN 20573/java </font></div><div><font face="Courier New">tcp 0 0 :::34077 :::* LISTEN 4050/rpc.statd </font></div><div><font face="Courier New">tcp 0 0 :::10050 :::* LISTEN 19024/zabbix_agentd </font></div><div><font face="Courier New">udp 0 0 0.0.0.0:111 0.0.0.0:* 3402/rpcbind </font></div><div><font face="Courier New">udp 0 0 10.103.20.29:123 0.0.0.0:* 4447/ntpd </font></div><div><font face="Courier New">udp 0 0 127.0.0.1:123 0.0.0.0:* 4447/ntpd </font></div><div><font face="Courier New">udp 0 0 0.0.0.0:123 0.0.0.0:* 4447/ntpd </font></div><div><font face="Courier New">udp 0 0 0.0.0.0:834 0.0.0.0:* 4050/rpc.statd </font></div><div><font face="Courier New">udp 0 0 0.0.0.0:38979 0.0.0.0:* 4050/rpc.statd </font></div><div><font face="Courier New">udp 0 0 0.0.0.0:42334 0.0.0.0:* - </font></div><div><font face="Courier New">udp 0 0 0.0.0.0:609 0.0.0.0:* 3402/rpcbind </font></div><div><font face="Courier New">udp 0 0 :::111 :::* 3402/rpcbind </font></div><div><font face="Courier New">udp 0 0 fe80::5477:49ff:fe7d:d45:123 :::* 4447/ntpd </font></div><div><font face="Courier New">udp 0 0 ::1:123 :::* 4447/ntpd </font></div><div><font face="Courier New">udp 0 0 :::123 :::* 4447/ntpd </font></div><div><font face="Courier New">udp 0 0 :::42649 :::* 4050/rpc.statd </font></div><div><font face="Courier New">udp 0 0 :::45386 :::* - </font></div><div><font face="Courier New">udp 0 0 :::609 :::* 3402/rpcbind </font></div><div><font face="Courier New">==============================================</font></div><div><font face="Courier New">Done</font></div><div><font face="Courier New">[root@</font></div></div><div><br></div><div><br></div><div><br></div><div><br></div></body></html>