Applications Systems

[SAM] Node hung (unknown status) causes port exhaustion

Every few days, the Orion Server 'hangs'. After it was rebooted, the Orion Server would operate normally, but then would hang again a few days later. The problem is port exhaustion. The cause is unknown.

First published date

11/13/2018 4:37 PM

Last published date

11/13/2018 4:37 PM

Overview

Every few days, the Orion Server 'hangs'. After it was rebooted, the Orion Server operates normally but then would hang again a few days later. The problem is port exhaustion. The cause is unknown.

Environment 

  • SAM 6.3 — 6.6
  • Orion Platform 2016.2 — 2017.3

If you encounter this issue in SAM 6.6.1 running on Orion Platform 2018.2, see Core 2018.2 Port Exhaustion Issue - Server restarts required .

Product section

Server Application Monitor

Resolution

  1. Once the Orion server begins to slow down, collect diagnostics. Diagnostics in SAM 6.3 and above will get a netstat -aon. However, if netstat -aon is NOT available, run the netstat command from the DOS prompt and pipe it to an output file such as netstat -aon > netstat.txt
  2. Look at the 3rd column of the netstat output. If the same IP address appears repeatedly, find out what is wrong with the server at that IP address.
 

For example, in one case, pinging a server from the Orion Server did not respond, and the node was listed as Unknown. After the node was rebooted, the ports to the server released as normal, and the Orion server performance stayed within limits.