Force outgoing traffic in Orion High Availability environment to use VIP only

4/12/2019 7:22 PM

10/28/2022 10:49 AM

Orion High Availability (HA) is designed to provide uninterrupted access to the Web Console as well as incoming traffic (i.e. NetFlow, Syslog, Traps, Agents). For this purpose, HA allows using Virtual IP address (VIP), Virtual Host Name (VHN) or both, depending on the environment where HA is implemented.

But from the other perspective of monitoring the outgoing traffic can utilize any of IP addresses associated with the HA pool - VIP (if configured) or any of IP addresses of pool members. That means devices have to answer polling queries (i.e. SNMP) coming from any of these three IP addresses. In most environments, it does not pose any trouble as devices are unlocked and allow traffic from any sources. But in some cases, devices are locked down to the certain IP address, mostly VIP which in many cases is the IP address of the initial Orion server preceding HA implementation, and would result in false monitoring results (packets being dropped/refused, data not returned).

Orion Platform

Behavior results from the way operating system (Windows) makes the decision which IP address to use as a source and prioritizes the first one on the list of available IP addresses.

Resolution 1:
We recommend following the simple solution described in Which IP address is used as the source when using a VIP? (solarwinds.com) before picking the default gateway IP, VIP and Primary /Secondary Poller IP's for SolarWinds Servers.

If the above solution is not feasible, then you can go for solution 2

Resolution 2:
The Windows implementation of the TCP/IP stack provides a way of letting the system know which IP addresses can be skipped during the decision-making process. For that purpose, each IP address has a property - SkipAsSource - which can be modified on the fly and immediately affect the way outgoing traffic is sent out. 

One of the easiest ways of doing it is to use PowerShell in conjunction with Windows Task Scheduler. Below you can find the PowerShell script which:

1. checks if the VIP exists on the server, in the example below 10.160.198.8, and sets its SkipAsSource to False and at the same time sets all remaining IP addresses SkipAsSource to True, which means that Windows will use VIP for outgoing traffic
2. in case VIP does not exist on the server sets all remaining IP addresses' SkipAsSource to False, which means Windows will use any of the IP addresses available for outgoing traffic.

# Scripts are not supported under any SolarWinds support program or service.
# Scripts are provided AS IS without warranty of any kind. SolarWinds further
# disclaims all warranties including, without limitation, any implied warranties
# of merchantability or of fitness for a particular purpose. The risk arising
# out of the use or performance of the scripts and documentation stays with you.
# In no event shall SolarWinds or anyone else involved in the creation,
# production, or delivery of the scripts be liable for any damages whatsoever
# (including, without limitation, damages for loss of business profits, business
# interruption, loss of business information, or other pecuniary loss) arising
# out of the use of or inability to use the scripts or documentation.

<#
.SYNOPSIS
  Script adjusts SkipAsSource setting on IP addresses.
 
.DESCRIPTION
  Adjusting SkipAsSource setting on IP addressess allows Windows to direct traffic using as source IP address for which SkipAsSource is set to False.
 
.INPUTS
  None
 
.OUTPUTS
  None

.NOTES
  Version:        1.0
  Author:         Mariusz Handke
  Creation Date:  2018-08-31
  Purpose/Change: Initial release
#>

$VIP = "10.160.198.8"
$IPS = Get-NetAdapter | Get-NetIPAddress -AddressFamily IPv4 | foreach { $_.ipaddress }
If ($IPS -Match $VIP) {
    foreach ($IP in $IPS) {
        Set-NetIPAddress –IPAddress $IP –SkipAsSource $True

 }
    Set-NetIPAddress –IPAddress $VIP –SkipAsSource $False
} Else {
    foreach ($IP in $IPS) {
        Set-NetIPAddress –IPAddress $IP –SkipAsSource $False
    }
}

Implementing it as an all-the-time running solution:

1. save the above script into the file on the server (i.e. C:\Orion_HA_set_IP_addresses.ps1)
2. using Windows Task Scheduler create the simple task which will execute the above script, be aware that the shortest repetition interval it can be executed is 5 minutes and if you require more frequent execution simply create multiple triggers within the task (i.e. 00:00, 00:01, 00:02, 00:03, 00:04 each one repeated every 5 minutes resulting in execution every minute)

 

Behavior description:

1. when the HA pool is set up with VIP and the pool is enabled, the HA service will assign VIP to the network interface card (NIC) of the active server
2. at this point, all IP addresses have their SkipAsSource set to False
3. when the script executes it adjusts the SkipAsSource property of IP addresses resulting in the active server sending traffic with VIP as the source
4. when failover happens, the HA service removes VIP from the server resulting in a short period of outgoing traffic failure due to the remaining IP addresses set to be skipped
5. when the script executes again (quicker the better) the failover process completes as the IP addresses have now been available to be selected for outgoing traffic
6. at this point, HA completes the process letting the standby server take over, on which the process repeats from 1.

 

Note:

Worth notice is the fact that when using AD-integrated DNS, and NIC(s) on servers set with enabled "Register this connection's addresses in DNS" may result in some unexplained behavior at the DNS. I have seen alias records (A) being removed upon failovers by Windows updating/registering connections in DNS. And sometimes result in Web Console not even allowing to click on buttons, for example when trying to Force Failover.

 

Recommendations:

1. Make sure the "Register this connection's addresses in DNS" option for IPv4 is disabled.
2. Make sure the "Register this connection's addresses in DNS" option for IPv6 is disabled.
3. Make sure the required DNS hosts records (A/AAAA) are manually added to the DNS.

Published on Thwack with test results in comments: https://thwack.solarwinds.com/docs/DOC-203318

Please note that Thwack is a community space where users may post any content as a suggestion or recommendations to you for your internal use. The information set forth herein may come from third-party websites or customers. SolarWinds is not liable for any downtime or any issue that may occur if you perform the following suggestions on the link provided. Your organization should internally review and assess to what extent, if any, such custom scripts or recommendations will be incorporated into your environment.