Submit a ticketCall us
Home > Success Center > Virtualization Manager (VMAN) > VMAN - Knowledgebase Articles > Status of vCenter/ESXi virtual machines switches between green and gray every few minutes

Status of vCenter/ESXi virtual machines switches between green and gray every few minutes

Updated December 6, 2018

Overview

 

Status of vCenter/ESXi virtual machines switches between green and gray every few minutes

Environment

  • Virtualization Manager
  • vCenter and ESXi hosts

 

Cause 

 

This issue can happen when the following conditions are met:

  • A vCenter is set to be polled through Orion VMAN polling while some of the ESXi host children of that vCenter are set to be polled through Basic polling.
  • The permissions for a user are incorrectly set up in a conflicting manner as follows in two different possible scenarios:

 

Conflicting roles in settings set manually

For the vCenter:

- the user is assigned a specific role with at least read access (for example, "Administrator")

 

For a child ESXi host:

- A conflicting role is assigned for the user, such as "No Access"

 

 

- or - 

 

 

Conflicting roles in settings with the propagate to children option

For the vCenter:

- the propagate to children option is checked -- this option will override all other options for child entities, including VMs under an ESXi host with a "No Access" permission.

 

For a child ESXi host:

- the propagated role (in this case, "Administrator") is manually removed or its access level is changed manually to "No Access". In this case, it no longer matters if the "Propagate to children" option is checked or not, because the manually assigned role will persist in child VMs.

 

 

The resulting symptom of switching green and gray status is caused by:

  1. Orion VMAN polling to the vCenter reads "No access" from the ESXi host's credential for child VMs, resulting in gray status for these VMs.
  2. A short time later, the Basic polling to an ESXi host child of the affected vCenter reads "Administrator", resulting in green status.
  3. The behavior will then oscillate back and forth as Orion VMAN polling and Basic polling alternates.

 

Correct and incorrect permission states chart visualization:

permission-states.png

 

Confirming the issue

 

The most reliable way to identify this type of issue is to check the XML results from JavaWorker (content of VMwareJobResultsOutput folder).

 

There we can see results of direct (basic) polling of hosts containing all information and results of VMAN Orion polling of vCenter containing all information about vCenter,

 

When this issue is occurring, you will also see the data for virtual machines where all fields contain NULL except for the "Name" field, which corresponds to virtual machine's name.

 

These NULL fields are then causing the following error messages in Collector.Service.log:

 

Check the Collector.Service.log file found in C:\ProgramData\SolarWinds\Collector\Logs\

 

If the following error is present in Collector.Service.log, this issue is confirmed:

ERROR SolarWinds.Collector.PollingController.PollingController - Resolve polling plan failed!! We will try to resolve it in next round
System.NullReferenceException: Object reference not set to an instance of an object.
at SolarWinds.VIM.Collector.PollingInfoProviders.Vanilla.VMwareBasePollingInfoProvider.CreateJob()

Resolution

 

Fix the permission settings so that the permissions are set correctly for the entire virtual environment:

 

  1. Correctly set the user permissions for affected vCenters and children ESXi hosts. The permissions must agree with no conflict. For example, a user cannot be given an "Administator" role for a vCenter while also having a "No Access" role for a child ESXi host.
  2. Permissions must be set correctly for the entire virtual environment. For example, if you want to prevent user's access to a specific virtual machine, the user's access must be prevented on all levels: for the parent entities as well as the particular virtual machine.

 

 

Last modified

Tags

Classifications

Public