Submit a ticketCall us

Systems Monitoring for Dummies
Our new eBook will teach you the fundamentals and help you create monitors and alerts that are effective, meaningful, and actionable. Monitoring is more than a checkbox on your to-do list. This free eBook will give you practical advice to help you succeed in all aspects of monitoring – discovery, alerting, remediation, and troubleshooting. Don’t miss out on this indispensable resource for newbies, experienced IT pros, and everyone in between. Register Now.

Home > Success Center > Server & Application Monitor (SAM) > Microsoft Azure returns Status Code 503 Service Unavailable error when adding cloud account with 2000 VMs

Microsoft Azure returns Status Code 503 Service Unavailable error when adding cloud account with 2000 VMs

Updated November 27, 2017

Overview

Per the Cloud Infrastructure Monitoring feature description, SolarWinds recommends a limit of 1000 cloud VMs/instances for optimal performance. If a Microsoft Azure account includes over 1000 VMs, the Orion Platform may receive a Status Code 503 - Service Unavailable error, as recorded in the VMwareJobs log file stored in the SolarWinds\Logs\VIM\Jobs folder.  

Environment

  • SAM 6.5 

Cause 

The Status Code 503 - Service Unavailable error is specific to batch operation, as recorded in the SolarWinds\Logs\VIM\Jobs log file.  Here is an excerpt of the VMwareJobs log file:

*** Assembly SolarWinds.CloudMonitoring.Azure, Version=1.1.0.258, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
2017-11-16 09:13:17,583 [STP SmartThreadPool Thread #0][(null)] INFO  Azure.AccountAutomonitoringPoller - Automonitoring Polling started
2017-11-16 09:14:03,812 [13][(null)] ERROR Azure.AzureClient - One of the resquests in batch has failed with status code: 503.
2017-11-16 09:14:05,261 [13][(null)] ERROR Azure.AzureClient - One of the resquests in batch has failed with status code: 503.
2017-11-16 09:14:39,351 [STP SmartThreadPool Thread #0][(null)] INFO  Azure.AccountAutomonitoringPoller - Polled 1491 instances
2017-11-16 09:14:39,357 [STP SmartThreadPool Thread #0][(null)] ERROR Azure.AccountAutomonitoringPoller - Unable to get azure VMs.
SolarWinds.CloudMonitoring.Azure.Contract.Exceptions.AzureClientException: StatusCode: 503
   at SolarWinds.CloudMonitoring.Azure.AzureClient.<GetAsync>d__29`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SolarWinds.CloudMonitoring.Azure.AzureClient.<GetVirtualMachinesState>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SolarWinds.CloudMonitoring.Azure.AzureClient.<GetInstancesDetailsAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SolarWinds.VIM.Pollers.Cloud.Azure.AccountAutomonitoringPoller.<CollectCloudInstancesAsync>d__12.MoveNext()
*** Assembly SolarWinds.VIM.VMware.Jobs, Version=8.1.0.6400, Culture=neutral, PublicKeyToken=null, .NET version v4.0.30319 ***
2017-11-16 09:15:38,189 [STP SmartThreadPool Thread #0][(null)] INFO  Azure.AccountAutomonitoringPoller - Creating azure client...
2017-11-16 09:15:38,566 [STP SmartThreadPool Thread #0][(null)] INFO  Azure.AccountAutomonitoringPoller - Automonitoring Polling started
2017-11-16 09:15:42,258 [JobMonitor][(null)] INFO  WorkerProcess.JobMonitor - Job 2a019a21-7a3f-43b9-9876-35262463c0aa timeout after 00:00:04 
2017-11-16 09:15:42,263 [JobMonitor][(null)] WARN  WorkerProcess.WorkerExecutionEngine - Job timeouted, cancelling job

Resolution

To work around this issue, either reduce the number of monitored Azure VMs or follow these steps to halt batch operations:

  1. In the Orion Web Console, navigate to the Global tab on the Advanced Configuration page  (http://localhost/Orion/Admin/AdvancedConfiguration/Global.aspx).
  2. Disable the SendAzureRequestsInBatch setting.
  3. Increase the AzureOneTimeJobTimeout value from 00:03:00 to 00:08:00 to allow extra time to add Azure accounts when SendAzureRequestsInBatch is disabled.

 

Last modified

Tags

Classifications

Public