Submit a ticketCall us

Systems Monitoring for Dummies
Our new eBook will teach you the fundamentals and help you create monitors and alerts that are effective, meaningful, and actionable. Monitoring is more than a checkbox on your to-do list. This free eBook will give you practical advice to help you succeed in all aspects of monitoring – discovery, alerting, remediation, and troubleshooting. Don’t miss out on this indispensable resource for newbies, experienced IT pros, and everyone in between. Register Now.

Home > Success Center > Network Performance Monitor (NPM) > RPC errors

RPC errors

Table of contents

Updated August 24, 2016

Overview

There have been a few posts and cases recently where people have been running into issue with RPC in some form or other. This article provides brief information on RPC, some troubleshooting and where to look for more information. Microsoft is the best source of information for RPC. Most of the details in this article come from Microsoft's Website: Technet.

Environment

  • All SolarWinds products using RPC / WMI
  • All Windows versions 

Detail

Errors and Fixes

 

1. The RPC Server is Unavailable

This error occurs when the remote server is unable to receive or respond to RPC Requests.

 

Verify there is no firewall preventing the Orion host from performing RPC calls. A simple method is to verify with an administrator account that you can connect the the remote computer through Computer Management (compmgmt.msc) or that Windows file sharing is possible.

  • When checking the firewall/ACLs also check your anti-virus, some AV systems have firewalls built in.
  • Windows RPC will operate over a large range of ports depending on the method of RPC in use, by default port 135 is used and the RPC endpoint mapper then will offload the request to the ports the service is running on.

 

 

Service Name UDP TCP

UDP

TCP

HTTP

80, 443, 593

80, 443, 593

Named Pipes

445,137,138,139

445,137,138,139

RPC Endpoint Mapper

135

135

Services

ephermeral ports

ephermeral ports

 

Sources: Microsoft Service overview and network port requirements for WindowsSolarWinds port requirements.  (© 2016 Microsoft, available at https://support.microsoft.com, obtained on August 23, 2016.)

For SAM RPC monitors the majority will be using Named Pipes. 

 

It also goes without saying to insure the RPC service is started on the remote computer but you will also need to check the Remote Registry Service.

  • Check DNS resolution (and NetBIOS).
  • Check Windows File and printer sharing is enabled.
  • If all fails, check your Windows System logs.

 

2. Access is denied

This is a pure permissions error, adding the user account being used on the RPC client to the administrators group on the remote computer (the RPC server) will clear this. In newer versions of Windows there is a Performance Monitor Users" group you can use in most cases. If this gives problems you can also look at using SC (User permissions for Windows Performance Counters over RPC. has more detail.).

 

3. Network Path not found

This points to permissions again but to an issue on the server itself, The Network Path Was Not Found errors.

 

4.  The specified object is not found on the system

This can be as easy as restarting the Remote Registry service. To confirm this will fix try find the counters SAM is polling in perfmon when connecting from the Orion server and another remote server. If they don’t display remotely but do display locally then the problem is with the remote registry service or registry permissions.

 

5. RPC server is too busy to complete this operation

This is tricky, it is likely resources or just plain load on the server. If you’ve Exchange running on the server with this error then you are going to be looking to get it fixed quickly. Error Message:  The RPC server is too busy to complete this operation might help but if you are the admin the link mentions, well... :/  (© 2016 Microsoft, available at https://technet.microsoft.com, obtained on August 23, 2016.)

 

 

For more in depth troubleshooting:

 

If the above information do no help, then you are looking at a Microsoft Support case or at improving your understanding of RPC so you can troubleshoot further.

 

General Information on RPC

 

RPC (Remote Procedure Call) is an open standard, the OSI and DCE were involved in the protocol definitions and RFC 5531 gives the protocol specifications. RPC is a client/server model implementing a method of initiating a procedure on one device (client) and having the procedure process and return information from another device (server).

 

To do this RPC needs to locate the remote server, connect to it, authenticate and make a request the server can understand and process. The server will then respond, the client will consider the call as completed and continue, processing the information it received.

 

To locate the remote server RPC relies on DNS or WINS.

 

To connect to the server RPC will use, TCP, UDP, HTTP, HTTPS, Named Pipes, Winsock or other configured method (I'm pretty sure I don't know them all). RPC itself is totally unconcerned with the network, reliability and routing and it leaves this to the lower layers. This is why you can get RPC over TCP, over UDP, etc.

The client will initially connect to RPC on port 135, the client will be speaking to the endpoint mapper on this port. The endpoint mapper will direct the client to the correct port for the service it is trying to access (the client can and does cache this information). Services can run on any port but are often on the user ports 1025-5000 or 49152-65535 depending on the version and patch level of your server.  (© 2016 Microsoft, available at https://blogs.msdn.microsoft.com, obtained on August 23, 2016.) These ports are often known as the ephermeral (© 2016 Wikipedia, available at https://en.wikipedia.org, obtained on August 23, 2016.) ports and some anti-virus systems can limit or restrict this, most will log to the system or application log if they do and most will have a KB on configuring the anti-virus so they don't filter/limit the ports.

 

 

For authentication Microsoft RPC uses Kerberos or NTLM authentication. The method is configured through group policy on a whole server level, in general you shouldn't have to adjust the authentication method unless it is changing across your environment.

 

If you are having any authentication issues in your environment it is likely to affect RPC and if you are having intermittent authentication issue RPC is likely to have them also. Some for the further reading links point to running dcdiag for these sorts of problems in domain environments.

 

When it comes to permissions needed it depends on what you are using RPC for, The Network Path Was Not Found errors covers the permissions needed for SAM's default monitors.

 

 

So, what is RPC used for? Well pretty much everything, RPC is a good technology and much of Windows relies on it. While many services depend on RPC, Exchange, DCs and SQL are probably where people run into RPC errors that cause them the most headaches. Exchange provides performance counters to review how many RPC requests it is getting and how many are outstanding. SAM's default Exchange monitors include these by default (Performance and Scalability Counters and Thresholds has more information on the counters).  (© 2016 Microsoft, available at https://technet.microsoft.com, obtained on August 23, 2016.)

 

Besides Exchange, How to Test Microsoft Remote Procedure Call Performance details Microsoft's recommended counters for performance monitoring of RPC and if you are having a problem it can be worth monitoring these along with the Windows Event logs.  (© 2016 Microsoft, available at https://support.microsoft.com, obtained on August 23, 2016.)



Disclaimer: Please note, any content posted herein is provided as a suggestion or recommendation to you for your internal use. This is not part of the SolarWinds software or documentation that you purchased from SolarWinds, and the information set forth herein may come from third parties. Your organization should internally review and assess to what extent, if any, such custom scripts or recommendations will be incorporated into your environment.  You elect to use third party content at your own risk, and you will be solely responsible for the incorporation of the same, if any.
 

 

 

Last modified

Tags

Classifications

Public