Orion Module Engine service bouncing after applying IPAM 4.3 Hotfix 1

Overview

This article describes the issue about the Module Engine Service constantly restarting after applying the IPAM 4.3 Hotfix 1.

When the hotfix is installed and the services restart, you see that the Orion Module Engine service constantly restarts and the website displays an HTTP 500 error when trying to open. Upon further review, you see the following errors in the BusinessLayerHost.log and Core.BusinessLayer.log files under ProgramData\Solarwinds\Logs\Orion:

BusinessLayerHost.log:

2016-01-21 16:55:35,036 [7] ERROR SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Plugin "IPAM Business Layer" failed to start.
System.Runtime.Serialization.SerializationException: Type 'System.ServiceModel.Channels.ReceivedFault' in Assembly 'System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' is not marked as serializable.
Server stack trace: 
   at System.Runtime.Serialization.FormatterServices.InternalGetSerializableMembers(RuntimeType type)
   at System.Runtime.Serialization.FormatterServices.GetSerializableMembers(Type type, StreamingContext context)
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitMemberInfo()
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter, SerializationBinder binder)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
   at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.SerializeMessageParts(ArrayList argsToSerialize)
   at System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage..ctor(IMethodReturnMessage mrm)
   at System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage.SmuggleIfPossible(IMessage msg)
   at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoDispatch(Byte[] reqStmBuff, SmuggledMethodCallMessage smuggledMcm, SmuggledMethodReturnMessage& smuggledMrm)
   at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatchCallback(Object[] args)
Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at SolarWinds.BusinessLayerHost.Contract.BusinessLayerPlugin.Start()
   at SolarWinds.BusinessLayerHost.PluginInstanceAppDomain.Start()

Core.BusinessLayer.log:

2016-01-21 16:54:39,047 [7] FATAL SolarWinds.Orion.Core.BusinessLayer.CoreBusinessLayerPlugin - Unhandled Exception caught in plugin startup.
System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: RunQuery failed, check fault information. (Fault Detail is equal to InfoServiceFaultContract [ SolarWinds.Data.Query.EntityNotFoundException: Source entity [Orion.Package] not found in catalog
   at SolarWinds.Data.Query.Processor.SemanticChecker.HandleSourceEntityExpression(SourceExpression sourceExpression)
   at SolarWinds.Data.Query.Processor.SemanticChecker.Visit(SourceExpression sourceExpression)
   at SolarWinds.Data.Query.Swql.ParseTreeDuplicatorNoState.Visit(FromClause from)
   at SolarWinds.Data.Query.Processor.SemanticChecker.Visit(SelectStatement statement)
   at SolarWinds.Data.Query.Processor.SemanticChecker.Process(SelectStatement statement)
   at SolarWinds.Data.Query.Processor.PreProcessor.ProcessInternal(PreProcessorState& preProcessorState)
   at SolarWinds.Data.Query.Engine.QueryProcessor.ProcessInternal(String query, IQueryExecutionContext context)
   at SolarWinds.InformationService.Core.QueryPlanCache.GetQueryPlan(String query, String username, Func`1 buildQueryPlan)
   at SolarWinds.InformationService.Core.InformationService.RunQuery(String...).

Environment

IPAM version 4.3 after applying Hotfix 1

Cause 

The cause of this issue is that the businesslayer host is attempting to load the plugins, but found a duplicate schema for IPAM under SWIS. Typically, this happens because the hotfix requires the replacement of the file IPAM.Core.xml under the SWIS schema directory, but the original file is still present. Usually, individuals remain the old file to IPAM.Core - Copy.xml which even though it changes the old file name, it's still a .xml with somewhat the same name structure that makes SWIS to attempt to load it.

The loading of both the xmls causes the core businesslayer to crash due to the conflict, which results in the module engine service constantly bouncing.

Resolution

1. Navigate to Program Files (x86)\SolarWinds\Orion\Information Service\3.0\Schemas.
2. Find the modified file for IPAM.Core.xml that is still present. 
3. Remove it from the directory.
4. Restart the services.

For more information about this issue, click the link below:

https://solarwinds-prod.mindtouch.us...already_exists