Network Management

VNQM TroubleShooting CDR/CMR Polling Issues

Device detected as CUCM => SNMP is working, both AXL and FTP credentials are correct, but no activity from VNQM side is visible in FTP/SFTP server console/log or in IpSlaMFtp.Job_*.log

First published date

11/8/2018 11:50 AM

Last published date

10/16/2025 4:31 PM

Overview

Symptoms: Device detected as CUCM  => SNMP is working, both AXL and FTP credentials are correct, but no activity from VNQM side is visible in FTP/SFTP server console/log or in IpSlaMFtp.Job_*.log

Product section

VoIP & Network Quality Manager

Cause

Cluster name or cluster node ID (CTIID) is not detected.
Files are not uploaded to the FTP directory.
Sequence Numbers are lower than numbers in the database.

Resolution

Check VoipCCMMonitoring table 

  • ClusterName and CcmName columns NULL => SNMP issue, check IpSlaM.Snmp.Job_*.log for errors. In most cases
    • In most cases SNMP timeout issue - see bellow
  • ClusterNodeID columns is NULL, CcmName and ClusterName not NULL=> AXL issue, check IpSlaM.Axl.Job_*.log for errors.
    • Haven’t seen yet
  • ClusterNodeID, CcmName and ClusterName are all NULL=> SNMP and AXL issue, check  IpSlaM.Snmp.Job_*.log, IpSlaM.Axl.Job_*.log for errors.
     
  • The general remedy to make the system running and downloading CDR/CMR (FTP must be set correctly) even if there are still issue with SNMP or AXL
    • use the application AXLTestDriver.exe enter any CCM in a cluster to the following AXL query (select proper AXL API version based on CCM version 6,7 should work with v10, 8+ should work with V80, 8.5+ should work with V85)

      SELECT ctiid as ctiid, c.Name as ccmname, p.Name as hostname FROM CallManager c JOIN ProcessNode p ON c.fkprocessnode=p.pkid

  • take cctid as a ClusterNodeID, ccmname as CcmName, find out the ClusterName  check e.g. ftp files or Call Manager settings
  • insert all the ClusterNodeID, CcmName, ClusterName values into the VoipCCMMonitoring
  • once the information is stored to the db FTP polling would be started in a while

Issue: SNMP Timeout 

Symptoms: CDR polling haven't started, Phone table incomplete

Cause: 

Low priority of Cisco Call Manager SNMP Service process

Resolution: 

  • avoid/reduce parallel SNMP request from other application (SAM, 3rd party apps)

  • tuning up SNMP polling

    • increase CallManager Polling Interval on VNQM Settings page (UI), by default it's 5 minutes (Module Engine restart needed)
    • CCMPhoneBatchSize, CCMMaxPhones in SolarWinds.Orion.IpSla.BusinessLayer.dll.config (Module Engine restart needed)
      • by default 500 phones per one job, 5 jobs in total (0-500, 500-1000, 1000-1500,1500-2000, the rest)
      • ask for the approximate number of phones in the system, set CCMMaxPhones to little bit bigger value. By decreasing CCMPhoneBatchSize, task can be distributed to more smaller and faster jobs
    • SnmpJobSnmpTimeOut in SolarWinds.Orion.IpSla.BusinessLayer.dll.config (Module Engine restart needed) - timeout of SNMP requests, 30 sec by default

Issue: CDR/CMR polling started but no calls found 

Symptom: Device detected as CUCM  => SNMP is working, both AXL and FTP credentials are correct activity from VNQM side is visible in FTP/SFTP server console/log but search page returns no results

Cause 1: FTP directory is incorrect. An error will be logged such as:
WeOnlyDo.Exceptions.FtpDLX.ServerException: Server returned an error: No such file or folder

How to solve: Full path from FTP root has to be provided, not path from home directory – could be detected in (S)FTP server logs or console or IpSlaMFtp.Job_*.log

Cause 2: The CUCM is publisher node or backup subscriber without any call activity, no files of <cdr/cmr>_<clusterName>_<nodeId>_... format exists on the (S)FTP server

How to solve: Add also other cluster nodes to the VNQM including AXL and FTP credentials (possible complaints about providing AXL access to subscribers)

Symptom: AXL credentials couldn't be validated. Same credentials work for other CUCM but not for all.

Cause: Send Valid Namespace in AXL Response were set to False on troubled CUCM.

How to solve: Follow instruction from this doc: https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/devguide/7_1_2/AXL_DevGuide/axl.html

Issue: Missing Files

Symptoms: No CMR or CDR files are seen on the FTP server
Cause: Files are not being uploaded to the FTP share.
How to solve: Confirm the call manager is uploading files to the correct FTP server and directory and nothing else is deleting files from the FTP server.

Issue: Missing Regions 

Symptoms: some calls haven't region assigned  - visible on VoIP Search page
Cause 1: Not all devices polled yet
How to solve: SNMP issue – check IpSlaM.Snmp.Job_*.log for errors, see SNMP Timeout upper

Cause 2: The calling/called party is unknown device for VNQM, e.g. SIP Trunk, We're polling only information about VoIP Phones, MGCP Gateways and H323 devices

Issue: Incorrect Region assigned to the call 

Symptom: Region is assigned to the call but seems to be incorrect

Cause: Verify that what customer is talking about is really Region, not Location - could be mistaken

Issue: Missing CMRs 

Symptom: some calls haven't call quality information (Jitter, PacketLoss, Latency, MOS)

Possible causes:

  • Only IP Phones and MGCP gateways provides call quality metrics, if the calling/called party is e.g. SIP trunk or any other device, then we display unsupported
  • Call Diagnostics Enabled CUCM Service Parameter is not set
  • Zero length duration calls (failed, busy, ...) don't contain voip quality metrics

How to solve: investigate particular call details - see bellow

How to get more details about one particular CDR 

Show and search by Global Call Id if there is some issue/question with one particular call

Run following SQL query against Orion db to get all details we have in the db, and get  the result for the further investigation

SELECT cdr.*,cdrd.*,cmro.*,cmrd.*
FROM VoipCDRs cdr
LEFT JOIN VoipCDRDetails cdrd ON cdr.Pkid=cdrd.Pkid
LEFT JOIN VoipCMRs cmro ON
  cdr.VoipCCMMonitoringID=cmro.VoipCCMMonitoringID AND   cdr.GlobalCallID_callId = cmro.GlobalCallID_callId AND   cdr.OrigLegCallIdentifier = cmro.CallIdentifier
LEFT JOIN VoipCMRs cmrd ON
  cdr.VoipCCMMonitoringID=cmrd.VoipCCMMonitoringID AND
  cdr.GlobalCallID_callId = cmrd.GlobalCallID_callId AND
  cdr.DestLegIdentifier = cmrd.CallIdentifier
WHERE cdrd.GlobalCallID_callId=85810

Localize appropriate CDR and CMR file containing data about the call - get call timestamp (DateTimeDisconnectUTC column) - can be used only if VNQM is not deleting files from the FTP server

In Ftp directory search for the first or second file with timestamp just after the cdr timestamp, to recognize right this search for call ID in it

To get CMR file search for the file with the same name except for the prefix (cmr instead of cdr)

No CDR/CMR polled - Reset CCM sequence number 

Symptom: all is set as it should be, it was recently working but then it stopped (there was an upgrade/maintenance/reconfiguration of CCM happening in close past)

check the FTP location where the files are downloaded from and look at the sequence number

files look following way: cdr|cmr_[clustername]_[clusterID]_[timestamp]_[sequencenumber]

check the DB table VoipCcmFtpConnectionInfo and compare with columns CDRSequenceNumberToDeleteFrom, CDRSequenceNumberToDownloadFrom, CMRSequenceNumberToDeleteFrom, CMRSequenceNumberToDownloadFrom

if those number especially CDRSequenceNumberToDownloadFrom, CMRSequenceNumberToDownloadFrom are higher then it means the sequence number was reset and the reset of the sequence has to be performed in the VoipCCMFtpConnectionInfo table as well. Please reset all 4 indicated number (set them to NULL or 0)