Performance Tuning in Web Help Desk to fix slow performance

The Web helpdesk application may experience poor performance such as slow navigation of the management console. Usually, the symptoms are:

·       Experiencing performance issues creating tickets, adding tech notes/ticket updates in WHD,

·       When updating and clicking Save&Email Button, it takes 30 seconds for the page to load,

·       Web Help desk stopped processing emails into tickets,

·       Getting a daemon activity error that displays in the Web Help Desk console.

Error: "No daemon activity detected. Check that at least one instance of the application has been configured as a dedicated or background daemon.”

·       Tickets are periodically not saving

·       JVM Memory spikes and reaches 90 to 100% causing the application to become unresponsive

·       Slow Search results

·       Slow running of Reports

·       Action Rules and Tasks are not processing automatically.

Web helpdesk resources are not properly configured or the Application is corrupted.

1.  Check Web help desk Global Settings:

a. Navigate to Setup > General > Options and adjust the following settings:

·         Max Items Per Page: Set to 25.

·         Max Weeks of Data in Dashboard: Set to one week.

·         Max Search Results. Set to 250

b. Navigate to Setup>General>System Information and check if JVM memory is not reaching more than 70%. If yes, please increase the JVM memory

c. Make sure that the Log Level was set to Info. Note that setting the log into debug messages will record greater details that will affect the performance of the Web Help Desk.

d. Navigate to Setup>E-mail>Options. Set Scheduled E-Mail Delivery Threshold to 0 and save.

2. Check if you have a scheduled Jobs and Synchronization. Make sure that Auto-Sync Schedule is not running during Production that will affect the performance.  We recommend setting up during midnight/non-production hours:

a. Go to Setup>Clients>AD/LDAP Connections. Under Advanced section, revisit and modify Auto-Sync Schedule,

b. Go to Setup>Assets> Discovery Connections. Revisit and modify Discovery and Auto-Sync Schedule, &

c. Go to Reports>Report Schedules

3. Check the Web Helpdesk Database Performance that is causing impact with the application. This step needs to be done inside the Database Server.

a. Delete the Ghost Session that consumes process and affects performance

·         Stop WHD Services

·         Run the query below:

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

SELECT * FROM ACTIVE_SESSION;

SELECT * FROM JOB_TICKET_ACTIVE_SESSION;

·         If there are ghost session, run the following query to remove it:

Note: Before executing this step, make sure to create a database backup

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

DELETE FROM ACTIVE_SESSION;

DELETE FROM JOB_TICKET_ACTIVE_SESSION;

b. Lower the number of Dashboard Widget per Tech

We recommend having the techs dashboard widget lower than 4.

·         Run the query below that will show techs having dashboard widget display from their account more than 4.

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

SELECT foo.*,tech.user_name FROM (SELECT COUNT (id) cnt, tech_id
FROM dashboard_widget
GROUP BY tech_id) as foo
INNER JOIN tech ON foo.tech_id = tech.client_id
WHERE foo.cnt > 4;

c. Delete all messages across all techs as it will also affect the performance of WHD UI/Web console.

Note: Before performing this step, please verify and ask permissions from the customers. There are some instances that they need this data for business audit requirements. Don’t forget to perform a database backup

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

DELETE FROM TECH_GROUP_MESSAGE;

DELETE FROM READ_MESSAGE;

DELETE FROM CLOSED_MESSAGE;

DELETE FROM MESSAGE;

d. Check if there are Tickets Looping

·         Run the following script in the database to list all tickets that are stuck in infinite loop: 

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

SELECT F.JOB_TICKET_ID AS FIRST_TICKET_ID,F.MERGED_PARENT_TICKET_ID AS FIRST_TICKET_PARENT_ID,
S.JOB_TICKET_ID AS SECOND_TICKET_ID, S.MERGED_PARENT_TICKET_ID AS SECOND_PARENT_ID,
HE.ENTRY_TEXT, HE.ENTRY_DATE FROM job_ticket as F
JOIN JOB_TICKET as S ON S.MERGED_PARENT_TICKET_ID=F.JOB_TICKET_ID AND F.MERGED_PARENT_TICKET_ID = S.JOB_TICKET_ID
JOIN HISTORY_ENTRY AS HE on F.JOB_TICKET_ID = HE.JOB_TICKET_ID
WHERE HE.ENTRY_TEXT LIKE 'Merged in%' ORDER BY HE.ENTRY_DATE DESC;

·         If there is a parent ticket ID, this can also be examined in the database by running the following query: 

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

SELECT MERGED_PARENT_TICKET_ID FROM JOB_TICKET WHERE JOB_TICKET_ID = <Put Parent Ticket ID here>;

This will return the parent ticket ID.

·         Once the proper parent ticket is identified, run the following query to break the loop behavior: 

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

UPDATE JOB_TICKET SET MERGED_PARENT_TICKET_ID = NULL WHERE JOB_TICKET_ID = <parentID>;

·         Restart Web Help Desk.

4. Check the settings of Web Helpdesk Configuration Files and physical resources

a. Revisit the Recommended System Requirements

Unsupported or non-compliant system requirements will also be a factor in performance issues

b. If your Web helpdesk has a larger number of techs and if you are using external databases such as MS SQL/MySQL, you need to increase the value of the DATABASE_CONNECTIONS. One connection for every five simultaneous users is a conservative ratio.

·         Remote into WHD Server

·         Open the whd.conf file and navigate to Database Connections section:

Windows Server: C:\Program Files\WebHelpDesk\conf

Linux: cd /usr/local/webhelpdesk/conf

Apple OS X: cd /Library/WebHelpDesk/conf

·         Save

·         Restart WHD Services
Note: If your database is PostgreSQL, follow Database Connections and PostgreSQL Max Connection in Web Help Desk

 

5. Other External Factors that will affect the Web Helpdesk Performance

a. External Databases (MySQL/MS SQL) and Web Helpdesk Application are sitting on the same server

b. Scheduled Automatic SQL Database Backup

c. Verify if there are any databases tied up with the Database Server where Web helpdesk is configured. Used the MS SQL Reports to check which Database container consuming the CPU and IO of Database application

Note: Applicable for External Databases
 

If all Steps above did not resolved the issue, performed clean installation (uninstall/install of Web helpdesk and restore the config files).

Pre-Implementation Steps
1. Create a Database Backup
2. Create a WHD Server Snapshot
3. Create a copy of conf folder

Implementation Steps:
- Remote into WHD Server
- Stop WHD Services

1. Uninstall WHD Application

2. Install WHD Application

3. Stop WHD Services and restore the configuration files and database
- On the target server, navigate to the webhelpdesk directory.
- Copy the conf directory files from the original server to the same location on the target server.
- For PostgreSQL database, check the link on how to restore the database.

4. Start WHD Services