3/13/2020 8:46 AM

3/19/2020 11:06 AM

One of the customers having issue F5 device when trying to download the configuration in NCM after successful Test connection when its keep on "Downloading......." and will not move further.  Also ‘config download’ job is in progress forever will not complete.

We checked the Session Trace and found the following messages repeatedly. It seems to be in the never-ending loop and I had to stop the download job manually every time.

[03/02/2020 23:18:19.351] Start check prompt for menu-based device

[d796065d] [03/02/2020 23:18:19.444] Process line = <>
[d796065d] [03/02/2020 23:18:19.444] Process line = <>
[d796065d] [03/02/2020 23:18:19.538] -->y
[d796065d] [03/02/2020 23:18:19.632] -->y
[d796065d] [03/02/2020 23:18:19.726] -->y
[d796065d] [03/02/2020 23:18:19.819] -->y
[d796065d] [03/02/2020 23:18:19.913] -->y

Network Configuration Manager

Regarding this ending loop – this is known issue on F5 device.

When it is not able to establish an SCP connection after sending a command like SCP /var/local/NCM.ucs Guest@10.111.215.189:z0xcuz520hs.config

It goes into some strange loop.

So if you try to send something like ‘yes’ after copy command fails you will make device continuously send newer ending ‘y’ character.

F5 device template behavior

When using device templates for F5 devices, consider the following behavior.

Device setup prerequisites

• Set the Terminal Access user on the device to Advanced Shell. Do not use Traffic Management Shell (TMSH).
• Verify the device can reach the NCM server for configuration transfers using SCP.
• If you plan to upload binary user configuration set (UCS) files, the upload can take a long time. Increase the NCM timeouts to have a complete upload report.
  1. Click Settings > All Settings.

  2. Under Product Specific Settings, choose CLI settings.

  3. Set Telnet/SSH Connection Timeout: 360 seconds
  4. Telnet/SSH Prompt Timeout: 300 seconds

Template details

If you set up NCM to transfer configurations using SSH:

• The template can download the current running-config.
• You cannot download the startup-config.
• You cannot upload the running or startup-config.

If you set up NCM to transfer configurations using SCP:

• The template can download and upload a text single configuration file (SCF) as a running-config.
• The template can download and upload a binary UCS as startup-config.
• You cannot upload the running-config to startup, or startup to running-config.

  Downloading or uploading configs using TFTP is not supported.

Regarding this ending loop – this is a known issue on the F5 device.
This device bug can be easily recreated in putty as well. In order to correct this please make sure the device does  have access to the SCP server and it will make a successful connection to the SCP  only then this issue will be resolved 

This is a known issue on F5 devices. When the connection to the SCP server cannot be established after sending the copy command - the device enters into a continuous loop and sending y.

Please verify if the connection to the SCP server works fine. To do so connect to the device in putty and execute the command: ’ssh {NCMServerIPAddress}’ where the place you NCM server IP instead of {NCMServerIPAddress}.

You can always check the SCP connection using the Putty to SSH the SCP ip address it should give you a login screen this will confirm the SCP is UP and running. 

If the SCP service is running you will see a login screen. 



If Network  L2/L3 issues or SCP services is stopped you will receive the Error. 



We need to connect to the F5 devices to make sure they can access the Service once this will be resolved the download will start as normal. 

The below will confirm that we have no issues from the device to SCP transferring the files. 
Using command-line SCP to transfer files  

You can transfer a file between an F5 device and a remote host using command line SCP. To do so, perform the following procedure:

1. Log in to the F5 device/remote host command line.
2. Run SCP by using the following command syntax:

   scp -p <local_filename> <username>@<server>:<remote_filename>

   In this command syntax, note the following:

   • -p preserves the modification times, access times, and modes from the original file.
   • <local_filename> is the name of the file you want to transfer.
   • <username> is the name of a valid user account on the server to which you want to transfer the file.
   • <server> is the IP address of the server to which you want to transfer the file.
   • <remote_filename> is the full path and file name of the file on the system to which you are transferring the file.

   For example:

   scp -p myfile.bin root@10.90.101.50:/var/tmp/myfile.bin

3. Follow the prompts to accept the key of the remote system and supply a password.

   You should receive output that appears similar to the following example:

   The authenticity of host '10.90.101.50 (10.90.101.50)' can't be established.
   DSA key fingerprint is b5:0c:53:97:d2:72:78:ab:37:be:e4:0a:56:58:14:7d.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '10.90.101.50' (DSA) to the list of known hosts.
   root@10.90.101.50's password:
   myfile.bin 100% 99 0.0KB/s 00:00

   This output indicates that the file transferred successfully. 

In case it does not work and the connection refused message is reported that points to network issues.
Some firewalls for example could block the connection.
You will need to resolve that connectivity issue and once resolved NCM should work fine as well.

Please Note: You can start the Windows Services console, services.msc, and set the SolarWinds SFTP/SCP Server service to Automatic, and start it. It will keep running even if the systray app dies or any issues with SCP GUI issues / Error there is no need to access the SCP GUI. You do not need a SCP GUI application to access. All you need to make sure the SCP Service is UP and running. 

Please Note: 
NCM doesn't support connecting through "source" device - If there is any Firewall or other device which is working in between through NAT NCM will not be able to communicate and fail therefore make sure the device has direct access to the NCM without any source device in-between only then the download will work. 

Please Note: If a single F5 device will fail it will affect the whole NCM Job and will not be complete.
NCM Jobs will show TimeOut for other devices. Please remove the device which doesn't have access to the SCP in order to let the NCM job run smoothly 

I have received an Error while SSH to SCP from F5 devices why?

failed to SSH SCP Server : ssh 192.168.xx.xx  no matching cipher found: client aes128-cbc,aes256-cbc server aes128-ctr,aes192

The error means that SSH ciphers supported by the device are not any of the ciphers supported by the SCP server.

This means device support only: aes128-cbc, aes256-cbc

SCP server support these ones: aes128-ctr,aes192-ctr,aes256-ctr,chacha20-
 
Cause: 
All CDC ciphers were disabled on purpose due to vulnerability issues

Suggest trying to configure the device if possible to support one of the SSH ciphers listed above by the server. In case this is not possible: Please consult the vendor for an upgrade firmware or provide support only for supported ciphers.

Another solution to the cipher issue when authenticating to the SCP Server is to specify the cipher in the SCP command using the -c option (-c cipher_spec).
An example would be SCP -c aes192-ctr.

Here is a real example of the device template's command lines in full:

<Command Name="DownloadConfigIndirectSCP" Value="${ConfigType}"/>

<Command Name="Running" Value="tmsh save /sys config file /var/local/scf/NCM.ucs no-passphrase${CRLF} ${TransferProtocol} -c aes192-ctr /var/local/NCM.ucs ${SCPServerUserName}@${SCPStorageAddress}:${StorageFilename}${CRLF}yes${SendConditionRegEx:Are you sure}${CRLF}${SCPServerPassword}"/>

<Command Name="Startup" Value="tmsh save /sys ucs /var/local/NCM.ucs${CRLF}${TransferProtocol} -c aes192-ctr /var/local/NCM.ucs ${SCPServerUserName}@${SCPStorageAddress}:${StorageFilename}${CRLF}yes${SendConditionRegEx:Are you sure}${CRLF}${SCPServerPassword}" IsBinary="true"/>

Another issue can be the fingerprint question prompt, for this, review this article.
 

[72cc9000] [12/31/2018 06:02:42.500] -->Ffdevice@(ABC-F5-device-Example)(cfg-sync Changes Pending)(Standby)(/Common)(tmos)#
[72cc9000] [12/31/2018 06:02:42.500] -->
[72cc9000] [12/31/2018 06:02:42.501] ProcessLogin State: Banner
[72cc9000] [12/31/2018 06:02:42.501] Custom Prompt detector detected a '>/#' prompt
[72cc9000] [12/31/2018 06:02:44.498] TimerTick: mstrData=<> State=Connected
[72cc9000] [12/31/2018 06:02:44.499] Pending Disconnect = False
[72cc9000] [12/31/2018 06:02:44.499] TimerTick: send CRLF to get the prompt
[72cc9000] [12/31/2018 06:02:44.500] <--