FBBR – Troubleshooting – SCP
November 27, 2023
FBBR – Troubleshooting – SFTP Backup
November 29, 2023
FBBR – Troubleshooting – SCP
November 27, 2023
FBBR – Troubleshooting – SFTP Backup
November 29, 2023

FBBR – Troubleshooting – SFTP

Default Port: 22 tcp

SFTP (SSH File Transfer Protocol) is a secure FTP protocol that sends files over secure shell (SSH), providing a high level of protection for file transfers. SFTP provides two methods for authenticating connections:

How SFTP Works

The SSH protocol employs public key cryptography to authenticate hosts and users, with the generation of authentication keys, known as SSH keys, facilitated by the keygen program.

Public key authentication in SSH serves as a more secure alternative to the older .rhosts authentication. This enhancement in security results from the avoidance of storing passwords in files, thereby eliminating the risk of a compromised server pilfering user passwords.

Nevertheless, SSH keys, being authentication credentials akin to passwords, necessitate analogous management to user names and passwords. It is imperative to implement a proper termination process to remove keys when they are no longer required.

SFTP utilizes SSH keys for authentication, a process involving the generation of both SSH private and public keys. The SSH public key is shared with a trading partner, who loads it onto their server and associates it with the relevant account. During connection to the SFTP server, the client software transmits the public key for authentication. If the public key aligns with the private key, along with any supplied user or password, the authentication succeeds.

User ID authentication can be employed with various combinations of key and/or password authentication.

SSH keys are generated in pairs and stored in plain-text files. This key pair comprises a private key (typically named id_rsa), stored securely on the local computer with restricted permissions, and a public key (usually named id_rsa.pub) placed on the intended server for login. The public key can be shared freely, enabling access to servers where it is added. Encryption using the public key requires the private key for decryption, and vice versa, facilitating secure access between servers by sharing public keys.

How it works?

The Key has to be generated

ssh-keygen
root@vc70b [ ~ ]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:P48WRpfevLw3IXxaCDSBo6oaQvlH2LGwrym/Cb8PaRM root@vc70b.vmware.local
The key's randomart image is:
+---[RSA 2048]----+
|          …    |
|         o o     |
|   . .  . o ..   |
|  . = o.  ..o    |
| o E +. S. ooo.  |
|. . =.   .o .+o+ |
|.o *.o   .o. .=..|
|..=.B     .+ .o..|
| .+O+.   .. . .o.|
+----[SHA256]-----+
root@vc70b [ ~ ]# cat /root/.ssh/id_rsa -----BEGIN OPENSSH PRIVATE KEY----- b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcn NhAAAAAwEAAQAAAQEArqnXZukswC8xPlk7PIZ/BtDVyYE4YL4xtM86mT+4CAAvDJ9vj/rb … ld+ceNqIjq0fJMKILsYEQ6ixk+EeA4Iyx2EV+ojmxnvLuXwWk8Xfo/zatk4VZk2mq+rFsY qzmlUXdORV+HkHqLAAAAF3Jvb3RAdmM3MGIudm13YXJlLmxvY2FsAQID 
-----END OPENSSH PRIVATE KEY-----
root@vc70b [ ~ ]# cat /root/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuqddm6SzALzE+WTs8hn8G0NXJgThgvjG0zzqZP7gIAC8Mn2+P+ttpz508UyKyT1FjZ1v0//YoVwkxcOjpTRuC6mkUv4eAJKx4FCDw2PpprN9sofDkPZUjvUAzzLIhHWnGPUePjwEx9QGxfDY3CZ9xI/AfK6BAMqTkJNyua/SzkOIHpEnOkOMrXBMS3fUcE1sl83ecVcMyrT1ZPNEpgM+nOAL1hp22QNJ4FY9o06/A/WwNhy40QCoThDZywj+xHl5diWAkbDZhl14b8cjDXxynhE9TDv7E/kHs34spYttIYa9UceLeJbUp/h6kx0qTN+lV0vbepPjmNBQongbtFvaD root@vc70b.vmware.local

applmgmt.log

2021-06-15T12:56:06.380 [24804]DEBUG:plugins.SftpStorageIOLib:generating key with command: ssh-keygen -q -t rsa -f /root/.ssh/id_rsa -N
2021-06-15T12:56:08.265 [24804]INFO:plugins.SftpStorageIOLib:Ssh keys are successfully generated and persisted in /root/.ssh/id_rsa

Copy the public key to the Server

ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts @
root@vc70b [ ~ ]# ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts navin@192.168.7.9
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
FIPS mode initialized
navin@192.168.7.9's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -p '22' -o 'UserKnownHostsFile=/root/.ssh/br_temp_known_hosts' 'navin@192.168.7.9'" and check to make sure that only the key(s) you wanted were added.

applmgmt.log

2021-06-15T12:56:08.267 [24804]INFO:plugins.SftpStorageIOLib:Authorizing localhost with command sshpass -f /tmp/tmpo616q8go ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts user1@192.168.7.9
2021-06-15T12:56:08.476 [24804]ERROR:plugins.SftpStorageIOLib:Failed to authorize local host with command: sshpass -f /tmp/tmpo616q8go ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts user1@192.168.7.9. Will proceed with backup through curl

If the user does not have home directory and is just and sftp account, you would get the below message.

# ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts user1@192.168.7.9

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.7.9 (192.168.7.9)' can't be established.
ECDSA key fingerprint is SHA256:5HERHfc9Vkhsp2SUY3NyZ2DdPFrDWOvNpEc2fpGY9u0.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
FIPS mode initialized
user1@192.168.7.9's password:
This service allows sftp connections only.

On the server you would see the copied public key

navin@ubuntu:~/.ssh$ cat ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCl0nbM2ww13n7Ila9sV7nKzQiVzN+gxYBaTgLP5ZO0NHR3khjFaspI53asW5N48Oapvdyl7ULU7R4TOhfBEa1pnHeKQumtxop3nDPNpBRW2bl1mC9sTKVHwcwu4/Wf4thqcQ95MDVYtMETdoA7lRBN8tfSE7Tb6Y+26KSgj2rPpieiGB8Yw9XyMcycosjQ+YG6zsBpPLtilcJX1YMH2B2xKJBG6zMFjdsxVh4cB9SKhFDPBWmvz9sDR+DV4jTYPd7mufmDZu/eYfXXUci0ac0BT7nyBo7+PrlIpSXep0Cbgf5SU2HH4y5A3naNeFB8cY6JRJL5jfGuxEhpiSnDa6DX root@vc70b.vmware.local

Now SSH works with Authorization keys rather with Username and password.

# ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts <username>@<IP>
# ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts navin@192.168.7.9
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-65-generic x86_64)
navin@ubuntu:~$

applmgmt.log

2021-06-15T12:56:08.478 [24804]INFO:plugins.SftpStorageIOLib:Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts user1@192.168.7.9 echo ~

If the user is an sftp user only and does not have a home directory or if the key generation fails; more or less if the communication through authorized keys fail, vcsa will automatically failback to the curl.

applmgmt.log

2021-06-15T12:56:08.562 [24804]DEBUG:plugins.SftpStorageIOLib:Passwordless ssh tunnel is not working, falling back to CURL

Concurrent Sessions

Certain environments would have a limitation of concurrent SSH Sessions. This would affect the functionality of the SFTP Backup because the file transfers in FBBR is initiated in parallel in VCSA.

The SFTP Utility / WinSCP / SSH Connection to SFTP Server would work but Backup Operation fails.

backup.log

2021-06-28T19:53:09.290 [20210628-195220-17958471] [ConfigFilesBackup:PID-41030] [ConfigFiles::BackupConfigFiles:ConfigFiles.py:324] ERROR: Failed configuration files backup- rc: 2; stdout: ; stderr:b'curl: (2) Failure establishing ssh session\n'; exception: Checksum not generated at /dev/shm/backupRestoreSumFile-20210628-195220-17958471-qeick8iv
2021-06-28T19:53:09.291 [20210628-195220-17958471] [ConfigFilesBackup:PID-41030] [ConfigFiles::BackupConfigFiles:ConfigFiles.py:335] ERROR: Failed configuration files backup
Underlying process status. rc: 2
stdout:
stderr: b'curl: (2) Failure establishing ssh session\n'
Traceback (most recent call last):
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ConfigFiles.py", line 328, in BackupConfigFiles
status)
util.Common.BackupRestoreError: Failed configuration files backup
Underlying process status. rc: 2
stdout:
stderr: b'curl: (2) Failure establishing ssh session\n'
2021-06-28T19:53:10.49 [20210628-195220-17958471] [MainProcess:PID-40494] [Proc::VerifyProcStatusAndGetArchive:Proc.py:158] ERROR: Error at process ConfigFilesBackup; rc:2.
2021-06-28T19:53:10.49 [20210628-195220-17958471] [MainProcess:PID-40494] [Proc::VerifyProcStatusAndGetArchive:Proc.py:162] ERROR: stderr:Failed configuration files backup
b'curl: (2) Failure establishing ssh session\n'

A workaround would be to changing Backup from Parallel to Serial.

vi /etc/applmgmt/appliance/appliance.conf

change

{
"backupRestore": {
"logLevel": "INFO",
"restoreTimeout": 72000,
"fileTransferMode": "parallel",
"backupTimeout": 72000
},

to

{
"backupRestore": {
"logLevel": "INFO",
"restoreTimeout": 72000,
"fileTransferMode": "serial",
"backupTimeout": 72000
},

In the Series:

Related posts

January 8, 2024

Exporting and viewing LDIF from vCenter using SSH / JXplorer

Read more
December 9, 2023

SSH Key based authentication for vCenter using Putty

Read more
December 8, 2023

SSH Key based authentication for vCenter using Openssh

Read more

Leave a Reply

Your email address will not be published. Required fields are marked *