Creating an NSX-T Workload Domain fails when configuring the backup schedule in VMware Cloud Foundation.

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

Messages similar to the following are seen:

Failed to configure backup schedule for NSX-T Manager - nsxt-vip.domain.corp.local

Message: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) => { messages = [], data = struct => {error_message=Field level validation errors: {passphrase is too week. It must be at least 8 characters long and contain at least one each: lowercase, uppercase, numeric character and special character.}, httpStatus=BAD_REQUEST, details=Field level validation errors: {passphrase is too week. It must be at least 8 characters long and contain at least one each: lowercase, uppercase, numeric character and special character.}, error_code=255, module_name=common-services} }

Environment

VMware Cloud Foundation 3.7.x

Cause

NSX-T password policy requires that that the backup credentials meet the following requirements:

Must be at least 8 characters in length.
Contain at least one each: lowercase, uppercase, numeric character and special character.

Resolution

To ensure that all backup tasks for NSX-T are taken successfully, validate that all passwords meet the password policy requirements mentioned above.

Workaround:
Before proceeding, please ensure that all NSX backup schedules are turned off on all existing Workload Domains, including the Management domain.

For Cloud Foundation 3.7.x
This workaround will be changing the password for the FTP backup user so that it will meet the requirements of NSX-T password policy.

Open an SSH to SDDC Manager VM and change to the root user.

su -

Run the following command to update the backup user password:

passwd backup

Note: Make sure to meet password policy requirement in choosing new password.

Run the following command on the SDDC Manager:

curl http://localhost/css/credentials/ | json_pp

Find and note down the "id" where "credentialType" equals FTP.
Using the below command, update the FTP password:

curl -X PUT -H “Content-Type:application/json” http://localhost/css/credentials/{credential-id} -d “{new-password}”

In the SDDC Manager UI, navigate to the failed task. Note the Workflow ID.
Get the Workflow Spec information by running the below command using the Workflow ID from Step 6.

curl -s http://localhost/domainmanager/internal/vault/{workflow-id} | json_pp > /tmp/workflow.json

Open the /tmp/workflow.json file:

vi /tmp/workflow.json

Update 'passphrase' and 'password' value to the new password.

"backupScheduleConfiguration" : [
"com.vmware.evo.sddc.common.plugin.model.contract.nsxt.NsxtBackupConfiguration",

{

"backupFileServer" : [ "com.vmware.evo.sddc.common.plugin.model.contract.nsxt.NsxtBackupFileServer",

{
"port" : 22,
"username" : "backup",
"directoryPath" : "/nfs/vmware/vcf/nfs-mount/backup",
"server" : "172.18.63.5",
"sshFingerprint" : null,
"password" : "VMware123!" <------------------ change here
}
],
"backupEnabled" : true,
"secondsBetweenBackups" : 3600,
"passphrase" : "VMware123!" <------------------ change here
}
],

Save and exit the file by pressing ESC and then typing :wq.
Update Workflow Spec by passing the edited json file with the new password using the below command:

curl -H 'Content-Type:text/plain' -X PUT http://localhost/domainmanager/internal/vault/{workflow-id} -d @/tmp/workflow.json

Navigate to the failed add VI workflow and restart the task.

For Cloud Foundation 3.8.1

Open an SSH to SDDC Manager VM and change to the root user.

su -

Run the following command to rotate the password:

curl 'https://sddc-manager.vcf.corp.local' -i -u -k 'admin:admin_Password' -X PATCH -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'privileged-password:secure_user_password' -H 'privileged-username: [email protected]' -d '{
  "operationType" : "ROTATE",
  "elements" : [ {
    "resourceName" : "sddc-manager.vcf.corp.local",
    "resourceType" : "BACKUP",
    "credentials" : [ {
      "credentialType" : "FTP",
      "username" : "backup"
    } ]
  } ]
}

Run the lookup_password command to return the new password:

lookup_passwords -u [email protected] -p {vcf-secure-user_password} -e BACKUP

In the SDDC Manager UI, navigate to the failed task. Note the Workflow ID.
Get the Workflow Spec information by running the below command using the Workflow ID from Step 6.

curl -s http://localhost/domainmanager/internal/vault/{workflow-id} | json_pp > /tmp/workflow.json

Open the /tmp/workflow.json file:

vi /tmp/workflow.json

Update 'passphrase' and 'password' value to the new password.

"backupScheduleConfiguration" : [
"com.vmware.evo.sddc.common.plugin.model.contract.nsxt.NsxtBackupConfiguration",

{

"backupFileServer" : [ "com.vmware.evo.sddc.common.plugin.model.contract.nsxt.NsxtBackupFileServer",

{
"port" : 22,
"username" : "backup",
"directoryPath" : "/nfs/vmware/vcf/nfs-mount/backup",
"server" : "172.18.63.5",
"sshFingerprint" : null,
"password" : "VMware123!" <------------------ change here
}
],
"backupEnabled" : true,
"secondsBetweenBackups" : 3600,
"passphrase" : "VMware123!" <------------------ change here
}
],

Save and exit the file by pressing ESC and then typing :wq.
Update Workflow Spec by passing the edited json file with the new password using the below command:

curl -H 'Content-Type:text/plain' -X PUT http://localhost/domainmanager/internal/vault/{workflow-id} -d @/tmp/workflow.json

Navigate to the failed add VI workflow and restart the task.

Important: Post completion of the workaround, reconfigure all NSX backups for existing workload domains , this includes the Management NSX manager backups through the NSX manager management console with changed password & ON the backup schedule.