• Data storage
  • IOPS capacity
  • Shared vs. dedicated
• Host resources
  • Overall ESXi host resources for all services
  • CPU & MEM reservations for the IX appliance VM
  • pNIC/VMNIC capacity and shared load
  • Dedicated vmk interfaces for different services like mgmt/vMotion/vSR.
• Network Infrastructure throughout the entire data path
  • Data Center local network
  • Service Provider network infrastructure between source/target sites
  • Bandwidth availability
  • Latency and path reliability (packet loss)
    • vSphere replication (vSR) performance drops exponentially with higher packet loss and/or higher latency.
    • There is a built-in tolerance for high latency in vSphere replication but throughput will be reduced significantly.

• Workload VM conditions
  • Number of disks
  • Total and size per disk
  • Active services/applications
  • Data churning/disk changes

• 300 concurrent migrations per Manager
• 200 concurrent migrations per Service Mesh/IX Appliance.
• 1Gbps max per migration workflow
• 1.6Gbps max per IX appliance (any number of concurrent migration workflows)

• Do NOT exceed recommended allocations as that may cause the HCX Connector/Cloud Manager to malfunction.
• Both HCX Cloud Manager and Connector must be running version HCX 4.7.0 or later.
• There should be NO active migration or configuration workflows when making these resource changes.
• Changes must be made during a scheduled Maintenance Window.
• There is NO impact to Network Extension services.
• There is NO change of concurrency for HCX vMotion/Cold Migration workflow.
• The concurrent migration limit specified for HCX Replicated Assisted vMotion (RAV) is ONLY for Initial & Delta sync. During RAV switchover stage, only one relocation will be serviced at a time on a serial basis.
• Additional service meshes/IX appliance should be deployed for unique workload clusters to aggregate the replication capacity of multiple IX appliances. A different Services Mesh can be deployed for each workload cluster at source and/or target.
• If there are multiple service meshes/IX Appliances then RAV can switchover in parallel, however per SM/IX Pair it will always be sequential.

• Login to vCenter that hosts the HCX Manager.
• Shutdown HCX Manager VM's GuestOS using vCenter UI.
• Edit HCX Manager's VM to increase the vCPU and MEM reservations. Refer to:
  • Virtual CPU Configuration
  • Virtual Memory Configuration
• Power ON the HCX Manager VM.

• Mount the created disk to HCX managers.

mount  /dev/sdc1 /common_ext   
df -hT 
# Check if /common_ext has been mounted and has the correct type

• Add an entry to "/etc/fstab" to ensure mounted disk will sustain a reboot and HCX Manager upgrade.

vi /etc/fstab   
/dev/sdc1 /common_ext ext3 rw,nosuid,nodev,exec,auto,nouser,async 1 2

# systemctl stop postgresdb
# systemctl stop zookeeper 
# systemctl stop kafka 
# systemctl stop app-engine 
# systemctl stop web-engine 
# systemctl stop appliance-management

• Move directory "/common/kafka-db" to "/common/kafka-db.bak".

cd  /common
mv kafka-db kafka-db.bak

• Create a new directory "/common_ext/kafka-db".

cd  /common_ext
mkdir kafka-db

• Change the ownership and permissions of this directory same as "/common/kafka-db.bak".

chmod 755 kafka-db
chown kafka:kafka kafka-db

• Make a soft link from "/common/kafka-db" to "/common_ext/kafka-db".

cd  /common
ln -s /common_ext/kafka-db kafka-db

• Move directory "/common/postgres-db" to "/common/postgres-db.bak" as a backup

cd  /common
mv postgres-db postgres-db.bak

• Copy the content for directory "/common/postgres-db.bak" to "/common_ext/postgres-db" and change the ownership to postgres.

cp -r /common/postgres-db.bak /common_ext/postgres-db
chown -R postgres:postgres /common_ext/postgres-db

• Make a soft link from "/common/postgres-db" to "/common_ext/postgres-db".

cd  /common
ln -s /common_ext/postgres-db postgres-db

# systemctl start postgresdb
# systemctl start zookeeper
# systemctl start kafka
# systemctl start app-engine
# systemctl start web-engine
# systemctl start appliance-management

Login to HCX Connector/Cloud Manager Root Console

# systemctl stop postgresdb
# systemctl stop zookeeper 
# systemctl stop kafka 
# systemctl stop app-engine 
# systemctl stop web-engine 
# systemctl stop appliance-management

• Edit "app-engine-start" file to increase JAVA memory allocation and max perm size.

vi /etc/systemd/app-engine-start  
JAVA_OPTS="-Xmx4096m -Xms4096m -XX:MaxPermSize=1024m ...

• Edit "MobilityMigrationService.zql" and "MobilityTransferService.zql" to increase thread numbers.

vi /opt/vmware/deploy/zookeeper/MobilityMigrationService.zql 
"numberOfThreads": "50",   

vi /opt/vmware/deploy/zookeeper/MobilityTransferService.zql  
"numberOfThreads":50,

• Edit "vchsApplication.zql" and update "kafkaMaxMessageSizeBytes" from "2097152" to "4194304".

vi /opt/vmware/deploy/zookeeper/vchsApplication.zql 
"kafkaMaxMessageSizeBytes":4194304

• Edit "kafka server.properties" and update "message.max.bytes" from "2097152" to "4194304".

vi /etc/kafka/server.properties  
message.max.bytes=4194304

# systemctl start postgresdb
# systemctl start zookeeper 
# systemctl start kafka 
# systemctl start app-engine 
# systemctl start web-engine 
# systemctl start appliance-management

admin@hcx [ ~ ]$ systemctl --type=service | grep "zoo\|kaf\|web\|app\|postgres"
  app-engine.service                   loaded active     running       App-Engine                                                        
  appliance-management.service         loaded active     running       Appliance Management                                              
  kafka.service                        loaded active     running       Kafka                                                             
  postgresdb.service                   loaded active     running       PostgresDB                                                                                              
  web-engine.service                   loaded active     running       WebEngine                                                         
  zookeeper.service                    loaded active     running       Zookeeper 

• As a best practice, use vSphere Monitoring and Performance to monitor HCX Connector & Cloud Manager CPU utilization and MEM usage.
• Do NOT exceed the recommended limits as that could cause system instability and failed migration workflows.
• In a scaled up environment, when migration operations are being processed, expect for the CPU utilization to increase significantly during a short periods of time and there may be a temporary delay in the UI response for migration progressing events.
• Limit the concurrency of MON operations on target cloud when making configuration changes while having active concurrent Bulk migrations into MON enabled segments during switchover.
• Follow the migration events and estimation on the HCX UI to determine any slowness that may be caused by the infrastructure or the network.
• Additionally, vSphere Replication status can be monitored from the source ESXi host. Refer VMware Knowledge base 87028
• If a source ESXi host is heavily occupied from memory, I/O rate perspective, then replication performance will be affected. As a result, Bulk/RAV workflow may takes more time to complete initial base sync provided there are no slowness in the underlying datapath.

• The Bulk/RAV migration workflow consists of multiple stages (i.e. initial/delta sync, off-line sync, disk consolidation, data checksum, VM instantiation, etc.) and most are not dependent of network infrastructure hence the time to complete a migration for any given VM, from start to finish, may vary depending on the conditions and it is not a simple calculation based on the size of the VM and the assumed network bandwidth.