Cross vCenter vMotion fails with error "A general system error occurred, Connection reset by peer"

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

In the vCenter UI, you will see the below error during compatibility checks:

"A general system error occurred, Connection reset by peer. The connection is terminated by the remote end with a reset packet. Usually this is a sign of a network problem, timeout or service overload "

In the /var/log/vmware/vpxd/vpxd.log on source vCenter, you will find entries similar to:

2023-02-14T12:16:38.134Z info vpxd[18631] [Originator@6876 sub=Default opID=le2uwc2r-1007594-auto-llgr-h5:70062956-38] [VpxLRO] -- ERROR session[5253d1d0-d88
c-b864-914e-5631c26c4aff]52cdce65-d283-d05a-50ca-c32e496b2be9 -- ProvChecker -- vim.vm.check.ProvisioningChecker.checkRelocate: vmodl.fault.SystemError:
--> Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem, timeout, or service overload."
-->    msg = ""
--> }
--> Args:

In the /var/log/vmware/vpxd/vpxd.log on destination vCenter, you will find entries similar to:

2023-02-14T12:15:57.476Z error vpxd[58923] [Originator@6876 sub=Req@internalvpxd3...] XML document element count exceeds configured maximum 500000
-->
--> while parsing property "data" of static type ArrayOfByte
-->
--> while parsing serialized DataObject of type vim.host.ScsiLun.DurableName
--> at line 573, column 11005630
-->
--> while parsing property "alternateName" of static type ArrayOfScsiLunDurableName
-->
--> while parsing serialized DataObject of type vim.host.ScsiDisk
--> at line 573, column 11001977
-->
--> while parsing property "scsiLun" of static type ArrayOfScsiLun
-->
--> while parsing serialized DataObject of type vim.host.StorageDeviceInfo
--> at line 573, column 52193
-->
--> while parsing property "storageDevice" of static type HostStorageDeviceInfo
-->
--> while parsing serialized DataObject of type vim.host.ConfigInfo
--> at line 573, column 51522
-->
--> while parsing property "hostConfig" of static type HostConfigInfo
-->
--> while parsing serialized DataObject of type vpx.vmcheck.HostState
--> at line 573, column 44373
-->
--> while parsing property "host" of static type VpxVmcheckHostState
-->
--> while parsing serialized DataObject of type vpx.vmcheck.CheckProvision.CheckState
--> at line 10, column 675
-->
--> while parsing property "prevOutput" of static type ArrayOfVpxVmprovActionOutput
-->
--> while parsing serialized DataObject of type vpx.vmcheck.CheckRelocateContext
--> at line 10, column 286
-->
--> while parsing property "context" of static type VpxVmprovContext
-->
--> while parsing serialized DataObject of type vpx.vmprov.Action.Input
--> at line 10, column 279
-->
--> while parsing call information for method WorkflowExecute_Task
--> at line 10, column 0
-->
--> while parsing SOAP body
--> at line 9, column 0
-->
--> while parsing SOAP envelope
--> at line 2, column 0
-->
--> while parsing HTTP request for method execute
--> on object of type vpx.vmprov.Workflow
--> at line 1, column 0
--> req: <[N7Vmacore4Http22Http2ServerTransactionE:0x00007fdf4002ce10], "urn:internalvpxd3/7.0.3.0">

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 6.7.x

Cause

Destination vCenter server is not able to read the workflow context completely causing the connection to break.
By default vCenter supports 500000 elements in the workflow XML document. This issue occurs when the workflow XML document exceeds 500000 elements.

Resolution

To resolve this issue, follow the below procedure on the destination vCenter server.

Add the below vCenter advanced settings to increase the maxDocSize to 50MB and maxElements to 1 million elements

config.vmomi.soapRequest.maxDocSize = 50000000
config.vmomi.soapRequest.maxElements = 1000000

Example screenshot:

Note: If the issue still persists, the value of config.vmomi.soapRequest.maxElements can be further increased to 1500000 or 2000000.

For information on how to configure vCenter Advanced settings, refer VMware Documentation .