Upgrading vCenter Server 7.0 fails during precheck with "Host(s) were found in the vCenter Inventory, that are potentially problematic for a vCenter upgrade"
search cancel

Upgrading vCenter Server 7.0 fails during precheck with "Host(s) were found in the vCenter Inventory, that are potentially problematic for a vCenter upgrade"

book

Article ID: 318867

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Upgrading to vCenter Server 7.0U3c+ now requires an additional precheck. This precheck ensures that the ESXi hosts managed by the vCenter Server are at the correct ESXi versions and will not have any issues related to driver conflicts. There are two situations that can cause the vCenter precheck to fail:
ESXi hosts running 7.0 U2c/U2d outside of a vLCM image-enabled cluster are not impacted

If the precheck detects versions of ESXi that could potentially have dual i40en driver conflicts, a more detailed check of these hosts is automatically triggered to detect if the dual-driver condition truly exists on the hosts flagged by the VC precheck.

Note that the precheck will fail and an error similar to the below will be reported in the VMware vCenter VAMI.
(Please see KB 87319 for details on the errors reported by the CLI upgrade option)

Fail.png
This error will be reported based on the presence of ESXi hosts running the versions detailed above and the results from the more detailed check will need to be examined to determine if any further action is required. It is possible that the more granular scan will not return any problematic host(s) and the upgrade can be restarted without any remediation efforts.

 

Reading the Results

Using an SSH client such as Putty, log in to the vCenter Server CLI. SSH may need to be enabled first.

Open the dual_driver_check_<timestamp>.log file reported in the error message to determine when the script has finished running. It can take up to 5 seconds per ESXi host to complete. The image below shows what to expect when the script has finished.

image.png

The contents of the "dual_driver_check_faulty_hosts_<timestamp>.txt", also contained in the "/var/log/vmware/applmgmt/" directory, will list any confirmed hosts that are impacted by the dual driver issues and will need manual remediation.

The image below is an example of the output generated in the "dual_driver_check_faulty_hosts_<timestamp>.txt". In this case, there are 3 hosts confirmed as having both the i40en and i40enu driver installed. These hosts need remediation using one of the options listed in the Solution section below.

faulty_hosts output.png

However, as detailed above, it is possible that the results of the more detailed scan will not list any hosts. In this case, please restart the upgrade and the precheck will pass on the 2nd attempt.
 

Output file descriptions

  • dual_driver_check_faulty_hosts_<timestamp>.txt: Contains the hosts that were found to have the i40en driver conflict. These need to be addressed before restarting the upgrade.
  • dual_driver_check_skipped_hosts_<timestamp>.txt: Contains any hosts that were skipped because they were disconnected or not responding at the time. These are ignored.
  • dual_driver_check_errored_hosts_<timestamp>.txt: Contains any hosts that the script was unable to process for any other reason than being disconnected. Use the dual_driver_check_<timestamp>.log to further investigate why these hosts were unable to be scanned. These need to be addressed before restarting the upgrade.
  • dual_driver_check_<timestamp>.log: Primary logging for the script process.



Environment

VMware vCenter Server 7.0.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 7.0.3
VMware vCenter Server 7.0.3

Resolution

 Option 1 (Recommended) 

Upgrade the hosts listed in dual_driver_check_faulty_hosts_<timestamp>.txt to 7.0 U3c+ with an ISO upgrade baseline (refer to Related information section of the Knowledge Base article - Post vSphere 7.0 Update 3c Notes for ESXi releases with no ISO) or vLCM image before upgrading the vCenter Server. Although this is not the normally recommended order, it's necessary for this circumstance and fully supported by VMware. The hosts must be upgraded using an ISO image instead of a rollup patch baseline in vSphere Lifecycle Manager.Also note that ISO import depend on respective VC version.

  Upgrading ESXi using an ISO Upgrade Baseline
  1. Download the ISO image via the VMware Downloads Portal in Customer Connect.
  2. Upload the ISO into the Lifecycle Manager Depot.
  3. Create a baseline that uses the imported ISO image.
  4. Upgrade the host using the ISO baseline.
  5. Repeat this for every host listed in the /var/log/vmware/applmgmt/dual_driver_check_faulty_hosts_<timestamp>.txt file generated by the script.
  6. Once all affected hosts are corrected, restart the upgrade process.
   Upgrading ESXi as part of a vLCM image-enabled cluster
  1. Please review the documentation for Using vSphere Lifecycle Manager Images to find instructions on upgrading the hosts via a vLCM desired-state image.
  2. Repeat this for every host listed in the /var/log/vmware/applmgmt/dual_driver_check_faulty_hosts_<timestamp>.txt file generated by the script.
  3. Once all affected hosts are corrected, restart the upgrade process.
 

Option 2

Alternatively, you can manually correct the i40en driver conflict on each host before restarting the upgrade. This isn’t the recommended option because the precheck will fail one more time before being able to upgrade. To understand why, see the precheck flowchart in Related Information.
  1. Put the ESXi host in maintenance mode, then run the following command from an SSH session to remove the obsolete i40enu VIB :
esxcli software vib remove --vibname=i40enu
  1. Once the VIB has been removed, reboot the ESXi host.
  2. Repeat this for every host listed in the /var/log/vmware/applmgmt/dual_driver_check_faulty_hosts_<timestamp>.txt file generated by the script.
  3. Once all affected hosts are corrected, restart the upgrade process
 
 

Important Upgrade Notes

  1. Once the vCenter precheck is clean, it is recommended to proceed to upgrade immediately. Prior to the upgrade, do not introduce additional hosts with potential driver conflicts or transition any clusters to be vLCM image-managed.

  2. Once vCenter has been upgraded, it's important NOT to use the non-critical patch baseline to upgrade any hosts that are 7.0 GA - 7.0 U3a, as it has the potential of re-introducing the issue or fail during remediation with an esxupdate error. Ensure that an image-based upgrade baseline or the predefined critical host patch baseline is used to upgrade them. A banner has been added on the Updates tab as a reminder of this.

Additional Information

Precheck FlowChart
image.png

Background
This issue originated when the Intel driver was renamed to i40enu in ESXi 7.0u2 and back to i40en in 7.0u3. During those renames, the relationship between the drivers was not created correctly, which led to both drivers being present on the same system. These conflicting drivers would cause solutions that installed agent vibs to fail, such as HA, WCP, and NSX on the hosts. The fix for this problem is included in ESXi 7.0 U3c. However, because these agent vibs can be pushed to the ESXi hosts as soon as vCenter gets upgraded, this issue would still manifest during the normal upgrade order (vCenter -> ESXi), as the ESXi hosts may still be running affected versions.
 
The vCenter precheck was created to notify users whether potentially affected ESXi versions existed in the vCenter inventory pre-upgrade. The automated script was added to help users narrow down the scope of truly affected hosts, so they can remediate the issue by either upgrading them or manually removing the duplicate i40enu driver.


For other information related to this issue see:

 
Post vSphere 7.0 Update 3c Notes –

The scenario mentioned in the Details section of this KB, is applicable to future vSphere 7.0 patch releases post 7.0 Update 3c.

If the source system is of version  between ESXi 7.0 Update 2 and Update 3c, before upgrading to future vSphere patch release, ensure VMware vCenter server upgraded to 7.0 Update 3c or later to avoid potential vSphere HA failure due to dual i40en driver conflicts condition.

The above solution section is applicable for all 7.x future patch releases regardless of patch being ESXi-only or VC-only or both, but with additional instructions to Solution 1 as applicable:

In case of *no* ISO deliverable in the target ESXi patch post 7.0 Update 3c, then follow either of the options below:

  1. Create custom/stock ESXi ISO using the target ESXi patch ZIP bundle referring to the steps mentioned in the Create a Custom ESXi ISO Image with PowerCLI Cmdlets documentation.
  2. In vSphere Lifecycle Manager, use Baseline Group upgrade method to create and combine the baselines as described below:
    1. Create baseline say “baseline-A” of ESXi ISO (custom/stock) from the last release (example: ESXi 7.0 Update 3d) 
    2. Create baseline say “baseline-B” of target ESXi patch using rollup 
    3. Create baseline Group say “ baselineGroup-C” to combine baselines – “baseline-A” and “baseline-B” 
    4. Remediate the baseline group “ baselineGroup-C” against the intended ESXi hosts or Cluster.
For the steps to use Baseline Group upgrade method refer to Creating and Working with Baselines and Baseline Groups documentation.


Powered by Wolken Software