This article provides information about deploying a vSphere Metro Storage Cluster (vMSC) across two data centers or sites using HPE Nimble Storage.

What is vMSC?

vSphere Metro Storage Cluster (vMSC) is a tested and supported configuration for stretched storage cluster architectures. A vMSC configuration is designed to maintain data availability beyond a single physical or logical site. All supported storage devices are listed on either the VMware Storage Compatibility Guide or the Partner Verified and Supported Products (PVSP) listings.

What is HPE Nimble Storage Peer Persistence?

Built from a resilient architecture, HPE Peer Persistence is available on HPE Nimble Storage. Paired arrays located at metropolitan distances, present a continuous storage system to hosts connected to them.
This capability allows you to configure a high-availability solution between two sites where storage failover and failback remains completely transparent to the hosts and applications running on those hosts.
This results in elimination of recovery times because unlike traditional failover models, your applications don’t have to be restarted after a failover. 

HPE Nimble Storage Quorum Witness

The HPE Nimble Storage Quorum Witness is a component provisioned as a virtual machine that is recommended to be deployed at a third site. The HPE Nimble Storage Quorum Witness, along with the two  Nimble Storage systems, forms a three part quorum system. This quorum system monitors  the status of both the HPE Nimble Storage systems and the storage site inter-links. A number of site and inter-link failure scenarios are recognized by this three part quorum system, and appropriate failover actions are implemented. In the event of a disaster that brings down one of the storage  sites, a failover to the surviving HPE Nimble Storage site is automatically initiated. During this failover operation, replicated volumes on the remaining storage system are made active. The host paths to those volumes are also made active, thereby ensuring that hosts can continue to access their volumes without any disruption or outage. Communication between the three sites for quorum is via the Quorum Witness IP and the service management IP’s of the two HPE Nimble Storage systems. HPE Nimble Storage Quorum Witness does not actively participate in data storage and a failure or removal of the HPE Nimble Storage Quorum Witness from an otherwise functioning environment will have no impact on data storage. However it will impact the automatic switch over from one site to another incase of a disaster. Thus the HPE Nimble Storage Quorum Witness only comes into play when one site or the Inter-Site Link (ISL) has failed.

Configuration Requirements

These requirements must be satisfied to support a vMSC configuration with HPE Nimble Storage.

• VMware ESXi 6.0 or higher, Metro Storage Cluster configured for uniform host access per VMware requirements and best practices. 
• In the first release, HPE-Nimble Peer Persistence can only be configured between arrays of the same model. Meaning an AF40 can replicate to another AF40 but not to a HF40, as one is All Flash, the other is Hybrid. Capacity and interface cards may differ on both systems, but the type and model of the arrays must to be the same. This is to ensure the best ISL performance.
• Peer Persistence is supported on the current 'Gen5' and previous generation of HPE Nimble Storage hardware, including: 

• In order to provide maximum protection, redundant links for Replication and Client Access are required. A minimum of 2 x 1Gb Ethernet links are required per controller for Replication traffic.
• Both FC and iSCSI attached hosts are supported with Peer Persistence, however the ISL is available over IP. 
• Nimble Peer Persistence supports both upstream and downstream volume replication. The volumes are grouped into Volume Collections (VolCol). A VolCol may either be upstream or downstream. Additionally an individual VolCol will only serve data on the host side either on FC or IP, not  both i.e. mixed protocols in a VolCol are not supported.
• Latency should remain <5ms round-trip response time between arrays in the group on all synchronous replication links. The replication traffic will be classified as “Group traffic” within the scale-out group.
• If using Automatic Switch Over (ASO) for application transparent failover, it is highly recommended that the Witness client be located at a third site for failure domain redundancy. 
• NimbleOS 5.1 comes with updated host-side Nimble Toolkits, and these should be installed and correctly working for VMware, Windows and/or Linux hosts.

Solution Overview

A test qualified VMware Metro Storage Cluster using HPE Nimble Storage is supported in accordance with the VMware description of a uniform access vMSC configuration. Particular to the uniform access configuration, host data path connections at each site are cross-connected to the peer Data Center site storage array. ESXi hosts access volumes in the Data Center local array via active paths. Connectivity to standby peer volumes on the distance array is maintained in standby mode until such time as a failover or switchover. In concert with HPE Nimble Storage, HPE Nimble Quorum Witness and with Automated Fail Over (AFO) enabled, a minimally disruptive switchover or failover of volume access across sites can be achieved.
A vital component is the HPE Nimble Storage witness server which arbitrates in a split-brain scenario.

AFO is triggered in following scenarios

1. Array is not reachable to its peer array and witness for certain amount of time i.e. the array is out of quorum 
   1. if Group Leader not reachable(no heartbeats to quorum), Backup Group Leader will become Group Leader and sync replication handover happens so that new group leader can start serving the IO. 
   2. if Backup Group Leader not reachable(not heart beating to quorum),  Group Leader (GL) will handover sync rep volumes served from Backup Group Leader (BGL). So that GL can start serve the IO for those volumes.
2. When Backup Group Leader (BGL) cannot reach Group Leader (even if BGL can reach witness), Then Group leader will perform AFO and start serving IO from group leader. This is due to NimOS dependency on group leader to be reachable to server IO

This diagram depicts a high level overview of the solution.
 

Failure Scenarios

It is important to understand how the system will react under various failure conditions. In some situations, the hosts may also be experiencing failures. HPE Nimble Storage Peer Persistence is designed to protect the integrity of the data while allowing ASO. Under certain failure conditions, ASO may not occur.
 

Scenario	HPE Nimble Storage System Behaviour
Witness Failure or disconnect	All volumes will remain online and serving data on their current array. Once the Witness is repaired, ASO capability will recover.
Volumes out of sync	When a downstream synchronously-replicated volume is unavailable, then the upstream volume collection goes out of sync, but host I/O is accepted. Note that I/O resynchronization is continuously attempted by the upstream volume in this state. When the volume recovers resynchronization brings the upstream and downstream volume collections into sync.
Local Controller failure	During a local controller failover operation, no ASO will occur. Controller operations will resume locally on the partner controller. Volume collections may go out of sync briefly and automatic resynchronization may be required
Array failure	If an entire array fails, an ASO will be triggered. Group functions and access to data will be provided by the surviving array. Resynchronization will occur after the array is recovered and brought back online. A manual handover will be required after recovery and resynchronization. Volumes not protected by Peer Persistence will be unavailable.
Replication Link failure	Volume collections will resynchronize after replication links are recovered.

https://infosight.hpe.com/InfoSight/media/cms/active/public/HPE_Nimble_Storage_Peer_Persistence_Deployment_Considerations.pdf