VMware KB: VMware Storage Appliance - Understanding and Investigating Degraded States

VSA Manager in vCenter Server shows datastores in a Degraded state.
Datastore view in vCenter Server indicates datastores are in a Degraded state.

Sometimes it is a expected behavior for VSA datastores to be in a Degraded state.

For example, after one of the VSA appliances has been placed into maintenance mode or is otherwise unavailable, disk replication does not take place, so the state is termed degraded.

However, sometimes you see something like this from the getStorageCluster output of the wscli command (wscli <IP Address of Cluster Service or Cluster Leader> getStorageCluster):

Storage Cluster ID = 9fb33b43-1fb8-40d7-902b-74de51b5e748
Name = vStorage Cluster
Maintenance mode = false
Management interface = 88.147.90.11/27
Master = fc83f3b4-ebff-4190-84f7-54e3a1b2af98
Pseudo-SVA address = 88.147.90.12
Pseudo-SVA State = false
Physical capacity = 2287468544 KB (2181.50 GB)
Storage capacity = 1143734272 KB (1090.75 GB)
### Members:
Member 0:
        SVA ID = fc83f3b4-ebff-4190-84f7-54e3a1b2af98
        Name = localhost
        Maintenenace mode = false
        Domain name = localdom
        Member ID = c5f416c8-6dcb-40e0-b4c6-f80214d7cda6
        Storage Cluster ID = 9fb33b43-1fb8-40d7-902b-74de51b5e748
        State = ONLINE
        Management interface = 88.147.90.13/27
        Internal interface = 192.168.0.1/24
        No DNS servers
        Gateway = 88.147.90.1
        Total storage = 1143734272KB
        Free storage = 0KB
        Used storage = 1143734272KB
        Storage pool 0:
                ID = 00000000-0000-0000-0000-000000000000
                Total storage = 1143734272
                Free storage = 0
                Used storage = 1143734272
        Export storage entities:
                b77bf48d-73eb-49e0-a640-f0e696869c26

Member 1:
        SVA ID = f6fc9403-d620-4b6e-8cb3-eea600f0be1a
        Name = localhost
        Maintenenace mode = false
        Domain name = localdom
        Member ID = d90e8e80-b593-47cf-9a5d-7d65a4823974
        Storage Cluster ID = 9fb33b43-1fb8-40d7-902b-74de51b5e748
        State = ONLINE
        Management interface = 88.147.90.15/27
        Internal interface = 192.168.0.2/24
        No DNS servers
        Gateway = 88.147.90.1
        Total storage = 1143734272KB
        Free storage = 0KB
        Used storage = 1143734272KB
        Storage pool 0:
                ID = 00000000-0000-0000-0000-000000000000
                Total storage = 1143734272
                Free storage = 0
                Used storage = 1143734272
        Export storage entities:
                a7728e41-94bd-4a36-b730-df5f83ff3e19

<snip>

### Storage Entities:
Storage entity 0:
NFSv3StorageEntity:
        Id = b77bf48d-73eb-49e0-a640-f0e696869c26
        Name = NFSv3StorageEntityBlc-b77bf48d-73eb-49e0-a640-f0e696869c26
        State = DEGRADED

<snip>

                Volume:
                        Name = Volume-60395363-5e8f-41c1-a7c5-f40b4329003b
                        Id = 60395363-5e8f-41c1-a7c5-f40b4329003b
                        Size = 571867136
                        Primary Owner = fc83f3b4-ebff-4190-84f7-54e3a1b2af98
                        Current Owner = fc83f3b4-ebff-4190-84f7-54e3a1b2af98
                        Export Id = b7e378cb-fcd0-4f75-9081-b959c1b702f3
                        State = DEGRADED
                        Type = RAID1
                        # Disks = 2
                        Sync Percent = 100.00 %

<snip>

                                Disk[0]
                                        Name = Disk-c46447bb-ae72-491d-9743-893bd35039c6
                                        Id = c46447bb-ae72-491d-9743-893bd35039c6
                                        Owner Id = fc83f3b4-ebff-4190-84f7-54e3a1b2af98
                                        State = ONLINE
                                        Synced = true
                                Disk[1]
                                        Name = Disk-27ce4fec-a736-438c-bd68-97a0d7c4ae8f
                                        Id = 27ce4fec-a736-438c-bd68-97a0d7c4ae8f
                                        Owner Id = f6fc9403-d620-4b6e-8cb3-eea600f0be1a
                                        State = AVAILABLE
                                        Synced = false

<snip>

Storage entity 1:
NFSv3StorageEntity:
        Id = a7728e41-94bd-4a36-b730-df5f83ff3e19
        Name = NFSv3StorageEntityBlc-a7728e41-94bd-4a36-b730-df5f83ff3e19
        State = DEGRADED

<snip>

                        Sync Percent = 100.00 %
                        Sync Time-to-Complete = 0 sec.
                        Sync Speed = 0 KB/s
                                Disk[0]
                                        Name = Disk-516cc77c-6f81-4e7d-80ee-650b2b38640b
                                        Id = 516cc77c-6f81-4e7d-80ee-650b2b38640b
                                        Owner Id = f6fc9403-d620-4b6e-8cb3-eea600f0be1a
                                        State = ONLINE
                                        Synced = true
                                Disk[1]
                                        Name = Disk-b2bebee9-eee9-4091-bcde-e72b4fb9e0e5
                                        Id = b2bebee9-eee9-4091-bcde-e72b4fb9e0e5
                                        Owner Id = fc83f3b4-ebff-4190-84f7-54e3a1b2af98
                                        State = AVAILABLE
                                        Synced = false

If the state of Degraded is not because of scheduled activity such as an appliance being placed into maintenance mode, or a known cause such as a VSA Appliance being offline, the first step is to obtain the getStorageCluster output:

Log into one of the VSA appliances
Run this command:

wscli 127.0.0.1 getStorageCluster | more

If you get a log in exception, this means that a different appliance is the cluster leader, so log into the other appliance(s) until you get valid output.

If the output includes states for the storage entities like those illustrated above, the replication may be stopped because the back-end interfaces cannot talk to each other.

From inside the appliances, ping the other appliances' back-end IP addresses.

If the pings are not successful, this is prevents replication.

Possible causes of this include:

The switch used for the back-end failed or is mis-configured
A misconfiguration of back-end vSwitch, like vLAN settings
vLAN setup is incorrect on the physical switch
back-end connections are disconnected.

Accordingly, investigate the above for issues.

After the appliances can successfully ping each other on the back end IP addresses, the synchronization should start and the state of the Storage Entities that were formerly AVAILABLE now show as SYNCING.

The estimated time to complete for the sync tasks can be seen in the getStorageCluster output, but is not always reliable due to sync speed variations as the task proceeds.

After the sync is fully complete, the datastore states show ONLINE in VSA Manager.

Another possible cause: Missing the IP address on the back-end - the eth1 interface.

When you log into each appliance and run a command to reveal the IP addresses, you may see that the eth1 interface which would normally be the back-end IP address, is missing.

This may be because the appliance has detected a duplicate IP address - you can confirm this by:

Inside the appliance, cd /var/log
Review the file sva.log for messages of the form:

INFO - [SVADomain:BaseMessage@53] - fault: IP address "192.168.100.2" is already in use

This may be a legitimately duplicated IP address.

This issue may also occur if one of the mirrored disks in one or more of the Storage Entities may be in a FAULTED state.

For example, the following is a sample output from a 3 node cluster's getStorageCluster wscli command, related to the disks:

Storage entity 2:

Disk[0]
Name = Disk-d3d48020-cf2d-4049-89ed-ec3b396b2772
Id = d3d48020-cf2d-4049-89ed-ec3b396b2772
Owner Id = 965246b0-1ea0-4a91-be3a-18acf58f9d3b
State = FAULTED
Synced = false
Disk[1]
Name = Disk-91604af7-6fcb-4875-af85-ba3752b0e21a
Id = 91604af7-6fcb-4875-af85-ba3752b0e21a
Owner Id = 391f462f-c16b-4658-825e-5115f134d76c
State = ONLINE
Synced = true

If you experience this isue:

Collect all logs for the VSA, vCenter Server, and ESXi hosts.
Open a Support Request with VMware support and quote this Knowledge Base article ID (2034943) in the problem description. For more information, see How to Submit a Support Request.

03/12/2013 - Added additional cause

Knowledge Base

VMware Storage Appliance - Understanding and Investigating Degraded States (2034943)

Symptoms

Cause

Resolution

See Also

Update History

Request a Product Feature

Feedback

Did this article help you?
This article resolved my issue. This article did not resolve my issue. This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)

1 2 3 4 5
2 Ratings

1 2 3 4 5
2 Ratings