Snapshot removal task stops at 99%
Article ID: 341355
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
This article provides steps to monitor snapshot removal on ESXi using the watch command.
Symptoms:
- The snapshot removal task hang at 95% or 99%.
- You cannot observe the progress when you delete a snapshot.
Note: For additional symptoms, see the Additional Information section.
Environment
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
Cause
Note: Time involved to commit snapshots is environmental and subjective.
To estimate the time it will take to remove snapshots, see Estimate the time required to consolidate virtual machine snapshots
The consolidation/Removal task hanging at 99% is in most cases related to:
To estimate the time it will take to remove snapshots, see Estimate the time required to consolidate virtual machine snapshots
The consolidation/Removal task hanging at 99% is in most cases related to:
- Virtual Machine snapshot size on disk (Delta or sesparse file size).
- Storage performance.
- Network connection interrupted between ESXI host and vCenter.
- Size of the Virtual Machine.
Resolution
Finding consolidation tasks that are at 99%
Monitor using watch in ESXi 6.x/ 7.x
For 5.x, see Monitor snapshot deletion using esxtop command (2146232)
Monitor using vim-cmd in ESXi
Finding consolidation tasks that are at 99%
- Log in as root to the ESXi host using SSH.
- Navigate to the virtual machine directory containing vmdk virtual disk files. Run the cd command to change the current directory.
For example:
cd /vmfs/volumes/Datastore_name/Virtual_Machine_name/
- Run less testdev02.vmx | grep vmdk to list vmdk (disks) of the VM in the vmx (configuration file of VM):
For example:
less testdev02.vmx |grep vmdk
scsi0:0.fileName = "testdev02.vmdk"
scsi0:1.fileName = "testdev02_1-000003.vmdk"
scsi0:2.fileName = "testdev02_2-000002.vmdk"
scsi0:0.fileName = "testdev02.vmdk"
scsi0:1.fileName = "testdev02_1-000003.vmdk"
scsi0:2.fileName = "testdev02_2-000002.vmdk"
- Run this command to list the snapshot files in the directory:
- less testdev02.vmx | grep *sesparse.vmdk
- For VMFS5 volume: less testdev02.vmx | grep *delta.vmdk
Listing the snapshot of the VM.
-rw------- 1 root root 18.2G Oct 29 10:04 testdev02_2-000001-sesparse.vmdk
-rw------- 1 root root 943.6G Oct 29 10:09 testdev02_1-000002-sesparse.vmdk
-rw------- 1 root root 1.8T Oct 29 10:09 testdev02_1-000001-sesparse.vmdk
-rw------- 1 root root 1.7G Oct 29 20:03 testdev02_2-000002-sesparse.vmdk
-rw------- 1 root root 3.1T Oct 29 20:03 testdev02_1-000003-sesparse.vmdk
-rw------- 1 root root 943.6G Oct 29 10:09 testdev02_1-000002-sesparse.vmdk
-rw------- 1 root root 1.8T Oct 29 10:09 testdev02_1-000001-sesparse.vmdk
-rw------- 1 root root 1.7G Oct 29 20:03 testdev02_2-000002-sesparse.vmdk
-rw------- 1 root root 3.1T Oct 29 20:03 testdev02_1-000003-sesparse.vmdk
Note: Sesparse is a default format for all delta disks on the VMFS6 datastores. On VMFS5, Sesparse is used for virtual disks of the size 2 TB and larger.
- In this example we have snapshots approximately 6 TB and this could be the main reason for the progress of the task. The process will continue to show 99% until the snapshot removal task completes and VM is using the flat files of the VMDK.
- Find the consolidation task with this command: vim-cmd vimsvc/task_list
For example:
[root@ESXi:~] vim-cmd vimsvc/task_list
(ManagedObjectReference) [
'vim.Task:haTask-91-vim.vm.Snapshot.remove-321615207',
'vim.Task:haTask-ha-host-vim.host.ImageConfigManager.installDate-321689752',
'vim.Task:haTask-ha-host-vim.host.ServiceSystem.start-321689774',
'vim.Task:haTask-ha-host-vim.host.ServiceSystem.refresh-321689779'
]
(ManagedObjectReference) [
'vim.Task:haTask-91-vim.vm.Snapshot.remove-321615207',
'vim.Task:haTask-ha-host-vim.host.ImageConfigManager.installDate-321689752',
'vim.Task:haTask-ha-host-vim.host.ServiceSystem.start-321689774',
'vim.Task:haTask-ha-host-vim.host.ServiceSystem.refresh-321689779'
]
- View the details of the consolidation with this command: vim-cmd vimsvc/task_info <task_name_from_step_6>
[root@ESXi:~] vim-cmd vimsvc/task_info haTask-91-vim.vm.Snapshot.remove-321615207
(vim.TaskInfo) {
key = "haTask-91-vim.vm.Snapshot.remove-321615207",
task = 'vim.Task:haTask-91-vim.vm.Snapshot.remove-321615207',
description = (vmodl.LocalizableMessage) null,
name = "vim.vm.Snapshot.remove",
descriptionId = "vm.Snapshot.remove",
entity = 'vim.VirtualMachine:81',
entityName = "TESTDEV",
locked = <unset>,
state = "running",
cancelled = false,
cancelable = true,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 99,
reason = (vim.TaskReasonUser) {
userName = "vpxuser"
},
queueTime = "2019-10-29T10:08:21.403596Z",
startTime = "2019-10-29T10:08:21.404185Z",
completeTime = <unset>,
eventChainId = -1078812089,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
activationId = <unset>
}
(vim.TaskInfo) {
key = "haTask-91-vim.vm.Snapshot.remove-321615207",
task = 'vim.Task:haTask-91-vim.vm.Snapshot.remove-321615207',
description = (vmodl.LocalizableMessage) null,
name = "vim.vm.Snapshot.remove",
descriptionId = "vm.Snapshot.remove",
entity = 'vim.VirtualMachine:81',
entityName = "TESTDEV",
locked = <unset>,
state = "running",
cancelled = false,
cancelable = true,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 99,
reason = (vim.TaskReasonUser) {
userName = "vpxuser"
},
queueTime = "2019-10-29T10:08:21.403596Z",
startTime = "2019-10-29T10:08:21.404185Z",
completeTime = <unset>,
eventChainId = -1078812089,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
activationId = <unset>
}
Monitor using watch in ESXi 6.x/7.x
To monitor the VMDK snapshot and base disks which are currently being updated, run this watch command:This command refreshes the file referred by the on going task every 2 seconds.
# watch -d 'ls -luth | grep -E "delta|flat|sesparse"'
where:
-d highlights the differences between successive updates
In addition, run this command to monitor the time stamp update of the base disks to confirm if the process is working. The result lists the files by their modification date.
ls -lrth |grep -E "flat|delta|sesparse"
Notes:
- In ESX/ESXi 4.0 Update 2 and later, the process works differently in that the data in snapshots (deltas) are written directly to the base disk (flat).
- If there are more than 10 snapshots, run this command to monitor the snapshot commit process and to prevent the screen from filling with many files:
#while true;do date;ls -lht *vmdk|head -10;echo ________;sleep 3;done
- You can quit the consolidation process monitoring by pressing Ctrl + C.
Additional Information
VMware Skyline Health Diagnostics for vSphere - FAQ
You experience these additional symptoms:
- The hostd process does not respond.
- Storage vMotion migration stays at 18% for a long time.
Tech Support Mode for Emergency Support
Consolidating snapshots in ESX/ESXi 3.x and 4.x
Identifying disks when working with VMware ESXi/ESX
Understanding VM snapshots in ESXi / ESX
Using Tech Support Mode in ESXi 4.1, ESXi 5.x, and ESXi 6.x
Editing files on an ESX host using vi or nano
ESXi/ESX でのスナップショットの削除タスクが 99% で停止する
在 ESXi/ESX 中，快照删除任务在进行到 99% 时停止
How to monitor snapshot deletion using the vim-cmd command
How to monitor snapshot deletion using esxtop command
Aufgabe zum Entfernen eines Snapshots wird in ESXi/ESX bei 99 % beendet
Impact/Risks:
The is no way to stop or cancel the consolidation process without risking data corruption. Do not attempt to create a new snapshot on the consolidating virtual machine.
Feedback
Yes
No
Powered by
