Monday, December 8, 2008

VMWare Snapshot Files (*.delta.vmdk) will not delete

Last week I discovered that one of my VMWare ESX servers had a few delta.vmdk files hanging out in one of my VM directories. After some investigation, I found out that the delta files were continuing to grow and never merging back into the flat flat.vmdk file.

These delta files represent snapshots of the VM. Normally, one would be able to view them in the snapshot manager of the VMWare infrastructure client. However, my delta files did not show up. A quick check of the VMWare knowledge base told me to create a new snapshot through the snapshot manager and then select "delete all." This should have removed and merged all the delta files back into the flat file.

No dice. It ran for about 15 minutes (hammering my host in the process) and then failed.

Time to move onto the command line. I opened a PuTTY session to my ESX 3.5 server.

The first command will create a new snapshot for us named "test":
vmware-cmd /vmfs/volumes/[volume name]/[dir of VM]/[VM name].vmx createsnapshot test test

The next command will remove all the snapshots and write the changes back to the flat file:
vmware-cmd //vmfs/volumes/[volume name]/[dir of VM]/[VM name].vmx removesnapshots

To make sure this was running, I opened a new PuTTY session and typed the follwoing:

/vmfs/volumes/[volume name]/[dir of VM]/watch 'ls -lah *.vmdx'

Every two seconds I would get an update. I had a total of 5 delta files (including the test one I had created) that need to be merged. When this happens, 5 writes to 4, 4 writes to 3, 3 writes to 2, 2 writes to 1 and 1 writes to the flat. After is is done, all the *.delta.vmdx files will be removed. In my case, this process took 16 hours. Yeah, I had some pretty large delta files.

I spoke to VMWare and they told me that this process can take a very long time if the delta files are large. IN addition, they confirmed with me that the GUI will time out after 15 minutes, so if you have large delta files you MUST use the command line.

If you get caught with some delta files that will not delete, this should do the trick for you. Just remember to be patient and DO NOT close the PuTTY session until it is finished. You will know it is done when you get your # back.