Knowledge Base & Discussion Forum

ISSUE: Critical issue with VMware's CBT (Change Block Tracking) function after extending VMDK (Possible data corruption for VMware VM backup in VDDK mode) (3169)

Known issues or frequently asked questions on the VMware VM module (vCentre, ESX / ESXi, Workstation, Fusion and Server).

ISSUE: Critical issue with VMware's CBT (Change Block Tracking) function after extending VMDK (Possible data corruption for VMware VM backup in VDDK mode) (3169)

Postby admin » Fri Oct 31, 2014 3:41 pm

Article ID: 3169
Reviewed: 07/11/2014

Product Version:
AhsayOBM: Pre-7.3.2.0
OS: All platforms

Problem Description:
Related to an issue identified by VMware with the CBT (Change Block Tracking) function ( VMware KB 2090639 ).

There may a potential for data corruption for virtual machines backed up by AhsayOBM's VMware VM backup module in VDDK mode (e.g. CBT enabled), when extending a virtual disk (vmdk) file past a 128 GB boundary (refer to Also See section).

Important:
This is an issue identified by VMware with its CBT function that affects AhsayOBM's VMware VM backup module in VDDK mode.



Cause:
According to VMware, the Virtual Disk Development Kit (VDDK) method QueryChangedDiskAreas() used by AhsayOBM to determine which areas of a virtual disk have changed since the last backup, this information is used to decide which blocks to include in an incremental backup (e.g. sub-sequence VMware VM backup in VDDK mode).

This issue occurs when you expand a virtual disk vmdk file which has Change Block Tracking (CBT) enabled past any 128 GB boundary (refer to Also See section).  When the disk is extended the change tracking data becomes unreliable.  Due to the faulty changed block information, some changed blocks might not be captured by an incremental backup, so that a restoring from an incomplete backup could cause a data loss.

A virtual machine is at risk if all of these occur in sequence:

  1. Changed Block Tracking is enabled.
  2. Virtual disk is extended to a size above 128 GB.
  3. A VDDK incremental backup is performed.
  4. The virtual machine is restored from that VDDK incremental backup.

Resolution:
This is a known issue affecting VMware ESXi 4.x and ESXi 5.x.

As this is not an Ahsay related issue, there is no way for AhsayOBM / AhsayOBS to detect if the backed up guest virtual machines are corrupted.

This issue is resolved in:

  • ESXi 5.0, Patch Release ESXi500-201412001
  • ESXi 5.1 Update 3
  • ESXi 5.5, Patch Release ESXi550-201501001

Refer to VMware's KB article ( VMware KB 2090639 - Section under Resolution) for patch details.

Currently, there is no resolution for ESXi 4.x.

To workaround this issue:

  1. Disable Changed Block Tracking (CBT) for the affected virtual machine.
  2. Take a snapshot (Does suspend / resume).
  3. Delete the snapshot (To recover space and performance).
  4. Enable CBT for the affected virtual machine.
  5. Take a snapshot (Again with the suspend / resume).
  6. Delete the snapshot.

The next backup after toggling CBT will be a full backup of the virtual machine.

Discard any existing sub-sequence backups that were captured after growing the disk, as they can be incomplete.

For instruction on how to reset CBT, see:

FAQ: How to reset Change Block Tracking (CBT) for VMware VM backup (3136).

We would recommend all users to perform a manual CBT reset for all virtual machines that had their virtual disks expanded at some point. Perform a backup with AhsayOBM afterward. The next backup after toggling CBT will be a full backup of the virtual machine, which may take longer to complete.

Note:
Since there is no way to detect if the previously backed up data are corrupted or not, we cannot guarantee that the guest VMs are recoverable.  You should also consider to re-create a new VMware VM backup set (after resetting the CBT setting) for the corresponding guest VMs (e.g. performing a Seed Load backup to start over).



Also See:
Some questions that you may have

  1. Is there a way to determine if a virtual disk has been expanded?

    Answer) Customers should rely on their own change control records to determine if a virtual disk has been expanded. This information is not tracked in the virtual machine.

  2. Are virtual machines grown in smaller increments affected?

    Answer) The amount of space the virtual disk is extended is not relevant, the increment of space by which a virtual disk is extended is not relevant. The virtual machine has this issue when its disk is grown past any 128 GB boundary in absolute size.  The issue is triggered at other sizes which are a power of 2 from 128 GB up. For example: 256 GB, 512 GB, and 1024 GB.

To check if CBT is enabled:

  • In the home directory of the virtual machine, verify if there is a vmname-ctk.vmdk file for one or more virtual hard disks.
  • To Query the advanced configuration parameters for the Virtual Machine:

    • Right-click the virtual machine and click Edit Settings.
    • Click the Options tab.
    • Click General under the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens.
    • Search for the ctkEnabled parameter entry for each disk and note if it is enabled or not.

Keywords:
VMware, VM, license, licensing, vddk, gvm
admin
Site Admin
 
Posts: 968
Joined: Mon Jul 10, 2006 3:19 pm

Return to VMware

Who is online

Users browsing this forum: No registered users and 0 guests

Looking for Rbackup Alternative | Vembu Alternative | Novastor Alternative | Asigra Alternative | BackupAgent Alternative? Try our product.


A wholly owned subsidiary of Ahsay Backup Software Development Company Limited  [HKEx Stock Code: 8290]