Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

The datarecovery service in the VMware Data Recovery (VDR) appliance fails with the message: kernel: Out of memory (2010589)

Symptoms

  • The Datarecovery service crashes intermittently in the VMware Data Recovery (VDR) appliance.
  • The VDR appliance uses a large amount of memory while running.
  • You see this VDR appliance plug-in event message:

    Task terminated unexpectedly, possibly due to a power failure or system crash

  • Following the out of memory (OOM) event, a Recatalog and Integrity Check begins upon restarting the appliance and no backups run during this period.
  • In the /var/log/messages file, you see entries similar to:

    kernel: Out of memory: Killed process 6990, UID 0, (datarecovery).
    kernel: Out of memory: Killed process 6991, UID 0, (datarecovery).
    kernel: Out of memory: Killed process 6996, UID 0, (datarecovery).

    kernel: datarecovery invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
    kernel: datarecovery invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
    kernel: udevd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=-17
    kernel: udev_run_hotplu invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=-17
    kernel: datarecovery invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
    - oom-killer killed datarecovery process.
    ...
    kernel: Out of memory: Killed process 6996, UID 0, (datarecovery).
    kernel: udev_run_hotplu invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=-17
    kernel: Call Trace:
    kernel: [<ffffffff800c7351>] out_of_memory+0x8e/0x2f3
    kernel: [<ffffffff8002e27c>] __wake_up+0x38/0x4f
    kernel: [<ffffffff8000f540>] __alloc_pages+0x27f/0x308
    kernel: [<ffffffff80008e9f>] __handle_mm_fault+0x73c/0x1039
    kernel: [<ffffffff80066b55>] do_page_fault+0x4cb/0x874
    kernel: [<ffffffff8001ad52>] remove_vma+0x5d/0x64
    kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84
    ...
    - The datarecovery process was restarted by watchdog-datarecovery
    watchdog-datarecovery: Executing '/usr/sbin/datarecovery'
    datarecovery: Starting VMware Data Recovery, version 2.0.0.1861
    datarecovery: Chunkdedupe WARNING: == Super Block Information ==
    datarecovery: Chunkdedupe WARNING: Version: 3.0

Resolution

This issue has been resolved in VMware Data Recovery 2.01. To download the latest version, visit the VMware Data Recovery download page.

To work around this issue with earlier versions of VMware Data Recovery, increase the memory resource and reservation allocated to the VDR appliance.

To increase the memory allocation:
  1. Ensure that the VDR appliance is idle and that there are no Integrity checks, Recatalog tasks, scheduled backups, or other tasks running.

    Note: The Recatalog and Integrity checks start running once the appliance is restarted.

  2. Open an SSH session to the appliance, then stop the datarecovery service by running the command:

    service datarecovery stop

    Note: This step is optional, but may speed up the shutdown of the VDR virtual machine.

  3. Shut down the VDR virtual machine.
  4. Go to Edit Settings of the VDR virtual machine and set these options, where applicable:

    • For a VDR appliance with Single Dedupe store:

      Memory: 4 GB
      Memory Reservation: 2 GB

    • For a VDR appliance with Two Dedupe stores:

      Memory: 6 GB
      Memory Reservation: 4 GB

  5. Power on the VDR virtual machine appliance.
  6. Monitor the VDR virtual machine performance using vCenter Server performance charts, and monitor the performance of the datarecovery service with the top command.

Additional Information

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

For translated versions of this article, see:

See Also

Update History

09/05/2012 - Added note to step 2 02/05/2013 - Added resolution in VMware Data Recovery 2.01

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 3 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 3 Ratings
Actions
KB: