VMware ESXi Patching Error – [Errno 32] Broken pipe vibs = VMware_locker_tools-light | [DatabaseIOError] Error in purging old directory | Permanent Fix

Recently, I was applying security updates using .zip file on my ESXi hosts and updates failed on one of the host with error – [Errno 32] Broken pipe vibs = VMware_locker_tools-light_6.5.0-1.47.8285314

I already wrote a blog how to fix it, you can check here. However this seems to be a workaround as next time when i tried to apply the updates, it failed again. Also, VMware tools were failing for underlying VMs, I wrote an article for that as well, check here

The issue seems to be with ESXi’s store partition which is corrupted, locker point’s to store, and productLocker is dependent on locker. If Product locker is broken, it will display red. Login into ESXi host and from root directory, run “ls / -lah”

In order to fix the issue permanently, we would need to delete the existing store partition and create new one. Before proceeding with that, first, we need to copy the “var” & “packages” folder from store partition to a local server or another VMFS datastore. If store partition is corrupted and packages folder has been renamed to something else, you can use the find command to find the content of packages folder, I used command, find /store/ -name “6.5.0” . You can also use this blog which may help to identify the packages directory. Once you have the correct packages folder, copy and save it.

Below are the steps which we need to use to re-create store partition:

Identify the device holding the Store vFat partition using below command –
vmkfstools -Ph Store_partition_location
Example: vmkfstools -Ph /vmfs/volumes/560531a1-98ec1cba-37f1-a0369f7b3900
[Output of command]
vfat-0.04 (Raw Major Version: 0) file system spanning 1 partitions.
File system label (if any):
Mode: private
Capacity 285.8 MB, 1.8 MB available, file block size 8 KB, max supported file size 0 bytes
UUID: 560531a1-98ec1cba-37f1-a0369f7b3900
Partitions spanned (on “disks”):
mpx.vmhba32:C0:T0:L0:8
Is Native Snapshot Capable: NO

Make a note of the vmhba device value. In this example: mpx.vmhba32:C0:T0:L0:8

To format the vFat partition:
vmkfstools -C vfat /vmfs/devices/disks/device_value
Note: Ensure that the command is run when the prompt is at the root directory. Formatting the device when the prompt is at store location gives the output as “Resource busy”
Example: vmkfstools -C vfat /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:8
[Output of command]
create fs deviceName:’/vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:8′, fsShortName:’vfat’, fsName:'(null)’
deviceFullPath:/dev/disks/mpx.vmhba32:C0:T0:L0:8 deviceFile:mpx.vmhba32:C0:T0:L0:8
Checking if remote hosts are using this device as a valid file system. This may take a few seconds…
Creating vfat file system on “mpx.vmhba32:C0:T0:L0:8” with blockSize 1048576 and volume label “none”.
Filesystem was created but mount failed on device “mpx.vmhba32:C0:T0:L0:8”.: Not found.


Successfully created new volume: 5b3a92e5-cd01dbd6-9d6c-a0369f7b3900

Make a note of the new UUID created by above command, in our case “5b3a92e5-cd01dbd6-9d6c-a0369f7b3900

To recreate the link to store, first we need to remove the old store link. below are steps –
Remove store:
rm store
Recreate a symbolic link:
ln -s location of new volume /store
Example: ln -s /vmfs/volumes/5b3a92e5-cd01dbd6-9d6c-a0369f7b3900 /store

Verify the status using cmmand:
ls -l
store -> new Symbolic link

Now we need to copy the package folder back to the new store location. we can use below command:
cp -r local_datastore_location new_store_location

Now, we are done with all the steps and ESXi hosts needs to be rebooted.

I hope this has been informative for you. Do comment in case of any query.

Advertisement

6 thoughts on “VMware ESXi Patching Error – [Errno 32] Broken pipe vibs = VMware_locker_tools-light | [DatabaseIOError] Error in purging old directory | Permanent Fix

  1. Great article!! Avoid a complete re install of one of my ESXi Hosts. Re-Creating the partition works like a charm even it was a 6.7.

    Liked by 1 person

  2. I tried following your instructions but I get the following error when I try to create the partition.
    Error: Read-only file system

    Any ideas?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s