Recently, I was applying security updates using .zip file on my ESXi hosts and updates failed on one of the host with error – [Errno 32] Broken pipe vibs = VMware_locker_tools-light_6.5.0-1.47.8285314
I already wrote a blog how to fix it, you can check here. However this seems to be a workaround as next time when i tried to apply the updates, it failed again. Also, VMware tools were failing for underlying VMs, I wrote an article for that as well, check here
The issue seems to be with ESXi’s store partition which is corrupted, locker point’s to store, and productLocker is dependent on locker. If Product locker is broken, it will display red. Login into ESXi host and from root directory, run “ls / -lah”
In order to fix the issue permanently, we would need to delete the existing store partition and create new one. Before proceeding with that, first, we need to copy the “var” & “packages” folder from store partition to a local server or another VMFS datastore. If store partition is corrupted and packages folder has been renamed to something else, you can use the find command to find the content of packages folder, I used command, find /store/ -name “6.5.0” . You can also use this blog which may help to identify the packages directory. Once you have the correct packages folder, copy and save it.
Below are the steps which we need to use to re-create store partition:
Identify the device holding the Store vFat partition using below command –
vmkfstools -Ph Store_partition_location
Example: vmkfstools -Ph /vmfs/volumes/560531a1-98ec1cba-37f1-a0369f7b3900
[Output of command]
vfat-0.04 (Raw Major Version: 0) file system spanning 1 partitions.
File system label (if any):
Capacity 285.8 MB, 1.8 MB available, file block size 8 KB, max supported file size 0 bytes
Partitions spanned (on “disks”):
Is Native Snapshot Capable: NO
Make a note of the vmhba device value. In this example: mpx.vmhba32:C0:T0:L0:8
To format the vFat partition:
vmkfstools -C vfat /vmfs/devices/disks/device_value
Note: Ensure that the command is run when the prompt is at the root directory. Formatting the device when the prompt is at store location gives the output as “Resource busy”
Example: vmkfstools -C vfat /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:8
[Output of command]
create fs deviceName:’/vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:8′, fsShortName:’vfat’, fsName:'(null)’
Checking if remote hosts are using this device as a valid file system. This may take a few seconds…
Creating vfat file system on “mpx.vmhba32:C0:T0:L0:8” with blockSize 1048576 and volume label “none”.
Filesystem was created but mount failed on device “mpx.vmhba32:C0:T0:L0:8”.: Not found.
Successfully created new volume: 5b3a92e5-cd01dbd6-9d6c-a0369f7b3900
Make a note of the new UUID created by above command, in our case “5b3a92e5-cd01dbd6-9d6c-a0369f7b3900“
To recreate the link to store, first we need to remove the old store link. below are steps –
Recreate a symbolic link:
ln -s location of new volume /store
Example: ln -s /vmfs/volumes/5b3a92e5-cd01dbd6-9d6c-a0369f7b3900 /store
Verify the status using cmmand:
store -> new Symbolic link
Now we need to copy the package folder back to the new store location. we can use below command:
cp -r local_datastore_location new_store_location
Now, we are done with all the steps and ESXi hosts needs to be rebooted.
I hope this has been informative for you. Do comment in case of any query.