Troubleshooting Healing Volumes

Heal

To check for inconsistencies that require healing, run the command below. If there’s no problem, all three should have number of entries equal to zero.

gluster volume heal $name info

To start healing a Gluster volume execute:

gluster volume heal $name

Full heal

To check for inconsistencies that require healing, run the subsequent command. If there’s no problem, all three should have number of entries equal to zero

gluster volume heal $name info

A full heal may help in cases, Gluster still has an inconsistency after a normal heal.

This may produce a lot of IO/Traffic and Load

gluster volume heal $name full

Transport Endpoint is not Connected

The heal info may give the following for one of the bricks:

# gluster volume heal $name info
Brick storage.host123.io:/data/shared_ssd/gluster-pvxx/brick
Status: Transport endpoint is not connected
Number of entries: -

In this case try to force-start the brick:

gluster volume start gluster-pvxx force

If it is currently running, you may need to stop it first:

gluster volume stop gluster-pvxx

After this operation, you must restart any pod that has mounted this pv as a volume.

Upstream documentation:

Brick down and Gluster volume fails to 'force' start

Split-Brain

gluster v heal gluster-pvxx split-brain source-brick storagex.lab.zrh.appuio.ch:/data/gluster-pvxx/brick $FILE
Healed gfid:53c85f9b-8761-4022-8d9f-afe21517f176.
Status: Connected
Number of healed entries: 1

gluster v heal gluster-pvxx split-brain latest-mtime $FILE

gluster v heal gluster-pvxx split-brain bigger-file $FILE

Upstream documentation:

If you require to mount all the bricks directly to be able to compare them you may use the splitbrain tool.