Gluster Operations

This page summarizes some operation tips on Gluster for daily business purposes.

Create, remove or change gluster volumes

This command should be run on the ansible master. You can find the a list of OpenShift clusters and their corresponding ansible masters here.

Use screen or tmux!

Note: use a filter to run only on the volumes you want to change.

-e 'mungg_gluster_volume_name_filter=^gluster-pv5(4[5-9]|5[0-4])$'

More details regarding the use of filters and how Gluster volumes are managed with mungg can be found in the Gluster volume management documentation.

Example:

ansible-playbook /usr/share/mungg/playbooks/gluster-volumes.yml

Startup Checks

The following commands should be run on one of the storage nodes.

Check whether bricks are up

The following snippet will list volumes that are not up. No output = all volumes are up

for v in $(gluster volume list); do /usr/lib64/nagios/plugins/check_gluster_volume --retries 0 "$v" | grep -vEq '(.*\[OK\]){3}' && echo "$v";
done

If the bricks aren’t up, check out the Troubleshooting Bricks section.

Check Gluster client sessions

To check, if the client and bricks are connected correctly. The Gluster client must have a connection to all bricks, if this is not the case, the volume is under a continuously healing and the monitoring heal checks are flapping. Restarting the pod that has mounted the volume is the only known workaround to recover all client sessions. Inform the customer to restart the pod, if the responsibility is not on VSHN side:

gluster volume status $name clients

Check all volumes for missing clients sessions:

for v in $(gluster volume list); do gluster volume status "$v" clients | grep "Clients connected" | sort | uniq -c | wc -l | grep 2 > /dev/null && echo "$v";
done

Check whether all volumes are healed

The heal process may take time, after all bricks are up and the Gluster clients have been reconnected!

The following snippet will list volumes that are not fully healed yet.

for v in $(gluster volume list); do /usr/lib64/nagios/plugins/check_gluster_volume_heal --retries 0 "$v" | grep -vEq '(.*\[OK\]){3}' && echo "$v";
done

For further troubleshooting tipps see Troubleshooting Healing Volumes.