Gluster Operations
This page summarizes some operation tips on Gluster for daily business purposes.
Create, remove or change gluster volumes
This command should be run on the ansible master. You can find the a list of OpenShift clusters and their corresponding ansible masters here.
Use screen or tmux!
Note: use a filter to run only on the volumes you want to change. -e 'mungg_gluster_volume_name_filter=^gluster-pv5(4[5-9]|5[0-4])$' |
More details regarding the use of filters and how Gluster volumes are managed with mungg can be found in the Gluster volume management documentation.
Example:
ansible-playbook /usr/share/mungg/playbooks/gluster-volumes.yml
Startup Checks
The following commands should be run on one of the storage nodes.
Check whether bricks are up
The following snippet will list volumes that are not up. No output = all volumes are up
for v in $(gluster volume list); do /usr/lib64/nagios/plugins/check_gluster_volume --retries 0 "$v" | grep -vEq '(.*\[OK\]){3}' && echo "$v"; done
If the bricks aren’t up, check out the Troubleshooting Bricks section.
Check Gluster client sessions
To check, if the client and bricks are connected correctly. The Gluster client must have a connection to all bricks, if this is not the case, the volume is under a continuously healing and the monitoring heal checks are flapping. Restarting the pod that has mounted the volume is the only known workaround to recover all client sessions. Inform the customer to restart the pod, if the responsibility is not on VSHN side:
gluster volume status $name clients
Check all volumes for missing clients sessions:
for v in $(gluster volume list); do gluster volume status "$v" clients | grep "Clients connected" | sort | uniq -c | wc -l | grep 2 > /dev/null && echo "$v"; done
Check whether all volumes are healed
The heal process may take time, after all bricks are up and the Gluster clients have been reconnected! |
The following snippet will list volumes that are not fully healed yet.
for v in $(gluster volume list); do /usr/lib64/nagios/plugins/check_gluster_volume_heal --retries 0 "$v" | grep -vEq '(.*\[OK\]){3}' && echo "$v"; done
For further troubleshooting tipps see Troubleshooting Healing Volumes.