AppCat Rollback
For on-call engineers responding to an AppCat release issue. Use this script to quickly revert affected Claims to a previous CompositionRevision and restore them once the issue is resolved.
Terminology
-
Claim - The user-facing, namespaced resource (for example
VSHNPostgreSQLin namespacemy-app). This is what customers create and interact with. -
Composite Resource (XR) - The cluster-scoped resource automatically created by Crossplane when a Claim is made (for example
XVSHNPostgreSQL). Claims reference their XR viaspec.resourceRef.name. -
CompositionRevision - A versioned snapshot of a Composition template. Each has a unique name (like
vshnpostgresql.vshn.appcat.vshn.io-a1b2c3d) and a revision label (likev3.59.0-v4.172.0). -
Function - The Crossplane function package that contains the logic to render a Composition. If a CompositionRevision’s function is MISSING, that revision cannot be used.
Before you start
|
Make sure you can run
|
|
If your user isn’t a cluster admin, pass |
Copy the script below into a file named rollback.sh and make it executable:
# Copy/paste the contents from the "Script" section into this file:
$EDITOR rollback.sh
# Make it executable (either run in-place or move onto your PATH)
chmod +x rollback.sh
Script
rollback.sh
#!/usr/bin/env bash
set -euo pipefail
AS_ADMIN_FLAG=""
KUBECTL="${KUBECTL:-kubectl}"
usage(){ cat <<'EOF'
Usage: rollback.sh [options]
-t TYPE Claim type (for example vshnpostgresqls.vshn.appcat.vshn.io).
Comma-separated allowed for multiple types (requires -r for rollback).
-i NAME Specific claim instance name (requires -n NAMESPACE and single -t).
-n NAMESPACE Namespace for the claim instance. Can be used with -i for a single claim,
or without -i to target all claims of the specified type in that namespace.
-r REVISION Explicit revision label (for example v3.59.0-v4.172.0).
Required when using multiple types. Optional otherwise (autodetects per claim).
Not used with --unpin.
--list-revisions List all available CompositionRevisions for the given type(s) and exit.
--unpin Remove compositionRevisionSelector to return to automatic policy.
--dry-run Show what would be patched without making actual changes.
-y|--yes Skip confirmation prompt and proceed immediately.
--as-admin Run kubectl commands as system:admin.
-h|--help Show help.
Examples:
# List available revisions for a type
rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io --list-revisions
# Dry-run: preview what would be rolled back (autodetect)
rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io --dry-run
# Rollback all claims of one type (autodetect revision per claim)
rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io
# Rollback all claims of one type in a specific namespace (autodetect revision per claim)
rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io -n my-namespace
# Rollback all claims of one type to specific revision
rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io -r v3.59.0-v4.172.0
# Rollback single claim (autodetect)
rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io -i my-postgres -n my-namespace
# Rollback multiple types to specific revision
rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io,vshnnextclouds.vshn.appcat.vshn.io -r v3.59.0-v4.172.0
# Unpin all claims of one type
rollback.sh --unpin -t vshnpostgresqls.vshn.appcat.vshn.io
# Unpin all claims of one type in a specific namespace
rollback.sh --unpin -t vshnpostgresqls.vshn.appcat.vshn.io -n my-namespace
# Unpin single claim
rollback.sh --unpin -t vshnpostgresqls.vshn.appcat.vshn.io -i my-postgres -n my-namespace
# Unpin multiple types
rollback.sh --unpin -t vshnpostgresqls.vshn.appcat.vshn.io,vshnnextclouds.vshn.appcat.vshn.io
EOF
}
fatal(){ echo "ERROR: $*" >&2; exit 1; }
warn(){ echo "WARN: $*" >&2; }
kubectl_cmd() {
"$KUBECTL" $AS_ADMIN_FLAG "$@"
}
list_instances(){
# List claims across all namespaces, returning "namespace name" pairs
kubectl_cmd get "$1" -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' || return 1
}
list_revisions_for_type(){
local claim_type="$1" base xr_type xr_name cur claim_ns claim_name
# Try to derive composition base from an actual claim's XR
# First, try to find any claim of this type
local first_claim="$(kubectl_cmd get "$claim_type" -A -o jsonpath='{.items[0].metadata.namespace}{" "}{.items[0].metadata.name}' 2>/dev/null || true)"
if [ -n "$first_claim" ]; then
read -r claim_ns claim_name <<<"$first_claim"
xr_name="$(kubectl_cmd get "$claim_type" "$claim_name" -n "$claim_ns" -o jsonpath='{.spec.resourceRef.name}' 2>/dev/null || true)"
if [ -n "$xr_name" ]; then
xr_type="x${claim_type}"
cur="$(kubectl_cmd get "$xr_type" "$xr_name" -o jsonpath='{.spec.compositionRevisionRef.name}' 2>/dev/null || true)"
if [ -n "$cur" ]; then
# Extract base from current revision (remove hash suffix)
base="$(printf '%s' "$cur" | sed -E 's/-[0-9a-f]{7,}$//')" || true
fi
fi
fi
# Fallback: if we couldn't derive base from a claim, try simple singularization
if [ -z "${base:-}" ]; then
warn "No claims found for type $claim_type, using fallback name derivation"
base="$(echo "$claim_type" | sed 's/s\(\.[^.]*\)/\1/')"
fi
echo "Available CompositionRevisions for $claim_type (base: $base):"
echo ""
printf "%-45s %-65s %-25s %s\n" "COMPOSITION REVISION" "REVISION LABEL" "CREATED" "FUNCTION STATUS"
printf "%-45s %-65s %-25s %s\n" "---------------------------------------------" "-----------------------------------------------------------------" "-------------------------" "---------------"
for r in $(
kubectl_cmd get compositionrevisions.apiextensions.crossplane.io \
--sort-by=.metadata.creationTimestamp -o name \
| sed 's|.*/||' \
| awk -v b="$base" 'index($0, b "-")==1 { print }' \
| tac
); do
timestamp="$(kubectl_cmd get compositionrevision "$r" -o jsonpath='{.metadata.creationTimestamp}' 2>/dev/null || echo "N/A")"
label="$(kubectl_cmd get compositionrevision "$r" -o go-template='{{ index .metadata.labels "metadata.appcat.vshn.io/revision" }}' 2>/dev/null || echo "<none>")"
# Check if the function exists
func_name="$(kubectl_cmd get compositionrevision "$r" -o jsonpath='{.spec.pipeline[0].functionRef.name}' 2>/dev/null || true)"
func_status="N/A"
if [ -n "$func_name" ]; then
if kubectl_cmd get function.pkg.crossplane.io "$func_name" >/dev/null 2>&1; then
func_status="EXISTS"
else
func_status="MISSING"
fi
fi
printf "%-45s %-65s %-25s %s\n" "$r" "$label" "$timestamp" "$func_status"
done
}
autodetect_prev_label(){
local claim_type="$1" claim_ns="$2" claim_name="$3" xr_type xr_name cur base prev lbl func_name
# Get XR name from claim's resourceRef
xr_name="$(kubectl_cmd get "$claim_type" "$claim_name" -n "$claim_ns" -o jsonpath='{.spec.resourceRef.name}' || true)"
if [ -z "$xr_name" ]; then
echo "ERROR: Claim $claim_type/$claim_name in namespace $claim_ns lacks .spec.resourceRef.name" >&2
return 1
fi
# Derive XR type by adding 'x' prefix
xr_type="x${claim_type}"
# Get current compositionRevisionRef from XR
cur="$(kubectl_cmd get "$xr_type" "$xr_name" -o jsonpath='{.spec.compositionRevisionRef.name}' || true)"
if [ -z "$cur" ]; then
echo "ERROR: XR $xr_type/$xr_name lacks .spec.compositionRevisionRef.name" >&2
return 1
fi
base="$(printf '%s' "$cur" | sed -E 's/-[0-9a-f]{7,}$//')" || true
if [ -z "$base" ]; then
echo "ERROR: Cannot derive base from '$cur' for XR $xr_type/$xr_name" >&2
return 1
fi
# Get all previous revisions (newest to oldest relative to current)
local all_prev
all_prev="$(
kubectl_cmd get compositionrevisions.apiextensions.crossplane.io \
--sort-by=.metadata.creationTimestamp -o name \
| sed 's|.*/||' \
| awk -v b="$base" -v c="$cur" 'index($0,b"-")==1{a[++n]=$0} END{for(i=1;i<=n;i++)if(a[i]==c&&i>1){for(j=i-1;j>=1;j--)print a[j]; exit}}'
)"
if [ -z "$all_prev" ]; then
echo "ERROR: No previous CompositionRevision for base '$base' (current=$cur) on XR $xr_type/$xr_name" >&2
return 1
fi
# Find the first previous revision with a valid function
while IFS= read -r prev; do
[ -n "$prev" ] || continue
# Check if function exists
func_name="$(kubectl_cmd get compositionrevision "$prev" -o jsonpath='{.spec.pipeline[0].functionRef.name}' 2>/dev/null || true)"
if [ -n "$func_name" ]; then
if ! kubectl_cmd get function.pkg.crossplane.io "$func_name" >/dev/null 2>&1; then
continue # Skip this revision, function is missing
fi
fi
# Check for revision label
lbl="$(kubectl_cmd get compositionrevision "$prev" \
-o go-template='{{ index .metadata.labels "metadata.appcat.vshn.io/revision" }}' 2>/dev/null || true)"
if [ -n "$lbl" ] && [ "$lbl" != "<no value>" ]; then
printf '%s\n' "$lbl"
return 0
fi
done <<<"$all_prev"
echo "ERROR: No usable previous CompositionRevision found for base '$base' (all previous revisions either missing function or revision label)" >&2
return 1
}
patch_claim(){
local claim_type="$1" claim_ns="$2" claim_name="$3" rev="$4"
if [ "${UNPIN:-0}" -eq 1 ]; then
# Unpin mode: remove the selector
if [ "${DRY_RUN:-0}" -eq 1 ]; then
echo "[DRY-RUN] Would unpin claim $claim_type/$claim_name in namespace $claim_ns (remove compositionRevisionSelector)"
else
echo "Unpinning claim $claim_type/$claim_name in namespace $claim_ns (removing compositionRevisionSelector)"
kubectl_cmd patch "$claim_type" "$claim_name" -n "$claim_ns" --type=merge \
-p '{"spec":{"compositionRevisionSelector":null}}'
fi
else
# Rollback mode: set the selector
if [ "${DRY_RUN:-0}" -eq 1 ]; then
echo "[DRY-RUN] Would patch claim $claim_type/$claim_name in namespace $claim_ns with revision=$rev"
echo "[DRY-RUN] Patch: {\"spec\":{\"compositionRevisionSelector\":{\"matchLabels\":{\"metadata.appcat.vshn.io/revision\":\"$rev\"}}}}"
else
echo "Patching claim $claim_type/$claim_name in namespace $claim_ns with revision=$rev"
kubectl_cmd patch "$claim_type" "$claim_name" -n "$claim_ns" --type=merge \
-p "{\"spec\":{\"compositionRevisionSelector\":{\"matchLabels\":{\"metadata.appcat.vshn.io/revision\":\"$rev\"}}}}"
fi
fi
}
# arg parsing
[ $# -gt 0 ] || { usage; exit 1; }
TYPES_CSV="" NAME="" NAMESPACE="" REV="" LIST_REVISIONS=0 DRY_RUN=0 SKIP_CONFIRM=0 UNPIN=0
while [ $# -gt 0 ]; do
case "$1" in
-t) TYPES_CSV="${2:?}"; shift 2;;
-i) NAME="${2:?}"; shift 2;;
-n) NAMESPACE="${2:?}"; shift 2;;
-r) REV="${2:?}"; shift 2;;
--list-revisions) LIST_REVISIONS=1; shift;;
--unpin) UNPIN=1; shift;;
--dry-run) DRY_RUN=1; shift;;
-y|--yes) SKIP_CONFIRM=1; shift;;
--as-admin) AS_ADMIN_FLAG="--as=system:admin"; shift;;
-h|--help) usage; exit 0;;
-*) fatal "Unknown option $1";;
*) fatal "Unexpected argument $1";;
esac
done
# validations
[ -z "$TYPES_CSV" ] && fatal "-t TYPE is required"
[ $UNPIN -eq 1 ] && [ $LIST_REVISIONS -eq 1 ] && fatal "Cannot combine --unpin with --list-revisions"
[ $UNPIN -eq 1 ] && [ -n "$REV" ] && fatal "Cannot specify -r with --unpin"
# Handle --list-revisions early exit
if [ $LIST_REVISIONS -eq 1 ]; then
TYPES_CSV=${TYPES_CSV//[[:space:]]/}
IFS=',' read -r -a types <<<"$TYPES_CSV"
FAILED=0
for claim_type in "${types[@]}"; do
if ! list_revisions_for_type "$claim_type"; then
FAILED=1
fi
echo ""
done
[ $FAILED -eq 0 ] || exit 1
exit 0
fi
[ -n "$NAME" ] && [ -z "$NAMESPACE" ] && fatal "-i requires -n NAMESPACE"
# build type list
TYPES_CSV=${TYPES_CSV//[[:space:]]/}
IFS=',' read -r -a types <<<"$TYPES_CSV"
# Additional validations
[ -n "$NAME" ] && [ ${#types[@]} -ne 1 ] && fatal "-i NAME requires exactly one TYPE (not comma-separated)"
[ ${#types[@]} -gt 1 ] && [ -z "$REV" ] && [ $UNPIN -eq 0 ] && fatal "Multiple types require -r REVISION (cannot autodetect)"
# Build list of what will be patched for confirmation
patches_preview=()
for claim_type in "${types[@]}"; do
claims=()
if [ -n "$NAME" ]; then
# Single claim specified
if ! kubectl_cmd get "$claim_type" "$NAME" -n "$NAMESPACE" >/dev/null 2>&1; then
fatal "Claim not found: $claim_type/$NAME in namespace $NAMESPACE"
fi
claims+=("$NAMESPACE $NAME")
else
# All claims of this type (optionally filtered by namespace)
if [ -n "$NAMESPACE" ]; then
# Get claims only in specified namespace
out="$(kubectl_cmd get "$claim_type" -n "$NAMESPACE" -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
else
# Get claims across all namespaces
out="$(list_instances "$claim_type" 2>/dev/null || true)"
fi
if [ -z "${out:-}" ]; then
if [ -n "$NAMESPACE" ]; then
echo "No claims of type $claim_type in namespace $NAMESPACE; skipping."
else
echo "No claims of type $claim_type; skipping."
fi
continue
fi
while IFS= read -r line; do
[ -n "$line" ] && claims+=("$line")
done <<<"$out"
fi
for claim_info in "${claims[@]}"; do
read -r claim_ns claim_name <<<"$claim_info"
[ -n "$claim_ns" ] && [ -n "$claim_name" ] || continue
rev="$REV"
if [ $UNPIN -eq 0 ]; then
# Only need revision for rollback mode
if [ -z "$rev" ]; then
if ! rev="$(autodetect_prev_label "$claim_type" "$claim_ns" "$claim_name" 2>&1)"; then
warn "$rev"
continue
fi
fi
else
# Unpin mode doesn't need revision
rev="(unpin)"
fi
patches_preview+=("$claim_type|$claim_ns|$claim_name|$rev")
done
done
# Show preview and ask for confirmation
if [ ${#patches_preview[@]} -eq 0 ]; then
echo "No claims to patch."
exit 0
fi
if [ $UNPIN -eq 1 ]; then
echo "The following claims will be unpinned:"
else
echo "The following claims will be patched:"
fi
echo ""
printf "%-50s %-30s %-30s %s\n" "TYPE" "NAMESPACE" "NAME" "REVISION"
printf "%-50s %-30s %-30s %s\n" "--------------------------------------------------" "------------------------------" "------------------------------" "------------------------------"
for patch_info in "${patches_preview[@]}"; do
IFS='|' read -r p_type p_ns p_name p_rev <<<"$patch_info"
printf "%-50s %-30s %-30s %s\n" "$p_type" "$p_ns" "$p_name" "$p_rev"
done
echo ""
if [ "${DRY_RUN:-0}" -eq 0 ] && [ "${SKIP_CONFIRM:-0}" -eq 0 ]; then
if [ $UNPIN -eq 1 ]; then
read -rp "Proceed with unpinning? (y/n): " confirm
else
read -rp "Proceed with rollback? (y/n): " confirm
fi
if [[ ! "$confirm" =~ ^[Yy]$ ]]; then
echo "Aborted."
exit 0
fi
fi
FAILED=0
for patch_info in "${patches_preview[@]}"; do
IFS='|' read -r claim_type claim_ns claim_name rev <<<"$patch_info"
if ! patch_claim "$claim_type" "$claim_ns" "$claim_name" "$rev"; then
warn "Failed to patch claim $claim_type/$claim_name in namespace $claim_ns"; FAILED=1
fi
done
[ $FAILED -eq 0 ] || exit 1
echo "Done."
What the script does
rollback.sh has two modes:
Rollback mode (default): Patches claims directly to set:
spec:
compositionRevisionSelector:
matchLabels:
metadata.appcat.vshn.io/revision: <REV>
If -r <REV> is not given, it autodetects the previous CompositionRevision relative to each claim’s Composite Resource current revision and uses that CompositionRevision’s label metadata.appcat.vshn.io/revision. Autodetect automatically skips revisions with missing function packages.
Unpin mode (--unpin): Removes the compositionRevisionSelector from claims:
spec:
compositionRevisionSelector: null
This returns claims to automatic policy, where they’ll use the latest active CompositionRevision.
Usage examples
|
All commands use claim types (for example |
Preview what would be rolled back (dry-run)
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io --dry-run
Rollback all claims of one type (autodetect revision per claim)
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io
Rollback all claims of one type to specific revision
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io -r v3.59.0-v4.172.0
Rollback all claims of one type in a specific namespace (autodetect)
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io -n my-namespace
Rollback single claim (autodetect)
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io -i my-postgres -n my-namespace
Rollback single claim to specific revision
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io -i my-postgres -n my-namespace -r v3.59.0-v4.172.0
Rollback multiple types to specific revision
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io,vshnnextclouds.vshn.appcat.vshn.io -r v3.59.0-v4.172.0
Rollback multiple types in a specific namespace to specific revision
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io,vshnnextclouds.vshn.appcat.vshn.io -n my-namespace -r v3.59.0-v4.172.0
Return to automatic policy (unpin)
# Unpin all claims of a type
./rollback.sh --unpin -t vshnpostgresqls.vshn.appcat.vshn.io
# Unpin single claim
./rollback.sh --unpin -t vshnpostgresqls.vshn.appcat.vshn.io -i my-postgres -n my-namespace
# Unpin multiple types
./rollback.sh --unpin -t vshnpostgresqls.vshn.appcat.vshn.io,vshnnextclouds.vshn.appcat.vshn.io
Verify after patch
Check what the claim is pinned to and watch for reconciliation:
TYPE=vshnpostgresqls.vshn.appcat.vshn.io
NAME=my-postgres
NAMESPACE=my-namespace
echo "Selector (REV label) in claim:"
kubectl get "$TYPE" "$NAME" -n "$NAMESPACE" -o jsonpath='{.spec.compositionRevisionSelector.matchLabels}'
echo ""
echo "Watch claim for reconciliation:"
kubectl get "$TYPE" "$NAME" -n "$NAMESPACE" -w
Discover available revisions
Use the --list-revisions flag to see all available CompositionRevisions for a type:
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io --list-revisions
This will display a table with:
-
CompositionRevision name - Full CompositionRevision identifier with hash
-
Revision label - Label to use with
-r -
Created - Timestamp when the revision was created
-
Function status - Whether the composition function package exists in the cluster (EXISTS/MISSING/N/A)
|
Don’t roll back to revisions with |
You can also list revisions for multiple types at once:
./rollback.sh -t vshnpostgresqls.vshn.appcat.vshn.io,vshnnextclouds.vshn.appcat.vshn.io --list-revisions
Operational patterns
-
Prefer autodetect: Let the script find the previous revision relative to the current one. Works even after multiple prior rollbacks.
-
Start small: Roll back a single instance or all instances of a single type first to confirm the fix.
-
Bulk rollback: Use only for truly widespread incidents affecting many types at once.
-
Pin explicitly: When you already know the exact good revision label to apply across instances.
-
Unpin after resolution: Once the incident is fixed, run
rollback.sh --unpinso healthy revisions roll out automatically again.