[bitnami/etcd] Stop relying on files for state

[pckhoi]

Description of the change

The current etcd container and chart have a few major problems:

1. It relies on files outside of the data directory which could contain conflicting information compared to the data dir and the cluster
2. Removing the member during pre-stop hook is problematic. I guess this was added to support scaling down the cluster. If so, this logic is leaky. There are 2 cases where this breaks down:
   1. If the pod was killed for reasons other than replicas update then the next time the pod starts, it will not be able to start from the existing data dir which means it must throw away the data dir and start from scratch.
   2. If the cluster is scaled down and the PVC is retained, the next time the cluster is scaled up, the new member will encounter a non-empty data dir which it must discard
3. If the member was removed, the container chokes up on non-empty data dir in most cases except when recovering from a snapshot
4. It might attempt to add a new member even when an old member with the same name already exists. This is caused by relying on files for state
5. It runs etcdctl member update for unclear reasons when the data dir is not empty and there is a member ID
6. It relies on ETCD_INITIAL_CLUSTER_STATE to know whether the cluster is new which could be inaccurate

This PR add the following changes:

• Add preupgrade.sh which should be run in a Helm pre-upgrade hook. When the cluster is scaled down, it detects and removes obsolete members with etcdctl member remove.
• Remove prestop.sh
• Stop storing/checking member ID from the member_id file. Instead, the remote member ID is read from the cluster with etcdctl member list, and the local member ID is checked for conflict during startup.
• Stop storing/checking member removal state from member_removal.log. Check with etcdctl member list instead.
• If the data dir is not empty, check if the member still belongs to the cluster (remote ID and local ID are the same). If there is a conflict, remove the data dir, remove the old member, add a new member, and start the member from scratch.
• Remove environment variable ETCD_DISABLE_STORE_MEMBER_ID
• Remove environment variable ETCD_DISABLE_PRESTOP
• Environment variable ETCD_INITIAL_CLUSTER_STATE becomes read-only

Benefits

• Not relying on files outside of the data directory means there is only a single source of truths (or only as many as there are live members in the cluster plus the data dir), which makes most operations more reliable
• Removing obsolete members in Helm pre-upgrade hook means the etcdctl member remove command tends to be executed against a healthy cluster
• If the pod was killed for reasons other than replica changes, it can rejoin the cluster on its own while keeping all its data intact
• The container no longer chokes up on a non-empty data dir, even when the old member is removed

Possible drawbacks

• If during initialization there is a network outage and the current member can't connect to other members, it will think that it must start a new cluster. That said, I don't think there is any good solution in this case except manual recovery.
• I have not tested this set of changes outside of Helm/K8s

Applicable issues

• fixes #16069

Additional information

Related changes in the Helm chart: bitnami/charts#31161 and bitnami/charts#31164

[pckhoi]

I'm planning to open a complementary PR in the charts repo. I will try to add more tests there.

[juan131]

Hi @pckhoi

Thanks so much for this amazing contribution! It'd definitely help on making the Bitnami etcd chart more stable.

I think the main concern/challenge with your changes would be providing a solution for users who may scale down the cluster via kubectl scale sts/etcd --replicas X (or via some HorizontalPodAutoscaler that may also scale down the cluster without Helm's control via hooks). Correct me if I'm wrong but this use case won't be covered, right?

[pckhoi]

@juan131 you're correct that the autoscaling use case isn't covered. People use Etcd for its consistency rather than for handling large, fluctuating traffic so I think autoscaling to handle large traffic is a niche use case.

As for manual scaling, running helm upgrade makes more sense if the cluster is installed via Helm. If people are scaling with kubectl scale then they're probably not using Helm which makes it difficult to operate. So no, deploying/upgrading without Helm is also not supported.

[juan131]

Thanks for confirming so @pckhoi ! In that case, I'd add a warning at the "Upgrading" section alerting about what these changes imply (I mean, warning users to use exclusively Helm to scale the cluster):

• https://github.com/bitnami/charts/tree/main/bitnami/etcd#upgrading

We could even add it in the chart NOTES:

• https://github.com/bitnami/charts/blob/main/bitnami/etcd/templates/NOTES.txt

[pckhoi]

Sure, I will do that.

[pckhoi]

@juan131 I have updated https://github.com/bitnami/charts/tree/main/bitnami/etcd#upgrading. As for https://github.com/bitnami/charts/blob/main/bitnami/etcd/templates/NOTES.txt, I don't see anything that needs to be updated.

[juan131]

Suggested change

	# Return a comma separated list of <host>:<port> for each endpoint
	# Globals:
	# ETCD_*
	# Return a comma separated list of <host>:<port> for each endpoint
	# based on "initial-cluster" flag value
	# ref: https://etcd.io/docs/latest/op-guide/clustering/#static
	# Globals:
	# ETCD_INITIAL_CLUSTER

[juan131]

Given this function isn't reused anywhere and it contains the actual logic of the script, I think it's simpler to directly use the code, check my suggestion:

Suggested change

	########################
	# Remove members that are not named in ETCD_INITIAL_CLUSTER
	# Globals:
	# ETCD_*
	# Arguments:
	# None
	# Returns:
	# None
	#########################
	remove_members() {
	local -a extra_flags current expected
	read -r -a extra_flags <<<"$(etcdctl_auth_flags)"
	is_boolean_yes "$ETCD_ON_K8S" && extra_flags+=("--endpoints=$(endpoints_as_host_port)")
	debug "Listing members"
	current="$(etcdctl member list ${extra_flags[@]} --write-out simple | awk -F ", " '{print $1 "," $3}')"
	if [ $? -ne 0 ]; then
	debug "Error listing members, is this a new cluster?"
	return 0
	fi
	info "Current cluster members are: $(echo "${current[@]}" | awk -F, '{print $2}' | tr -s '\n' ',' | sed 's/,$//g')"
	expected="$(echo $ETCD_INITIAL_CLUSTER | sed 's/,/\n/g' | awk -F= '{print $1}')"
	info "Expected cluster members are: $(IFS= echo "${expected[@]}" | tr -s '\n' ', ' | sed 's/,$//g')"
	for member in $(comm -23 <(echo "${current[@]}" | awk -F, '{print $2}' | sort) <(echo "${expected[@]}" | sort)); do
	info "Removing obsolete member $member"
	etcdctl member remove ${extra_flags[@]} $(echo "${current[@]}" | grep "$member" | awk -F, '{print $1}')
	done
	}
	remove_members
	# Remove members that are not listed in ETCD_INITIAL_CLUSTER
	# from the cluster before running Helm upgrades that potentially scale
	# down the etcd cluster
	read -r -a extra_flags <<<"$(etcdctl_auth_flags)"
	is_boolean_yes "$ETCD_ON_K8S" && extra_flags+=("--endpoints=$(endpoints_as_host_port)")
	debug "Listing members"
	if ! current="$(etcdctl member list ${extra_flags[@]} --write-out simple | awk -F ", " '{print $3 ":" $1}')"; then
	debug "Error listing members, is this a new cluster?"
	return 0
	fi
	info "Current cluster members are: $(echo "$current" | awk -F: '{print $1}' | tr -s '\n' ',' | sed 's/,$//g')"
	expected="$(echo $ETCD_INITIAL_CLUSTER | tr -s ',' '\n' | awk -F= '{print $1}')"
	info "Expected cluster members are: $(echo "$expected" | tr -s '\n' ',' | sed 's/,$//g')"
	for member in $(comm -23 <(echo "$current" | awk -F: '{print $1}' | sort) <(echo "$expected" | sort)); do
	info "Removing obsolete member $member"
	etcdctl member remove ${extra_flags[@]} "$(echo "$current" | grep "$member" | awk -F: '{print $2}')"
	done

[juan131]

Shorter:

Suggested change

	if ! etcdctl endpoint status --cluster ${extra_flags[@]}; then
	return 0
	else
	return 1
	fi
	! debug_execute etcdctl endpoint status --cluster ${extra_flags[@]}

[juan131]

Suggested change

	* Drop support for non-Helm deployment. Upgrading of any kind including increasing replica count must also be done with `helm upgrade` exclusively. CD automation tools that respect Helm hooks such as ArgoCD can also be used.
	* Drop support for non-Helm cluster deployment. Upgrading of any kind including increasing replica count must also be done with `helm upgrade` exclusively. CD automation tools that respect Helm hooks such as ArgoCD can also be used.

[juan131]

Is this still used?

[pckhoi]

The user does not use it. Is it fine to leave it in the read-only section then?

[pckhoi]

Thanks! I have addressed all the suggestsions.