Production grade clusters can be either setup from scratch (turnkey solution) or they can be managed by a cloud provider (hosted solution).
Turnkey Solution
Hosted Solution
To avoid single point of failure, production grade clusters should have at least 3 master nodes. Consider a cluster with 2 master nodes.
The API Server is just a REST API server, it can be kept running on both the masters in an active-active mode. To distribute the incoming requests to both the KubeAPI servers, use a load balancer like nginx in front and point the kubectl utility to the load balancer (by configuring KubeConfig).
The Controller Manager and the Scheduler look after the state of the cluster and make necessary changes. Therefore to avoid duplicate processing, they run in an active-passive mode.
The instance that will be the active one is decided based on a leader-election approach. The two instances compete to lease the endpoint. The instance that leases it first gets to be the master for the lease duration. The active instance needs to renew the lease before the deadline is reached. Also, the passive instance retries to get a lease every leader-elect-retry-period. This way if the current active instance crashes, the standby instance can become active.
The kube-controller-manager and the kube-scheduler have leader election turned on by default.
kube-controller-manager --leader-elect true [other options]
--leader-elect-lease-duration 15s
--leader-elect-renew-deadline 10s
--leader-elect-retry-period 2s
kube-scheduler --leader-elect true [other options]
--leader-elect-lease-duration 15s
--leader-elect-renew-deadline 10s
--leader-elect-retry-period 2s
The ETCD Server can be present in the master node (stacked topology) or externally on other servers (external ETCD topology). Stacked topology is easier to setup and manage but if both the master nodes are down, then all the control plane components along with the state of the cluster (ETCD servers) will be lost and thus redundancy will be compromised.