Teapot DNS

Project Teapot must provide a trusted way for DNS information generated by the (untrusted) tenant clusters to be propagated out to the network.

Each tenant cluster requires at least 2 DNS records – one for the control plane, and a wildcard for any applications. These would usually be subdomains of a zone delegated to the Teapot for this purpose. Teapot would be responsible for rolling up these records and making them available over DNS.

Since Teapot will be responsible for allocating public IP addresses, it will also need to be responsible for advertising reverse DNS records for those IPs.

Implementation Options

The Kubernetes SIG ExternalDNS project is a Kubernetes-native service that collects IP addresses for Services and Ingresses running in the cluster and exports DNS records for them (though it is not itself a DNS server). It supports many different back-ends – both traditional DNS servers and cloud-based services (including OpenStack Designate).

While tenants are of course free to run this in their own clusters already (perhaps pointing to an external cloud service), this is not sufficient to satisfy the above requirements. It requires them to use an external cloud service (which may not always be appropriate for internal-only applications in a private cloud), since tenants are untrusted and cannot be given write access to an internal DNS server. And reverse DNS records cannot be exported, because tenant clusters are not a trusted source of information about what IP addresses are assigned to them.

ExternalDNS in load balancing cluster

If Teapot implemented the load balancing option based on Ingress resources in the management cluster (or a separate load balancing cluster), and these were used for both Services and Ingresses, then ExternalDNS running in that same cluster would automatically see all of the external endpoints for the tenant clusters. It could even rely on the fact that the IP addresses will have been sanitised already before creating the Ingress objects. There would need to be provision made somewhere for sanitising the DNS names, however.

On its own this only satisfies the first requirement. Additional work might need to be done to export the wildcard DNS records for the tenant workloads. (Note that the tenant control planes would be running in containers on the management cluster or another centrally-managed cluster, and may well have Ingress resources associated with them already.) And additional work would certainly be needed to export the reverse DNS records.

A major downside of this is that it gives the tenant very little control over whether and how it exports DNS information.

Build a component to sync ExternalDNS

A component running in a tenant cluster could sync any ExternalDNS Endpoint resources (not to be confused with Kubernetes Endpoint resources) from the tenant cluster into the management cluster. (This component could even be written as an ExternalDNS provider.) This option is analagous to an Ingress-based API for load balancing.

On the management cluster side, a validating webhook would check for legitimacy prior to accepting a resource. More investigation is required into the mechanics of this – since the resources are not normally manipulated by anything other than ExternalDNS itself, having something else writing to them might prove brittle.

Again, additional work might need to be done to export the wildcard DNS records for the tenant workloads and would be needed for reverse DNS records.

OpenStack Designate

Designate is already one of the supported back-ends for ExternalDNS. By running a minimal, opinionated installation of Designate in the management cluster we could allow tenants to choose whether and how to set up ExternalDNS in their own clusters. They could choose to export records to the Teapot cloud, to some external cloud, or not at all.

Since Designate has an API, it would be easy to add the two top-level records for each cluster.

Designate has the ability to export reverse DNS records based on floating IPs. However, the current implementation is tightly coupled to Neutron. If Neutron is used in Teapot it should be as an implementation detail only, so other services like Designate should not rely on integrating with it. Therefore additional work would be required to support reverse DNS. There is an API plugin point to pull data, or it could be pushed in through the Designate API.

Ideally the back-end in the opinionated configuration would be CoreDNS, due to its status in the Kubernetes community (it is used for the internal DNS and is a CNCF project). However, there is currently no CoreDNS back-end for Designate. An alternative to writing one would be to write a Designate plugin for CoreDNS – similar plugins exist for other clouds already. The latter would provide the most benefit to OpenStack users, since theoretically tenants could make use of it even if CoreDNS is not chosen as the back-end by their OpenStack cloud’s administrators.

The Designate Sink component would not be required, but the rest of Designate is also built around RabbitMQ, which is highly undesirable. However, it is largely used to implement RPC patterns (call, not cast), and might be amenable to being swapped for a json-rpc interface in the same way as is done in Ironic for Metal³.