Crossplane is an infrastructure provisioning tool engineered to bridge the gap between infrastructure automation, Kubernetes and reconciliation on steroids. Its main ethos is to provide a unified declarative API to abstract different cloud implementations and leverage Kubernetes to run the provisioning workload.
After one year of intensive professional use at CECG, I will discuss our personal experience with it, highlighting the good, bad and ugly aspects of the tool.
TL;DR: Crossplane doesn’t kill like smoking, but you’ll frequently ask yourself the question: “Why am I doing this to myself?” and try quitting, unsuccessfully.
The Good
Automatic reconciliation
A new tool is shiny, bright and exciting. With an optimistic outlook, I installed Crossplane for the first time within my local kind cluster and deployed my first resource, a GCP storage bucket. When changing the bucket settings via the GCP console , I observed that Crossplane would automatically roll back the changes to the desired state. Excellent.
Declarative Infrastructure as Code
In Crossplane, the infrastructure resources - as well as the Crossplane configuration - can be represented as YAML files. The resources are applied via Provider pods, cognates of the well-known Terraform providers. Remember the terraform’s endless plan and apply? That’s an old story: Crossplane only requires a kubectl apply -f . and job done, the power of Kubernetes will automatically reconcile your infrastructure and apply the changes. This also means that you can deploy your infrastructure alongside your application manifests and keep both closely coupled.
Leveraging the power of Kubernetes
Another advantage of the Kubernetes world is the ability to take advantage of its native ecosystem. For example, the Crossplane manifests can be manipulated and deployed by Kube native configuration management tools, such as Kustomize or Helm . Prometheus can be used to monitor the provisioning cycles and ArgoCD can conveniently deliver the manifests to the control planes. Kubernetes’ RBAC and the namespace model can be also leveraged to granularly scope provisioning permissions to teams.
XRDs as-you-go
Cloud resources can be grouped into “compositions” and abstracted away using a single custom resource, which Crossplane calls “composite” or XR. Changes to the XR will trigger a composition run. Does it remind you of something familiar? Yes, exactly: Kubernetes custom controllers. It could be argued that Crossplane is essentially a generator of Kubernetes operators on the fly, so you don’t have to write a single line of code.
Everything function
Rather than using the built-in patch and transform to render YAML, it is possible to generate infrastructure resources directly in a programming language of choice (SDKs are currently available for Go and Python ). This overcomes Crossplane’s immaturity and provides custom functionality, augmenting the tool’s flexibility to function for use cases beyond infrastructure provisioning: a proper all-purpose Kubernetes operator generator.
The bad
Lack of safeguards between plan and apply
Generations of DevOps engineers have been duly pampered by Terraform’s plan and apply. There’s no better feeling than being confident not to destroy your employer’s systems when applying a new change. This assurance won’t be provided by Crossplane: any change will be planned and applied behind the scenes, which could lead to serious undesired consequences. Nevertheless, Crossplane will never actively destroy your infrastructure unless specifically instructed, for example by manually deleting the associated Kubernetes resource.
Control-plane hiccups
Installing Crossplane providers may result in API server unavailability, as hundreds of CRDs may be deployed with a given provider. For example, the provider-gcp , which allows the provisioning of GCP resources, contains 347 CRDs. In some scenarios, the additional control-plane burden triggers scale-ups that hang up the Kube API server for many minutes, or sometimes even hours. The recently introduced provider families try to address this issue.
Disastrous upgrades and restores
Everything is fine until it isn’t. And some legitimate operations may lead to unintended consequences. For example, we inadvertently deleted cloud resources when upgrading XRDs, providers or compositions because of resource ownership. Furthermore, most Kubernetes backup tools restore resources without preserving the status fields, which under specific conditions may result in dependencies being flagged for deletion. Morale of the story: be particularly careful, especially if running in production. Uncomfortable workarounds could involve setting Crossplane to “orphan” the resources after creation or scaling down the providers’ pods to zero before performing dangerous operations.
The Ugly
YAML logorrhoea
The success of a language lies in its ability to concisely and simply transmit a message. Far from being concise and simple, Crossplane’s YAML resources may span thousands of lines and be difficult to navigate and identify information. The beauty of the declarative language succumbs to the lack of native features, which often results in code shenanigans and workarounds. For example, we wrote tens of YAML lines just to transform an array of strings. If you suffer from insomnia, we recommend that you read Crossplane compositions before bed, on a warm background, for a more comfortable rest.
Evolving testing maturity
Please do not read this section if you are a testing freak. Crossplane does not provide an out-of-the-box testing framework, only resource rendering features. However, as for everything else, leveraging the Kubernetes ecosystem comes in handy. We have successfully set up the kubernetes-sigs’ e2e-framework , powered by kind , to run composition tests and validate the resources. Please remember to clean up afterwards: sometimes the tool breaks and resources are left hanging, depleting your cloud credits.
Patch and Transform
Patch and Transform is an ex Crossplane-native feature, now a composite function, which lets you emulate the terraform patching behaviour and move data between resources. Sounds too good to be true? It probably is. You can only patch data in a composition between resources and the XRs, which is ugly, ugly and - sorry if I didn’t mention it before - ugly. The lack of native for loops or if conditionals are atrocious so we decided we would stop using patch and transform altogether and just write custom go functions.
Final Thoughts
Some just see black, others see white and others see many colours. Engineers only see the shades between these colours. Even though we remain critical about Crossplane’s usability and we’re cautious about production-wide adoption at the current maturity level, at CECG we believe that the Upbound tool has potential and would like to endorse it, putting forward five reasons:
- The onus to provision infrastructure can be shifted to the development teams, embedding claims directly into application deployment manifests. The ability to provide declarative APIs to abstract this process is pivotal to this purpose.
- Permissions to provision infrastructure can be segregated into Kubernetes’ team namespaces, rather than CD pipelines.
- The Kubernetes reconciliation loops help reduce infrastructure drift.
- The Kubernetes ecosystem and relative tooling can be leveraged to plug the gaps in Crossplane’s immaturity.
- The use of general-purpose programming languages to address more complex use cases is welcome, especially as it allows more granular testing.