Introduction

In the digital landscape, where security breaches and data leaks are ever-looming threats, the art of managing secrets within Kubernetes takes on a paramount importance. “Securing the Unseen” delves into the comprehensive strategies essential for the effective management of Kubernetes secrets, offering a beacon of guidance for security and software professionals alike. This guide is meticulously crafted to uncover the layers of complexity in secret management, presenting a panoramic view of the tools and practices at the forefront of Kubernetes security.

With an emphasis on comprehensive strategies, this guide transcends traditional secret management approaches, exploring advanced methodologies to shield the unseen vulnerabilities and fortify the defences of your Kubernetes environment. From exploring native Kubernetes capabilities to leveraging cutting-edge external tools, “Securing the Unseen” serves as an indispensable resource for those seeking to master the nuances of secret management and elevate their security posture.

Embark on this journey to master the comprehensive strategies that underscore Kubernetes secret management. Through a blend of theoretical insights and practical advice, this guide promises to equip you with the knowledge and skills to navigate the challenges of securing the unseen, ensuring the confidentiality, integrity, and availability of your secrets in the complex Kubernetes ecosystem.

Secrets in Digital Environments

Secrets, in the context of information technology, are confidential pieces of data that are crucial for ensuring the security and integrity of systems and applications. These typically include passwords, encryption keys, API tokens, and other sensitive credentials that grant access to various digital resources.

Secrets are vital because they protect access to important data and systems, helping prevent unauthorised access and security breaches. Given their sensitive nature, it’s essential to manage and store them securely to mitigate the risk of exposure and potential cyber threats.

Securing Sensitive Data with Kubernetes Secrets

Kubernetes Secrets are objects within the Kubernetes ecosystem designed to store and handle sensitive data, such as passwords, OAuth tokens, and SSH keys, in a secure and controlled manner.

They are essentially small units of sensitive data which might be put in a Pod specification. By using Secrets, Kubernetes offers a more secure alternative to hard-coding this confidential information within application code or scripts.

Secrets are stored in Kubernetes' etcd, a distributed key-value store, and are accessible to Pods needing them, based on defined permissions and policies.

Kubernetes' native approach for managing secrets

Effectively managing secrets generally presents numerous challenges, with security being the primary concern. For example, storing secrets in version control systems poses a significant risk of unauthorised access. This reality necessitates a method to securely retrieve these secrets from an external source at runtime, ensuring they are kept safe from exposure. This is where Kubernetes steps in.

Kubernetes effectively prevents the risks associated with exposing sensitive data in plain text by offering a native approach to integrate and consume secrets in applications. This integration can be performed in two common ways: using a declarative approach by creating definition files, or imperatively, with kubectl commands.

These secrets can then be utilised in Kubernetes applications through either

• Data volumes or
• Environment variables.

Both methods have their use cases and implications, which can be further elucidated through practical examples and diagrams to provide a clearer understanding.

Limitations of Kubernetes' native Secret Management

Managing secrets with Kubernetes' native tools or through Infrastructure as Code (IaC) frameworks like Helm, Terraform, or Pulumi is a common practice, yet it comes with significant limitations.

Base64 encoding is often used to handle secrets within Kubernetes, but it merely obscures the text rather than securing it. Anyone with access to these encoded secrets, whether through the git repository or directly within the Kubernetes cluster, can easily decode and extract the sensitive information. This is far from a secure practice, especially for production environments.

The Kubernetes documentation clearly states that, by default, Kubernetes Secrets are not encrypted in the API server’s data store. etcd acts as the source of truth for all Kubernetes objects, including secrets. This lack of encryption means that anyone with API access, those who can directly access etcd, or any backups—such as volumes, S3 buckets, or even those managed by service providers—might retrieve or tamper with the secrets. The risk encompasses all types of secrets, from TLS certificates to registry credentials.

Additionally, Kubernetes' default Role-Based Access Control (RBAC) settings might not be sufficient for securing secrets. Any user with permissions to create a Pod within a namespace may have the ability to access all secrets in that namespace, potentially exposing them through deployments or other indirect methods.

There are several other, less obvious reasons why native Kubernetes secret management or IaC might not be sufficient:

Safely Using Secrets in Kubernetes

To address these challenges, the Kubernetes documentation outlines several steps for the safe usage of secrets. These include :

• Enabling encryption at rest for secrets,
• Configuring RBAC rules to ensure least-privilege access,
• Restricting secret access to specific containers, and
• Considering the use of external secret store providers.

These measures enhance the security and manageability of secrets in Kubernetes environments. However, given the inherent limitations of Kubernetes' native secret management as described previously, there’s a clear benefit to exploring a secret management solution that can offer advanced security features and management capabilities.

Choosing a Secret Management Solution

The journey to implementing a secret management solution should be carefully planned to address the intrinsic limitations of Kubernetes' native secret management. Selecting a secret management solution should be an informed decision, based on a set of well-defined standards that align with your organisation’s security posture and operational needs. The ideal solution should meet the following standards:

The ideal secret management solution should excel across the above mentioned key standards, as illustrated in the table below. In this table, a rating of ‘High’ indicates the highest level of alignment or compatibility with the given standard

In the subsequent sections, we will explore three approaches to managing secrets. The first option involves storing the secrets in an encrypted form. Next, we will examine various secret store providers to determine their alignment with our defined criteria. Following that, we will delve into comprehensive solutions that can enhance the capabilities of standalone secret storage providers. This approach ensures a holistic strategy for secret management in Kubernetes environments.

Storing Encrypted Secrets

A secure method for managing secrets involves encrypting them before storing them in a Git repository. In this process, secrets are encrypted using encryption keys, making them unreadable to humans. When deploying to a Kubernetes cluster, these encrypted secrets need to be decrypted.

Bitnami Sealed Secrets and Mozilla SOPS are two solutions that facilitate this process. Bitnami Sealed Secrets approach involves an operator within the Kubernetes cluster that possesses an encryption key to decrypt all secrets pushed into Git.

On the other hand, Mozilla SOPS works by encrypting each file with a set of keys and ensures that only authorised users or systems can decrypt these files. This method helps maintain the confidentiality and integrity of sensitive data throughout the deployment process.

Mozilla SOPS (Secrets OPerationS)

Mozilla SOPS (Secrets OPerationS) is an open-source project, specifically designed as a platform-agnostic CLI tool. Its primary function is to encrypt secrets within files, particularly focusing on formats like YAML and JSON.

It integrates with key management services including Azure Key Vault, AWS KMS, GCP KMS, and HashiCorp Vault, allowing the secure storage of encrypted Kubernetes secrets in source control.

These secrets are decrypted just before deployment, maintaining security in CI/CD pipelines. SOPS is user-friendly, supporting multiple authentication methods and seamlessly integrating with cloud services, making it an effective solution for managing sensitive data in Kubernetes and aligning with GitOps practices.

Components of Mozilla SOPS:

CLI Tool: The core component of SOPS is its command-line interface (CLI) tool, which is used for encrypting and decrypting files.

Encryption Backends: SOPS integrates with multiple key management backends, including:

• PGP: For using public and private key pairs.
• KMS Services: Such as Azure Key Vault, AWS KMS, GCP KMS, and HashiCorp Vault, which provide cloud-based key management solutions.

Authentication Methods: For cloud KMS services, SOPS supports various authentication methods including client credentials, client certificates, username & password, Managed Service Identity (MSI), and Azure CLI authentication.

How It Works:

Encryption Process:

1. A Kubernetes secret is created using kubectl and stored in a YAML file.
2. The SOPS CLI tool is used to encrypt this file. During encryption, SOPS utilizes the public key from the specified backend (e.g., Azure Key Vault) to encrypt the secret.
3. The encrypted file, now a SOPS file, contains the encrypted version of the secret along with metadata about the encryption (like which key was used).

Decryption Process:

1. When deploying the secret to Kubernetes, the SOPS file is decrypted using the corresponding private key.
2. The decryption can be performed locally or in a Kubernetes environment, where the private key is securely managed.
3. After decryption, the original Kubernetes secret is restored and can be applied to the Kubernetes cluster using kubectl.

Integration with GitOps:

Encrypted files can be safely committed to a Git repository, fitting into GitOps workflows.

Continuous Deployment tools like Flux can work with these encrypted files, decrypting them as part of the deployment process.

End to End Encryption

It encrypts secrets within files like JSON and YAML, using various key management services like AWS KMS, GCP KMS, Azure Key Vault, and HashiCorp Vault.However, after decryption in a Kubernetes cluster, these secrets are stored unencrypted in etcd, Kubernetes' key-value store.

Access Control

Mozilla SOPS ensures secure encryption of secrets for Git storage, but once decrypted in Kubernetes, access control relies on properly configured Kubernetes RBAC to secure these secrets.

Secret Rotation

SOPS does not inherently handle secret rotation. For rotating secrets, you would typically update the secret in your repository, and then during deployment would apply these changes in the cluster.

Audit Compliance

SOPS doesn’t provide specific audit logging features. Auditing compliance would need to be addressed through Kubernetes audit logs or other external tools.

Cloud Agnostic

SOPS is cloud-agnostic, supporting multiple cloud providers’ key management services. This makes it flexible for use in various cloud environments.

Centralised Storage

SOPS does not provide centralised storage for secrets; it encrypts secrets within files that are then stored in your version control system or other locations.

Tenant Interface

SOPS is primarily a CLI tool and does not provide a separate tenant interface for managing secrets.

API / CLI

SOPS is a CLI tool. It does not have a native API, but it can be integrated into scripts and workflows that can be executed programmatically.

Developer Friendly

SOPS is very developer-friendly, given its simplicity and integration with common file formats and version control systems.

Platform Friendly

SOPS integrates smoothly with multiple key management services (KMS) like AWS KMS, GCP KMS, Azure Key Vault, and HashiCorp Vault. This flexibility is beneficial for platform engineers who work across various cloud environments.

Bitnami Sealed Secrets

Bitnami Sealed Secrets, an open-source project, enhances the security of Kubernetes Secrets by encrypting them into “Sealed Secrets.” These can be safely stored in public repositories or version control systems.

Upon deployment into a Kubernetes cluster, the Sealed Secrets controller decrypts these sealed secrets, converting them into standard Kubernetes Secrets. This decryption process utilises a pair of cryptographic keys: a public key for encryption, and a private key, securely kept within the Kubernetes cluster, for decryption.

The Sealed Secrets system comprises three main components:

Operator: Deployed on the Kubernetes cluster, the operator is responsible for decrypting secrets.The private key used by the SealedSecrets operator for decrypting secrets is stored as a Kubernetes Secret within the cluster.

CLI Tool (KubeSeal): This tool encrypts your secret before you push it to your Git repository. The operator, upon detecting the encrypted secret, performs decryption and passes it to the containers.

Custom Resource Definition (CRD): Instead of creating regular Kubernetes Secrets, SealedSecrets are defined within Kubernetes, allowing the operator to recognize and decrypt them.

The deployment process of a Sealed Secret involves the operator generating a public-private key pair. This pair is used for encrypting and decrypting secrets.

Encrypting a Secret with Kubeseal

1. Utilise the KubeSeal CLI to pass a Kubernetes secret.
2. KubeSeal employs the public key to encrypt the secret, creating a SealedSecret.
3. The SealedSecret can then be pushed to your Git repository and applied via GitOps or manually with the kubectl command.

Once deployed, the operator detects the new Sealed Secret creation and uses the private key to decrypt it, converting it into a regular Kubernetes Secret.

While these methods enhance security on the Git side, it does raise concerns. For instance, if this operator malfunctions, all your secrets could be compromised. Similarly, losing the encryption key would result in the loss of access to these secrets. Additionally, this approach necessitates considerations for secret rotation. Overall, though, this method offers improved security compared to Kubernetes' default secret management.

End to End Encryption

Bitnami Sealed Secrets excels in encryption. It encrypts Kubernetes Secrets into Sealed Secrets, which can be safely stored in version control systems.However, after decryption in a Kubernetes cluster, these secrets are stored unencrypted in etcd, Kubernetes' key-value store.

Access Control

While Bitnami Sealed Secrets manages the encryption and decryption of secrets, access control within the Kubernetes cluster is governed by Kubernetes' own mechanisms like RBAC (Role-Based Access Control).The decryption of sealed secrets is intricately engineered to be an exclusive capability of the operator, more precisely, the SealedSecrets controller. This design principle underscores a stringent security model, where access to perform decryption operations is tightly controlled and restricted to this designated controller.

Secret Rotation

Secret rotation is not inherently managed by Bitnami Sealed Secrets. For rotating secrets, you would typically update the Sealed Secret in your repository, and the controller would apply these changes in the cluster.

Audit Compliance

Bitnami Sealed Secrets doesn’t provide specific audit logging features. Auditing compliance would need to be addressed through Kubernetes audit logs or other external tools.

Cloud Agnostic

Sealed Secrets is cloud-agnostic. It works with any Kubernetes cluster, regardless of the cloud provider or on-premise setup.

Centralised Storage

The tool doesn’t offer centralised storage for secrets itself; it’s more about encrypting secrets that are stored in a centralised version control system.

Tenant Interface

Bitnami Sealed Secrets is primarily a Kubernetes controller and CLI tool; it does not provide a separate tenant interface.

API / CLI

It primarily offers a CLI tool for creating Sealed Secrets. The interaction with Sealed Secrets in Kubernetes is through standard Kubernetes APIs.

Developer Friendly

It is developer-friendly in terms of its ease of use and integration with standard development workflows, especially those involving Kubernetes and CI/CD pipelines.

Platform Friendly

Given its compatibility with any Kubernetes setup and its role in standard Kubernetes workflows, Bitnami Sealed Secrets can be considered platform-friendly, especially for Kubernetes-focused environments.

External Secret Store Providers

After examining encryption methods such as Mozilla SOPS and Bitnami Sealed Secrets, our attention now shifts to external secret store providers. These specialised services complement and enhance the security mechanisms provided by in-cluster tools, offering a more sophisticated approach to managing secrets that aligns with diverse operational requirements.

The landscape of external secret store providers can be broadly divided into two main categories:

1. Cloud-Native Providers: Tailored for specific cloud platforms, these providers offer seamless integration with the respective cloud services, optimising the management of secrets within those ecosystems.
2. Cloud-Agnostic Providers: Designed to function across any cloud or on-premises environment, cloud-agnostic providers ensure flexibility and adaptability. Within this category, we further distinguish between:

• Licensed Options: Offering robust features, dedicated support, and reliability, these services cater to organisations seeking comprehensive solutions with guaranteed service levels.
• Open Source Options: These providers leverage community-driven development, offering cost advantages and customization opportunities, though they may require more self-management and expertise.

In the following sections, we’ll evaluate how these providers measure up against our essential standards for secret management, such as robust encryption, access control, and integration with Kubernetes. We will delve into the nuances of each category, emphasising how licensed and open-source options under the cloud-agnostic umbrella address the unique challenges of secret management.

1. Cloud Native Providers

As we explore cloud-native secret management solutions, we focus on services like AWS Secret Manager, Azure Key Vault, and Google Cloud Secret Manager. These platforms offer comprehensive secret management tools, seamlessly integrated with their respective cloud environments, to provide robust security and management features for secrets.

Google Secret Manager

Google Secret Manager (GSM) is Google Cloud Platform’s (GCP) solution for securely storing, rotating, and retrieving secrets, such as passwords, tokens, and encryption keys. It features a global API with Identity and Access Management (IAM) for authentication and granular access control, along with Audit Logs and integration with Cloud KMS for encryption.

The recommended way to access Secret Manager secrets from workloads running on Google Kubernetes Engine is to use a Secret Manager client library authenticated via Workload Identity.This method enables directly fetching secrets from GSM using its API. This approach is highly secure as secrets are stored only in the memory of the pod and not on the filesystem. It supports Workload Identity for refined access control and includes audit logging capabilities.

End to End Encryption

Secret Manager automatically encrypts secrets at rest without the need for additional services. This ensures secure storage of secrets within the Google Cloud environment.

Access Control

The service provides robust access control, allowing fine-grained policies to be set on individual secrets, controlling who can access or manage them.

Secret Rotation

Google Cloud Secret Manager supports secret versioning but does not automatically rotate secrets. The rotation process involves manually creating new versions of a secret. However, Secret Manager can trigger Cloud Functions or Pub/Sub events to notify when a secret should be rotated.

Audit Compliance

Integrates with Google Cloud’s Audit Logs, enabling tracking of secret access and modifications for compliance and security auditing.

Cloud Agnostic

As a Google Cloud service, Secret Manager is not cloud-agnostic and is optimised for the Google Cloud ecosystem.

Centralised Storage

Offers centralised storage for secrets, providing a unified platform for managing secret data across Google Cloud services.

Tenant Interface

Features a user-friendly interface in the Google Cloud Console for managing secrets.

API / CLI

Provides an API and CLI (gcloud tool), but integrating these into existing applications can require code changes, which may not be straightforward for all developers.

Developer Friendly

While it offers APIs and client libraries, requiring code changes for integration can be a challenge, making it less straightforward for some developers.

Platform Friendly

Platform engineers managing GSM and GKE integration deal with setting up IAM roles, managing secret rotation, and monitoring audit logs for compliance. They address challenges like vendor lock-in and cost optimization, ensuring efficient use of Google Cloud resources.

Azure Key Vault

Azure Key Vault is a robust service for securely managing secrets like passwords, tokens, and encryption keys on Microsoft Azure. It integrates with Azure Active Directory (AAD) for authentication and access control and provides encryption with Azure Key Management Service (KMS).

Integrating Azure Key Vault with AKS enhances secret management in Kubernetes applications. A common method is using the Azure Key Vault’s API directly. This method is secure as it allows secrets to be fetched dynamically and used in-memory without storing them on the pod’s filesystem, thus minimising the risk of exposure.

End to End Encryption

Azure Key Vault ensures strong encryption for stored secrets, which are decrypted only when required by applications in AKS.

Access Control

Integration with Azure AD provides fine-grained access control over secrets, based on the roles and requirements of AKS applications.

Secret Rotation

Azure Key Vault allows for the storage of secrets, keys, and certificates. While it supports secret versioning, automatic rotation is not natively provided. Typically, rotation needs to be implemented through additional Azure services or custom scripting.

Audit Compliance

Key Vault maintains audit logs for access and changes to secrets, which is vital for regulatory compliance in AKS.

Cloud Agnostic

Utilising Azure Key Vault with AKS can lead to vendor lock-in within the Azure ecosystem.

Centralised Storage

Offers centralised storage for secrets, providing a unified platform for managing secret data across Azure services.

Tenant Interface

Features a user-friendly interface in the Azure Cloud Console for managing secrets.

API / CLI

Provides an API and CLI (az tool), but integrating these into existing applications can require code changes, which may not be straightforward for all developers.

Developer Friendly

For developers, direct API integration involves handling secret retrieval and management within the application code, ensuring dynamic access to updated secrets and secure handling of these secrets.

Platform Friendly

Platform team is responsible for configuring and maintaining the Key Vault and AKS integration, including setting up appropriate Azure AD permissions, monitoring usage to manage costs, and ensuring compliance through audit log reviews.

AWS Secrets Manager

AWS Secrets Manager is Amazon Web Services' (AWS) solution for managing and protecting secrets used in your applications and services. It offers functionalities similar to those of Google Secret Manager and Azure Key Vault, such as secret storage and retrieval, and integrates with AWS Identity and Access Management (IAM) for access control.

When integrating AWS Secrets Manager with Amazon Elastic Kubernetes Service (EKS), one effective method is through direct API calls. This approach securely manages secrets by fetching them dynamically at runtime, ensuring they are not exposed or stored insecurely.

End to End Encryption

AWS Secrets Manager automatically encrypts secrets at rest using encryption keys that you can manage through AWS Key Management Service (KMS). This ensures strong protection for secrets both at rest and in transit.

Access Control

Integration with AWS IAM allows for granular permission settings. EKS can utilise IAM roles for service accounts to access AWS Secrets Manager, enabling specific pods to access only the secrets they require.

Secret Rotation

AWS Secrets Manager does offer automatic secret rotation, but it requires the integration of an additional AWS service, specifically AWS Lambda.

Audit Compliance

AWS Secrets Manager provides audit capabilities through integration with AWS CloudTrail, which tracks and logs every request made to Secrets Manager, aiding in compliance and security monitoring.

Cloud Agnostic

Utilising AWS Secrets Manager can lead to a degree of vendor lock-in within the AWS ecosystem, similar to GCP and Azure offerings.

Centralised Storage

Offers centralised storage for secrets, providing a unified platform for managing secret data across AWS services.

Tenant Interface

Features a user-friendly interface in the AWS Cloud Console for managing secrets.

API / CLI

Provides an API and CLI (aws tool), but integrating these into existing applications can require code changes, which may not be straightforward for all developers.

Developer Friendly

For developers, direct API integration involves handling secret retrieval and management within the application code, ensuring dynamic access to updated secrets and secure handling of these secrets.

Platform Friendly

Platform team is responsible for configuring and maintaining the AWS Secret Manager and EKS integration, including setting up appropriate IAM permissions, monitoring usage to manage costs, and ensuring compliance through audit log reviews.

Comparison

End to End Encryption

AWS Secrets Manager uniquely excels in customizable key management compared to Google Cloud Secret Manager and Azure Key Vault, thanks to its seamless integration with AWS Key Management Service (KMS). This allows for nuanced control over encryption keys, including detailed policies, scheduled rotations, and comprehensive auditing. For example, AWS Secrets Manager automates the rotation of secrets, such as database credentials, enhancing security with minimal manual intervention. While Google Cloud and Azure also support customer-managed keys and encryption practices, AWS stands out for its automation capabilities and native support for integrating secret rotations across its service ecosystem, offering a blend of flexibility and security that is especially valuable for organisations with complex compliance and operational requirements.

Access Control

Each platform’s approach reflects its ecosystem’s characteristics, with AWS focusing on granularity, Azure on integration within its services, and Google offering a balanced approach with strong Kubernetes support.

Secret Rotation

None of the secret management services from major cloud providers offer secret rotation entirely by themselves. Each service provides different levels of support and tools for secret rotation

Audit and Compliance

Each service offers audit and compliance features suitable for different organisational needs and cloud environments. AWS Secrets Manager stands out for its depth and detail, Azure Key Vault for its balance of thoroughness and accessibility, and Google Secret Manager for its straightforward, integrated approach within the Google Cloud. The choice depends on the specific audit requirements, the cloud platform used, and the level of detail needed in compliance records.

Cloud Agnostic

All three services, AWS Secrets Manager, Azure Key Vault, and Google Secret Manager, inherently lead to a degree of vendor lock-in. They are optimised for their respective cloud environments, which could be a significant consideration for organisations aiming for a multi-cloud strategy or those who may want to avoid dependency on a single cloud provider.

Centralised Storage

All three services offer centralised storage solutions for managing secrets within their respective cloud ecosystems. They each provide a secure and centralised way to handle sensitive data across various services and applications, tailored to their specific cloud environments. The choice among these depends on the cloud infrastructure and services an organisation predominantly uses.

Tenant Interface

Each service provides a tenant interface that reflects its respective cloud ecosystem. AWS Secrets Manager is feature-rich and integrates well within AWS, but may be more complex for newcomers. Azure Key Vault offers a streamlined experience within Azure, making it ideal for Azure-centric users. Google Secret Manager focuses on simplicity and ease of use, catering to users who prioritise a straightforward interface within the Google Cloud ecosystem. The choice depends on the user’s familiarity with the respective cloud platform and their specific interface preferences.

API / CLI

All three services provide robust API and CLI tools, but with varying degrees of complexity and integration within their respective ecosystems. AWS Secrets Manager offers a highly flexible and feature-rich set of tools, suitable for complex scenarios. Azure Key Vault’s tools are well-integrated within the Azure environment, making them ideal for Azure-centric workflows. Google Secret Manager focuses on simplicity and ease of use, which is beneficial for straightforward use cases and users who prefer a less complex toolset. The choice among them would depend on the specific needs of the organisation and familiarity with the respective cloud platform.

Developer Friendly

While AWS Secrets Manager, Azure Key Vault, and Google Secret Manager provide robust solutions for secret management, they do require developers to modify application code to retrieve and manage secrets, which can add complexity and development overhead. This factor might make these services less appealing from a developer-friendliness standpoint, particularly in scenarios where ease of integration and minimal code changes are prioritised. The decision to use a particular service should consider the trade-offs between security features offered and the additional development effort required.

Platform Friendly

All three services are quite platform engineering-friendly, but their suitability varies depending on the specific cloud environment and the complexity of the platform engineering tasks. AWS Secrets Manager is a robust choice for AWS-centric environments with complex needs, Azure Key Vault aligns well with platform engineering in Azure, and Google Secret Manager offers a simpler, more streamlined approach, particularly beneficial in Google Cloud-focused platform engineering. The decision would typically be influenced by the existing cloud infrastructure and the specific needs of the platform engineering team.

2. Cloud Agnostic Providers

Moving from cloud-specific solutions like AWS Secrets Manager, Azure Key Vault, and Google Secret Manager, we now explore Cloud Agnostic Secret Store providers. These alternatives are especially relevant for organisations operating in multi-cloud environments or looking to prevent vendor lock-in.

They offer flexibility and seamless integration across diverse cloud platforms, ensuring robust and uniform secret management. This segment of secret management is broadly divided into two categories: Licensed Options and Open Source Options, each catering to different organisational needs and preferences in a cloud-agnostic context.

Α. Licensed Options

HashiCorp Vault

HashiCorp Vault is a tool for managing secrets safely, designed to work well with applications in Kubernetes. It helps application teams by taking care of the tough job of handling sensitive information like passwords and API keys.

Vault is equipped with a variety of modular components, known as secret engines and authentication methods, which are key to its integration with external systems. These components are pivotal in managing and safeguarding secrets within dynamic infrastructures, such as database credentials, passwords, and API keys.

Secrets engines are the heart of Vault’s functionality, responsible for storing, generating, or encrypting data. They are activated on specific paths within Vault, and the system’s router is designed to direct requests to the appropriate secrets engine based on the path prefix. This structure allows each secret engine to have its distinct paths and attributes, catering to various types of secret management needs.

On the other hand, authentication methods within Vault are crucial for verifying the identity of Vault clients. They play a critical role in assigning a set of policies to authenticated clients, ensuring that access control is both secure and compliant. Particularly noteworthy is the Kubernetes Auth Method, which streamlines how applications interact with Vault. It eliminates the need to manage additional tasks, such as handling tokens for Vault access, thus simplifying the overall process for Kubernetes-based applications.

You can set up a Vault server cluster directly in Kubernetes using Helm charts, which is the recommended method. In this setup, you have the Vault server running in stateful pods. There’s also something called a Vault agent injector, which is another pod. This injector uses Kubernetes' mutating admission webhook feature to find pods with certain annotations and adds a Vault agent container to these pods. This setup is useful for managing secrets both for applications inside Kubernetes and those outside of it.

Let’s go over the main parts of HashiCorp’s Vault in Kubernetes:

Vault Server: This is the main part of Vault. It offers an API that clients use to work with Vault, like interacting with secrets engines, enforcing access control lists (ACLs), and handling the expiration of secrets (secret lease revocation).

Vault Agent: This is a client program that runs continuously. Its job is to log into the Vault server and manage things like tokens, which are used for accessing secrets in Vault.

Agent Injector: In Kubernetes, the Agent Injector functions as a Mutating Admission Webhook, designed to detect pods marked with a specific annotation (vault.hashicorp.com/agent-inject: true). Upon recognizing this annotation, it enriches the pod by integrating two containers: an init container (Vault Agent) and a sidecar container. The Vault Agent init container is tasked with preparing the secrets before the main application kicks off, essentially handling login and secret retrieval processes. Concurrently, the sidecar container is responsible for maintaining the freshness of these secrets throughout the application’s lifecycle. This setup ensures that applications can securely access and utilize secrets without hardcoding sensitive information, leveraging dynamic secret management capabilities. For more insights, HashiCorp provides a comprehensive overview of why using the Vault Agent for secrets management enhances security and efficiency.

Using Vault in Kubernetes like this has some advantages:

Applications don’t need to know about Vault because the secrets are kept in the file system of their container.

You don’t need to change existing deployments. You can just add annotations to them.

You can control who accesses the secrets using Kubernetes service accounts and namespaces.

How it works

Initial Token Acquisition:

• When a pod is deployed in Kubernetes, it automatically receives a Service Account token (JWT).
• This JWT is passed along to Vault, initiating the authentication process.

Authentication Against Kubernetes:

• Vault uses the provided JWT to authenticate against the Kubernetes cluster.
• It does this by calling the TokenReview API, which validates the JWT and returns the associated service account name and namespace to Vault.

Policy Validation:

• Vault then checks the service account details against its internal policies to determine if the account is authorized to access the requested secrets.
• This step ensures that only pods with the correct permissions can retrieve secrets.

Vault Token Issuance:

• Upon successful authentication and authorization, Vault issues a token back to the pod.
• This Vault token is then used for subsequent API calls to retrieve secrets.

Secret Retrieval:

• The pod makes API calls to Vault, including the Vault token.
• Vault verifies the token and, if the token has the appropriate permissions, returns the requested secrets to the pod.

Secrets Delivered to Pod:

• Secrets are securely delivered to the pod, with a common method being through mounted volumes, where they are readily accessible for the application within the container. While direct injection of secrets into the pod’s environment variables is not inherently supported by the Vault Agent, a workaround involves utilising templating features. This approach allows for the dynamic generation of scripts or configuration files at application startup, effectively exporting the secrets into the environment.

In addition to the agent injector, there is the distinct Vault Secrets Operator (VSO) integration:

Secret Synchronisation:

The primary function of the Vault Secrets Operator is to synchronise secrets between HashiCorp Vault and Kubernetes. It does this by watching for specific Custom Resource Definitions (CRDs) in the Kubernetes cluster that define how secrets should be synchronised from Vault to Kubernetes.

Custom Resource Definitions (CRDs):

VSO utilises CRDs to declare the desired state of Kubernetes secrets that are managed by Vault. These CRDs specify details like which Vault secrets to synchronise and how they should be represented within Kubernetes.

Operation Process:

Monitoring Vault Secrets: VSO continuously monitors Vault for any changes to the secrets that are specified in the CRDs.

Updating Kubernetes Secrets:

When a change is detected, VSO updates the corresponding Kubernetes secret with the new data from Vault. This process ensures that the Kubernetes secrets always reflect the current state of the secrets in Vault.

Secrets Accessibility:

Once updated, these Kubernetes secrets can be used by applications running in the cluster just like any other native Kubernetes secret. This means applications do not need to be aware of Vault’s existence; they can access secrets as if they were always part of Kubernetes.

Vault Policies:

The access to secrets in Vault is still governed by Vault’s own policies. VSO must be configured with the necessary permissions in Vault to access and synchronize the secrets.

These processes ensures that secret access is securely controlled and that only authenticated and authorized entities within the Kubernetes cluster can access the sensitive data they need. The integration between Vault and Kubernetes provides a secure and automated way to manage and inject secrets into applications running in containers.

End to End Encryption

Vault: Provides encryption for secrets at rest and in transit.

Injector: Injects secrets into pods as needed; these are encrypted by Vault and only decrypted within the pod.

VSO: Synchronises secrets from Vault to Kubernetes. In Kubernetes, secrets are stored in etcd and are encoded but not encrypted unless the Kubernetes cluster is configured for etcd encryption at rest.

Access Control

Vault: Utilizes robust access control mechanisms, allowing only authenticated and authorized users to manage and access secrets. This includes the application of fine-grained policies that govern the operations users can perform.

Injector: Operates under Vault’s access control regime, injecting secrets into Kubernetes pods based on permissions defined in Vault. This ensures that secrets are only made available to pods and services with the appropriate level of access.

Vault Secrets Operator (VSO): Synchronizes Vault’s access control policies with Kubernetes by reflecting secret updates in etcd, adhering to Kubernetes' RBAC settings. This alignment ensures that secret management across Vault and Kubernetes maintains a consistent security model, with access strictly controlled according to predefined policies and roles.

Vault:Supports automatic secret rotation for dynamic secrets, where it generates and revokes secrets dynamically.

For static secrets, rotation needs to be managed externally or manually.

Injector:If configured to inject dynamic secrets, these secrets benefit from Vault’s automatic rotation capabilities.

Does not handle rotation for static secrets; any such rotation must be managed in Vault or through external means.

Vault Secrets Operator (VSO):Reflects changes in Kubernetes if secrets are rotated in Vault, regardless of whether they are dynamic or static.

The rotation logic itself resides in Vault for dynamic secrets, or externally for static secrets. VSO simply syncs the updated secrets to Kubernetes.

Secret Rotation

Vault: Supports secret rotation, but the process itself must be defined and managed.

Injector: Does not handle rotation within Kubernetes; any rotation must be managed in Vault.

VSO: Reflects rotated secrets in Kubernetes, but rotation logic resides in Vault.

Audit Compliance

Vault: Offers comprehensive audit logs.

Injector and VSO: Both methods adhere to Vault’s audit logging, ensuring compliance.

Cloud Agnostic

Vault: Cloud-agnostic, can be used across different cloud providers.

Injector and VSO: These methods are also cloud-agnostic, as they depend on Vault and Kubernetes, not on cloud-specific features.

Centralised Storage

Vault: Acts as a centralised secret store.

Injector: Does not store secrets in Kubernetes; it pulls from Vault as needed.

VSO: Synchronises secrets into Kubernetes’ centralised storage (etcd).

Tenant Interface

Vault: Provides a web UI for managing secrets and policies.

Injector and VSO: No separate interface; they rely on Vault’s and Kubernetes' interfaces.

API / CLI

Vault: Has a robust API and CLI for interaction.

Injector and VSO: Integration occurs at the Kubernetes level, utilising Kubernetes and Vault APIs/CLIs.

Developer Friendly

Vault: Highly flexible but has a learning curve.

Injector: Simplifies secret management in Kubernetes, reducing developer workload.

VSO: Offers Kubernetes-native secret management, aligning with typical Kubernetes workflows.

Platform Friendly

Vault: Supports IaC tools like Terraform, is scalable, and works across multiple environments, fitting well into diverse platform engineering requirements.

Injector: Eases Vault’s integration in Kubernetes, aligning with cloud-native platform engineering practices and automation.

VSO: Offers Kubernetes-native secret management through CRDs, aligning with the declarative approach of modern platform engineering.

Β. Open source Options

Infisical

Infisical provides a streamlined solution for managing secrets within Kubernetes, offering a user-friendly alternative to more complex systems like HashiCorp Vault. Infisical stands out for its ease of use and robust security, being both open-source and end-to-end encrypted.

Infisical’s Integration with Kubernetes:

Infisical interacts with Kubernetes through a custom operator. This operator extends Kubernetes functionality, using custom resources to manage applications and their components. Once the Infisical Secrets Operator is installed in your Kubernetes cluster, it introduces a new resource type called InfisicalSecret.

InfisicalSecret Resource:

An InfisicalSecret resource allows you to define how you authenticate with Infisical to retrieve secrets and where those secrets should be stored. Here’s an example of what an InfisicalSecret resource might look like:

---
apiVersion: secrets.infisical.com/v1alpha1
kind: InfisicalSecret
metadata:
 # Name of this InfisicalSecret resource
 name: infisicalsecret-example
spec:
 # The host that should be used to pull secrets from. 
 # The default value is https://app.infisical.com/api
hostAPI: https://app.infisical.com/api

 # The Kubernetes secret that stores the Infisical token
 tokenSecretReference:
   # Kubernetes secret name
   secretName: infisical-example-service-token
   # The secret namespace
   secretNamespace: default

 # The Kubernetes secret that Infisical Operator will create
 # and populate with secrets from the above project
 managedSecretReference:
   # The name of managed Kubernetes secret that should be created
   secretName: infisical-managed-secret
   # The namespace the managed secret should be installed in
   secretNamespace: default

In this setup, the operator fetches secrets from Infisical using a service token and saves them to a native Kubernetes secret. This process is automated and occurs at a regular interval (e.g., every minute).

End to End Encryption

Infisical encrypts secrets both at rest and in transit using techniques like AES-256-GCM, ensuring protection against data breaches.

While Infisical maintains encryption during secret transfer to Kubernetes, the final encryption level in Kubernetes' etcd depends on the cluster’s encryption-at-rest settings.

Access Control

Infisical employs a sophisticated Role-Based Access Control (RBAC) system that allows for the creation of permissions for both user and machine identities, effectively limiting access to resources and permissible actions within the platform. Access Control Policies in Infisical offer a granular, declarative approach to manage access rights, allowing for distinctions across projects and organisations. At the organisation level, users are categorised as either admins or members, with admins granted comprehensive control over organisational settings, member management, and project creation, while members have more limited capabilities. This delineation ensures that administrative tasks are clearly separated from project-level access, where further differentiation occurs among users as viewers, developers, or admins, each with specific access rights to project resources and actions. Importantly, Infisical enables the creation of custom roles, providing organisations the flexibility to tailor access according to their unique needs, from super admin roles to roles with restricted access to certain secrets or folders. This level of customization in Infisical’s RBAC facilitates precise control over the security and management of sensitive information, aligning access permissions with organisational roles and responsibilities.

Infisical combines multiple authentication methods (like SSO and email/password) with a secure remote password protocol for enhanced security. It further strengthens access control through a comprehensive RBAC system, allowing fine-grained policy management and custom role assignments for organisational members, ensuring secure and tailored access to secrets.

Access control to Kubernetes secrets is governed by Kubernetes' native mechanisms, such as RBAC (Role-Based Access Control).This means the access control is as robust as your Kubernetes configuration, relying on service accounts, roles, and role bindings.

Secret Rotation

Infisical can automate the rotation of secrets by fetching updated secrets at regular intervals.

Kubernetes Sync: Once updated in Infisical, the changes are automatically synchronised with the Kubernetes secrets, though the rotation policy would be managed in Infisical or the source of the secret.

Audit Compliance

Infisical’s compliance with audit requirements would depend on its logging and monitoring capabilities.For actions within Kubernetes, audit trails would be part of Kubernetes’ own audit logs, assuming auditing is properly configured in the cluster.

Cloud Agnostic

As an external tool that integrates with Kubernetes, Infisical does not inherently lock you into a specific cloud provider.

Centralised Storage

Synchronises secrets into Kubernetes’ centralised storage (etcd).

Tenant Interface

Provides a web UI for managing secrets.

API / CLI

Has a robust API and CLI for interaction.

Developer Friendly

Highly flexible but has a learning curve.Simplifies secret management in Kubernetes, reducing developer workload.

Platform Friendly

Offers Kubernetes-native secret management through CRDs, aligning with the declarative approach of modern platform engineering.

Exploring Comprehensive Solutions

External Secret Operator

The External Secret Operator in Kubernetes extends the native secret management capabilities by interfacing with external secret stores like AWS Secrets Manager, Azure Key Vault, Google Secret Manager, and more.

It operates by introducing a Custom Resource Definition (CRD) in Kubernetes, which represents the external secrets. This CRD allows you to define secrets that are stored outside of the Kubernetes cluster, typically in a cloud provider’s secret management system.

When you deploy the External Secret Operator in your Kubernetes cluster, it monitors these CRDs and fetches the actual secret data from the external store. It then creates a native Kubernetes secret in the cluster’s etcd database with this data. Your applications deployed in Kubernetes can use these secrets as if they were regular Kubernetes secrets.

Integrating an external secret store with Kubernetes using the External Secret Operator brings several benefits aligned with key standards and best practices:

End to End Encryption

While external secret providers handle the encryption of secrets at rest in their own systems, once those secrets are brought into Kubernetes through the ESO, they are not automatically encrypted within etcd. As a result, additional measures are necessary to secure these secrets within the Kubernetes environment

Access Control

Access control policies are defined and enforced within the Kubernetes cluster, independent of the external secret store’s access control capabilities.

Secret rotation

Depend on the external secret store. The ESO itself does not manage secret rotation; it only synchronises the secrets from the external store into Kubernetes. If the external store rotates a secret, the ESO will update the corresponding Kubernetes secret.

Audit Compliance

Auditing for secrets managed by the ESO involves both the external secret store’s audit logs and Kubernetes' audit logs. The external secret store provides audit logs for actions within its system, while Kubernetes tracks access and modifications to the Kubernetes secrets.

Cloud Agnostic

The External Secret Operator is designed to be cloud-agnostic, offering flexibility and reducing the risk of vendor lock-in, even though the secret stores themselves may be provider-specific.

Centralised Storage

Synchronises secrets into Kubernetes’ centralised storage (etcd).

Tenant Interface

Does not provide any UI, The UI is served from secret store provider

API / CLI

Has no API or CLI

Developer Friendly

The operator simplifies secret management, reducing the burden on developers by abstracting the complexity of direct interactions with external stores.

Platform Friendly

The operator introduces some management overhead but generally reduces the overall complexity of secret management at scale within Kubernetes.

How External Secret Operator affect Secret Store Providers

CSI Secret Store

The Kubernetes CSI (Container Storage Interface) Secret Store is a method that allows Kubernetes to interface with various external secret stores, such as AWS Secrets Manager, Azure Key Vault, and Google Secret Manager. Unlike the External Secret Operator which uses Custom Resource Definitions (CRDs), the CSI Secret Store utilises CSI drivers to mount secrets from external stores into Kubernetes pods as volumes.

When the CSI Secret Store is deployed in a Kubernetes cluster, it utilises a CSI provider specific to the external secret store. This driver mounts the desired secrets as volumes into the Kubernetes pods. As a result, applications can access these secrets as if they were files in a volume, without the secrets ever being written to the cluster’s etcd database.

End to End Encryption

Since the CSI Secret Store mechanism mounts secrets directly into pods, the secrets are not stored in the Kubernetes etcd. This approach means that concerns about encrypting secrets within etcd are mitigated. However, the encryption of the secrets at rest and in transit is still handled by the external secret providers.

Access Control

Access to the mounted secrets is controlled by Kubernetes' native access controls and the specific configurations of the CSI driver. This can include Kubernetes' RBAC mechanisms and pod-specific access policies.

Secret Rotation

Secret rotation is managed by the external secret store. The CSI Secret Store will reflect any changes or rotations in the secrets as per the configurations and interactions with the external store.

Audit Compliance

Auditing involves the external secret store’s audit logs for tracking access and changes to the secrets. Since secrets are not stored in etcd, Kubernetes' audit logs would primarily show access to the volumes rather than the secrets themselves.

Cloud Agnostic

The CSI Secret Store is designed to be cloud-agnostic, offering flexibility and reducing the risk of vendor lock-in, even though the secret stores themselves may be provider-specific.

Centralised Storage

Does not store secrets in Kubernetes; it pulls from Secret Store Providers as needed.

Tenant Interface

Does not provide any UI, The UI is served from secret store provider

Developer Friendly

By providing a standardised way to mount secrets into pods, the CSI Secret Store simplifies the process for developers, reducing the complexity in handling secrets.

Platform Friendly

The CSI Secret Store introduces some overhead in terms of managing the CSI drivers and the associated configurations but simplifies the overall process of secret management in Kubernetes environments.

Comparison

End to End Encryption

ESO: Secrets are stored unencrypted in Kubernetes' etcd by default. Additional measures like etcd encryption are needed.

SSCSID: Focuses on encryption provided by the external secret store; no additional encryption within Kubernetes is required.

Access Control

ESO: Uses Kubernetes' Role-Based Access Control (RBAC) for accessing secrets.

SSCSID: Manages access through Kubernetes' native controls for volumes and pods.

Secret Rotation

Both ESO and SSCSID rely on the external secret store for secret rotation, with automatic reflection of changes in Kubernetes.

Audit Compliance

ESO: Requires audit log consolidation from both Kubernetes and the external secret store.

SSCSID: Relies mainly on the external secret store’s auditing capabilities; Kubernetes auditing tracks volume access.

Cloud Agnostic

Both are designed to be cloud-agnostic, supporting multiple external secret stores. However, there is some dependence on a specific secret store provider.

Centralized Storage

ESO: Creates Kubernetes secrets in etcd, available cluster-wide.

SSCSID: Does not store secrets in etcd; they are mounted directly into the pods.

Tenant Interface

ESO and SSCSID: No separate interface; they rely on secret store provider

API / CLI & Developer Friendly

ESO: Familiar to those used to Kubernetes secrets, minimal changes to application code.

SSCSID: Provides a seamless experience for developers as secrets appear as files in a volume.

Platform Friendly

ESO: Integrates well with Kubernetes, allowing secrets to be used with any resource natively.

SSCSID: Requires additional components like a pod webhook for broader integration. It’s less straightforward for uses outside of pod-specific scenarios.

The following table presents a comprehensive comparison between the External Secret Operator (ESO) and the CSI Secret Store Driver (SSCSID) in terms of their features and implementations.

Conclusion

Each organisation must carefully select a secrets management solution that best suits its specific needs and operational context.

The choice should be guided by a thorough evaluation of various factors such as encryption strength, access control mechanisms, ability to rotate secrets, compliance with audit requirements, cloud environment compatibility, and ease of integration into existing workflows.

Considering the diverse options available, from Kubernetes' native tools to sophisticated external solutions like cloud-native and cloud-agnostic providers, organisations should weigh the trade-offs between security, flexibility, and convenience to find the most suitable approach for their unique requirements.

Ultimately, the chosen solution should enhance the organisation’s overall security posture while aligning seamlessly with its technological infrastructure and operational strategies.

When assessing external secret store providers based on a range of critical features, two solutions stand out as the primary choices: “Vault Injector” and “Vault integrated with CSI Secret Store.”

These solutions excel across all evaluated criteria, including encryption, access control, audit capabilities, cloud agnosticism, centralised storage, tenant interface, and API/CLI availability. Their comprehensive capabilities make them the ideal choices for organisations seeking a versatile and robust secret management solution that can seamlessly operate in complex, multi-cloud, or hybrid cloud/on-premises environments.

As a secondary option, the integration of CSI Secret Store with cloud provider solutions like AWS Secrets Manager, Azure Key Vault, and Google Secret Manager presents a compelling alternative. This combination covers most of the essential features, with the notable exception of secret rotation. It offers strong encryption, access control, and a platform-friendly approach, making it suitable for organisations that are closely aligned with specific cloud providers. While it may not provide the complete range of features that Vault Injector and Vault with CSI Secret Store offer, this second-tier option still delivers substantial capabilities in secret management, particularly for those environments where secret rotation is not a critical requirement.

By leveraging the CSI Secret Store, organisations gain the ability to seamlessly integrate with both Vault and any cloud provider’s secret manager, allowing for a more fluid and adaptable secret management infrastructure.

Ultimately, the selection of the right secret management solution depends on the specific needs and strategic goals of an organisation. Organisations should carefully consider their requirements for features like secret rotation, cloud agnosticism, and platform integration to make the most informed decision for their secret management strategy.