Jeevan Kumar -- Cloud Center of Excellence

Imagine you’re running a highly secure, private Kubernetes environment in Azure. Everything is well-architected, adhering to the best security practices, until your deployments from Azure DevOps start failing. Your namespaces aren’t loading, containers aren't pulling, and debugging feels like navigating a maze. This is the reality for many teams working with private Azure Kubernetes Service (AKS) clusters and Azure Container Registry (ACR) instances with restricted network access.

Deploying workloads to such restricted environments presents challenges, including authentication issues, service connection limitations, and network access constraints. However, with the right strategy, we can establish a secure, efficient deployment pipeline. In this blog, we’ll explore these challenges, break down the limitations of traditional service connections, and discuss a robust solution using Azure Resource Manager (ARM)-based authentication with the right IAM roles.

How Azure DevOps connects to Kubernetes

Azure DevOps supports three primary methods for creating service connections to deploy Kubernetes manifests:

For public AKS clusters, all three methods work seamlessly. However, the moment you introduce private networking, the landscape changes dramatically.

Authentication failures with Kubernetes 1.24+

Starting with Kubernetes 1.24, long-lived tokens are no longer created by default. Kubernetes recommends avoiding long-lived tokens altogether, which impacts authentication when using:

When these service connections fail, the ‘Loading namespaces’ dialog in Azure DevOps hangs indefinitely, preventing deployments.

Service account authentication issues

Service account-based connections also encounter authentication problems due to:

These limitations make service accounts unreliable for private AKS clusters.

Network restrictions impacting ACR access

In a private ACR setup, organizations often disable public network access to enhance security. However, this introduces challenges such as:

To overcome these roadblocks, we need an alternative authentication and networking strategy.

To maintain security without sacrificing functionality, we can leverage ARM-based authentication with dedicated IAM roles. This approach aligns with Kubernetes' best practices and avoids long-lived tokens while ensuring seamless connectivity.

Step 1: Create an ARM-based service connection in Azure DevOps

Instead of using static tokens or service accounts, create an ARM-based service connection in Azure DevOps. This method generates short-lived tokens dynamically, enhancing security and compliance with Kubernetes authentication guidelines.

|Assign the following IAM roles to the service principal used in the service connection:

Step 2: Enable Kubernetes local accounts for entra ID authentication

If your AKS cluster uses Microsoft Entra ID authentication, ensure the “Kubernetes local accounts” checkbox is enabled. This setting allows the creation of temporary credentials when needed, improving authentication flexibility.

Step 3: Use a self-hosted agent for private ACR access

Since Azure DevOps managed agents cannot access private endpoints, use a self-hosted agent deployed within your virtual network. This agent should have network line-of-sight to:

By deploying a self-hosted agent, you ensure that deployments can push and pull container images without requiring public network access.

Step 4: Whitelist Azure DevOps public IP in ACR (If necessary)

If public network access is disabled on ACR, explicitly whitelist the Azure DevOps public IP to allow pipeline tasks to interact with the registry.

This can be configured via:

Step 5: Leverage short-lived tokens for Kubernetes authentication

One of the biggest advantages of ARM-based service connections is their ability to generate short-lived tokens dynamically. Each task execution in Azure DevOps receives a new token instead of relying on static credentials, ensuring:

Additional considerations for advanced scenarios

AKS Private Link Integration

For even greater security, integrate AKS with Azure Private Link to eliminate exposure over the public internet. This requires careful planning of:

Automating IAM role assignment

To simplify role assignments for new service connections, consider automating the process using Azure CLI or Terraform. For example:

Monitoring and troubleshooting

Monitor Kubernetes authentication and network connectivity using:

Building secure and efficient CI/CD pipelines

Deploying workloads to private AKS clusters and ACR instances presents unique challenges, but by leveraging ARM-based service connections, self-hosted agents, and proper IAM role assignments, we can achieve a secure, scalable, and efficient CI/CD pipeline.

TP is a Microsoft Azure Solutions Partner for Data & AI, highlighting our expertise in creating tailored analytics and AI solutions. We help businesses tackle challenges, boost efficiency, and gain valuable insights.

Visit our technology services page to learn more.