Cloud Patterns: Hub and Spoke Network Topology using Azure, Terraform and Kubernetes

An architectural pattern is a general, reusable solution to a commonly occurring problem in software architecture within a given context. One very useful pattern in a real world scenario is the hub and spoke network topology architecture.

A hub an spoke is a reference network architecture pattern where we have a central node called the hub where we can deploy common or central services, for example a VM Bastion Host, a Firewall or a VPN Gateway, and the hub acts as a central point of connectivity (entry point).

Then we have the spokes that are “clients”, which are logically segregated from the hub and can connect to or consume the services from the hub, this clients can range from normal VM’s to PaaS service integrated into a VNET.

Normally in Azure we deploy this kind of network topology using resource groups to separate the hub and spokes. In each hub and spoke we deploy VNETS and we use VNET Peerings to connect the spokes to the hub.

Use Cases

• Implementation of forwarded traffic using route tables to an Azure Firewall
• Implementation of a single point of entry using an Application Gateway.
• Segregation of resources
• Deployment of central common or shared services, for example a central Recovery Vault or Key Vault that is going to be shared by other services

Advantages

• The workloads are segregated, you can apply more granular security policies and best practices
• Easier setup of network connectivity to on-premises services by using a single connectivity point (hub)
• Proposes a clean organisation
• Combined with tags helps to better organise resources

Disadvantages

• The IP address space has to be defined in the beginning and carefully tracked, peering is not possible if there is overlapping in the VNET address spaces.
• Have also to manage carefully the VNET peering between the hub’s and spokes.

Rules of thumb

• Try to segregate your spoke functionally at the application level.
• 1 Spoke = 1 Service or Application = 1 Resource Group = 1 Environment
  • Example: rg-spoke-my-app-pre-prod-01
• Track your address space using any kind of solution (such as an excel sheet) to keep things well organised
• Don’t hesitate to add tags to better identify the type of resources, such as Topology:Hub or Topology: Spoke.

Architecture Example of an Hub and Spoke Architecture using terraform

In our practical example of the Hub and Spoke architecture we are going to deploy a hub that is going to be used as a central point of access using an application gateway with a WAF and a Virtual Machine used as a Bastion.

Modules

First of all we will make use of the following modules:

application_gateway.tf

data "azurerm_subnet" "snet-frontend" {
  depends_on           = [var.subnets]
  name                 = var.application_gateway.snet_frontend_name
  virtual_network_name = var.vnet.name
  resource_group_name  = var.resource_group
}

resource "azurerm_public_ip" "example" {
  depends_on          = [var.subnets]
  name                = var.application_gateway.public_ip_name
  resource_group_name = var.resource_group
  location            = var.location
  allocation_method   = "Dynamic"
}

# since these variables are re-used - a locals block makes this more maintainable
locals {
  backend_address_pool_name      = "${var.vnet.name}-beap"
  frontend_port_name             = "${var.vnet.name}-feport"
  frontend_ip_configuration_name = "${var.vnet.name}-feip"
  http_setting_name              = "${var.vnet.name}-be-htst"
  listener_name                  = "${var.vnet.name}-httplstn"
  request_routing_rule_name      = "${var.vnet.name}-rqrt"
  redirect_configuration_name    = "${var.vnet.name}-rdrcfg"
}

resource "azurerm_application_gateway" "network" {


  # App Gateway Config

  depends_on          = [var.subnets]
  name                = var.application_gateway.name
  resource_group_name = var.resource_group
  location            = var.location
  tags                = var.tags

  sku {
    name     = var.application_gateway.sku.name
    tier     = var.application_gateway.sku.tier
    capacity = var.application_gateway.sku.capacity
  }

  #Web Application Firewall Config

    waf_configuration {

  enabled                   = var.application_gateway.waf_configuration.enabled
  firewall_mode             = var.application_gateway.waf_configuration.firewall_mode
  rule_set_type             = var.application_gateway.waf_configuration.rule_set_type
  rule_set_version          = var.application_gateway.waf_configuration.rule_set_version
  disabled_rule_group       = var.application_gateway.waf_configuration.disabled_rule_group
  file_upload_limit_mb      = var.application_gateway.waf_configuration.file_upload_limit_mb
  request_body_check        = var.application_gateway.waf_configuration.request_body_check
  max_request_body_size_kb  = var.application_gateway.waf_configuration.max_request_body_size_kb
  exclusion                 = var.application_gateway.waf_configuration.exclusion
  #}


  # FrontEnd

  ssl_certificate {
    name     = "certificate"
    data     = var.certificate
    password = var.certificate-password
  }

  gateway_ip_configuration {
    name      = var.application_gateway.gateway_ip_configuration_name
    subnet_id = data.azurerm_subnet.snet-frontend.id
  }

  frontend_port {
    name = local.frontend_port_name
    port = 443
  }

  frontend_ip_configuration {
    name                 = local.frontend_ip_configuration_name
    public_ip_address_id = azurerm_public_ip.example.id
  }

  http_listener {
    name                           = local.listener_name
    frontend_ip_configuration_name = local.frontend_ip_configuration_name
    frontend_port_name             = local.frontend_port_name
    protocol                       = "Https"
    ssl_certificate_name           = "certificate"
  }

  # Backend  

  backend_http_settings {
    name                  = local.http_setting_name
    cookie_based_affinity = "Disabled"
    path                  = "/"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
  }

  backend_address_pool {
    name  = local.backend_address_pool_name
    fqdns = [var.application_gateway.fqdns]
  }

  # Rules

  request_routing_rule {
    name                       = local.request_routing_rule_name
    rule_type                  = "Basic"
    http_listener_name         = local.listener_name
    backend_address_pool_name  = local.backend_address_pool_name
    backend_http_settings_name = local.http_setting_name
  }
}

vnet.tf

locals {
  subnets = [
    for s in var.subnets : merge({
      name                      = ""
      address_prefix            = ""
      network_security_group_id = ""
      #route_table_id            = ""
      delegations               = []
      service_endpoints         = []
    }, s)
  ]

  subnets_delegations = [
    for s in local.subnets : {
      name                      = s.name
      address_prefix            = s.address_prefix
      network_security_group_id = s.network_security_group_id
     # route_table_id            = s.route_table_id
      service_endpoints         = s.service_endpoints
      delegations = [
        for d in s.delegations : {
          name = lower(split("/", d)[1])
          service_delegation = {
            name    = d
            actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
          }
        }
      ]
    }
  ]

  subnet_ids = { for s in azurerm_subnet.main : s.name => s.id }

  network_security_group_ids = {
    for s in local.subnets : s.name =>
    s.network_security_group_id if s.network_security_group_id != ""
  }

 

  network_security_group_associations = [
    for subnet, id in local.network_security_group_ids : {
      subnet_id                 = local.subnet_ids[subnet]
      network_security_group_id = id
    }
  ]


}

data "azurerm_resource_group" "main" {
  name = var.resource_group_name
}

resource "azurerm_virtual_network" "main" {
  name                = var.name
  resource_group_name = data.azurerm_resource_group.main.name
  address_space       = var.address_space
  location            = coalesce(var.location, data.azurerm_resource_group.main.location)
  dns_servers         = var.dns_servers
}

resource "azurerm_subnet" "main" {
  count                = length(local.subnets_delegations)
  name                 = local.subnets_delegations[count.index].name
  resource_group_name  = azurerm_virtual_network.main.resource_group_name
  virtual_network_name = azurerm_virtual_network.main.name

  address_prefix = local.subnets_delegations[count.index].address_prefix

  dynamic "delegation" {
    for_each = local.subnets_delegations[count.index].delegations

    content {
      name = delegation.value.name

      service_delegation {
        name    = delegation.value.service_delegation.name
        actions = delegation.value.service_delegation.actions
      }
    }
  }

  service_endpoints = local.subnets_delegations[count.index].service_endpoints

  lifecycle {
    #ignore_changes = ["network_security_group_id"]
  }
}

resource "azurerm_subnet_network_security_group_association" "main" {
  count                     = length(local.network_security_group_associations)
  subnet_id                 = local.network_security_group_associations[count.index].subnet_id
  network_security_group_id = local.network_security_group_associations[count.index].network_security_group_id
}

key_vault.tf

provider "azurerm" {
  version = "~>2.10.0"
  features {}
}

terraform {
    backend "azurerm" {    
    }
}

data "azuread_group" "main" {
  count = length(local.group_names)
  name  = local.group_names[count.index]
}

data "azuread_user" "main" {
  count               = length(local.user_principal_names)
  user_principal_name = local.user_principal_names[count.index]
}

data "azurerm_resource_group" "main" {
  name = var.resource_group_name
}

data "azurerm_client_config" "main" {}

resource "azurerm_key_vault" "main" {
  name                = var.name
  location            = data.azurerm_resource_group.main.location
  resource_group_name = data.azurerm_resource_group.main.name
  tenant_id           = data.azurerm_client_config.main.tenant_id

  enabled_for_deployment          = var.enabled_for_deployment
  enabled_for_disk_encryption     = var.enabled_for_disk_encryption
  enabled_for_template_deployment = var.enabled_for_template_deployment
  tags = var.tags

  sku_name = var.sku

  dynamic "access_policy" {
    for_each = local.combined_access_policies

    content {
      tenant_id = data.azurerm_client_config.main.tenant_id
      object_id = access_policy.value.object_id

      certificate_permissions = access_policy.value.certificate_permissions
      key_permissions         = access_policy.value.key_permissions
      secret_permissions      = access_policy.value.secret_permissions
      storage_permissions     = access_policy.value.storage_permissions
    }
  }

  dynamic "access_policy" {
    for_each = local.service_principal_object_id != "" ? [local.self_permissions] : []

    content {
      tenant_id = data.azurerm_client_config.main.tenant_id
      object_id = access_policy.value.object_id

      #certificate_permissions = access_policy.value.certificate_permissions
      key_permissions         = access_policy.value.key_permissions
      secret_permissions      = access_policy.value.secret_permissions
      #storage_permissions     = access_policy.value.storage_permissions
    }
  }


}

resource "azurerm_key_vault_secret" "main" {
  for_each     = var.secrets
  name         = each.key
  value        = each.value
  key_vault_id = azurerm_key_vault.main.id
}

vnet_peering.tf

resource "azurerm_virtual_network_peering" "peering_src" {
  provider = azurerm.src

  name = coalesce(
    var.custom_peering_dest_name,
    format("peering-to-%s", local.vnet_dest_name),
  )
  resource_group_name          = local.vnet_src_resource_group_name
  virtual_network_name         = local.vnet_src_name
  remote_virtual_network_id    = var.vnet_dest_id
  allow_virtual_network_access = var.allow_virtual_src_network_access
  allow_forwarded_traffic      = var.allow_forwarded_src_traffic
  allow_gateway_transit        = var.allow_gateway_src_transit
  use_remote_gateways          = var.use_remote_src_gateway
}

resource "azurerm_virtual_network_peering" "peering_dest" {
  provider = azurerm.dest

  name = coalesce(
    var.custom_peering_src_name,
    format("peering-to-%s", local.vnet_src_name),
  )
  resource_group_name          = local.vnet_dest_resource_group_name
  virtual_network_name         = local.vnet_dest_name
  remote_virtual_network_id    = var.vnet_src_id
  allow_virtual_network_access = var.allow_virtual_dest_network_access
  allow_forwarded_traffic      = var.allow_forwarded_dest_traffic
  allow_gateway_transit        = var.allow_gateway_dest_transit
  use_remote_gateways          = var.use_remote_dest_gateway
}

acr.tf

resource "azurerm_container_registry" "acr" {
  name                     = var.acr.name
  resource_group_name      = var.resource_group
  location                 = var.location
  sku                      = var.acr.sku
  admin_enabled            = var.acr.admin_enabled
  
  tags = var.tags
}

Hub

Then we are going to deploy the hub that makes use of the modules we have created before.

We will handle the peering at the hub level. Once there is a new resource that is going to be peered to the hub we have to modify the parameters from the hub and add the new origin and destination vnet in the peering section.

main.tf

provider "azurerm" {
  version = "~>2.10.0"
  features {}
}


terraform {
    backend "azurerm" {    
    }
}

locals {
    prefix          = ""
}

module "key-vault" {
  source = "./../../modules/key-vault/"
  name = var.key-vault.name
  resource_group_name = var.resource_group
  access_policies = var.key-vault.access_policies
  secrets = var.key-vault.secrets
  tags                              = var.tags
}

data "azurerm_key_vault" "keyvault" {
depends_on = ["module.key-vault"]
name = "${var.key-vault.name}"
resource_group_name = var.resource_group
}

data "azurerm_key_vault_secret" "secret-apgw-certificate" {
name = "secret-apgw-certificate"
key_vault_id = "${data.azurerm_key_vault.keyvault.id}"
}

data "azurerm_key_vault_secret" "secret-apgw-password" {
name = "secret-apgw-password"
key_vault_id = "${data.azurerm_key_vault.keyvault.id}"
}


module "network" {
    source = "./network/"
    resource_group = var.resource_group
    location = var.location
    vnet = var.vnet
    
}

data "azurerm_virtual_network" "network" {
depends_on = ["module.network"]
name = "${var.vnet.name}"
resource_group_name = var.resource_group
}

module "app_gateway" {
    source = "./application_gateway/"
    resource_group = var.resource_group
    location = var.location
    application_gateway = var.application_gateway
    tags=var.tags
    vnet=var.vnet
    subnets= data.azurerm_virtual_network.network.subnets
    certificate=data.azurerm_key_vault_secret.secret-apgw-certificate.value
    certificate-password=data.azurerm_key_vault_secret.secret-apgw-password.value
}

module "acr" {
    source = "./acr/"
    resource_group = var.resource_group
    location = var.location
    acr = var.acr
    tags=var.tags
}

data "azurerm_public_ip" "app_gateway_fqdn" {
depends_on = ["module.app_gateway"]
name = var.application_gateway.public_ip_name
resource_group_name = var.resource_group
}

# Peering module

module "peering" {
    source = "./common/peering/"
    resource_group = var.resource_group
    #Required to wait for subnets to be deployed
    subnets= data.azurerm_virtual_network.network.subnets
    vnet-peerings=var.vnet-peerings 
}

Spoke

Finally we are going to deploy our spoke with our AKS cluster

main.tf

provider "azurerm" {
  version = "~>2.10.0"
  
  features {}
}

terraform {
    backend "azurerm" {   
      
    }
}

locals {
    prefix          = ""
}

module "spoke_network" {
    source = "./network/"
    resource_group = var.resource_group
    location = var.location
    vnet = var.vnet 
}

module "aks" {
    source = "./aks/"
    vm_size = var.aks.vm_size
    os_disk_size_gb  = var.aks.os_disk_size_gb 
    resource_group = var.resource_group
    location = var.location  
    agent_count = var.aks.agent_count
    ssh_public_key_data= var.aks.ssh_public_key_data  
    dns_prefix = var.aks.dns_prefix
    cluster_name = var.aks.cluster_name
    log_analytics_workspace_name = var.aks.log_analytics_workspace_name
    log_analytics_workspace_location = var.location
    log_analytics_workspace_sku = var.aks.log_analytics_workspace_sku
    client_id = var.aks.client_id
    client_secret = var.aks.client_secret
    private_cluster_enabled = var.aks.private_cluster_enabled
    node_address_space=  var.aks.node_address_space
    node_address_prefix= var.aks.node_address_prefix
    node_resource_group= var.aks.node_resource_group
    vnet-aks= var.aks.vnet-aks
    tags=var.tags
  
       
}

aks.tf

resource "azurerm_virtual_network" "vnet" {
  name                = var.vnet-aks
  location            = var.location
  resource_group_name = var.resource_group
  address_space       = var.node_address_space
}

resource "azurerm_subnet" "aks-subnet" {
  name                 = "snet-aksnodes"
  resource_group_name  =  var.resource_group
  address_prefix       =  var.node_address_prefix
  virtual_network_name = "${azurerm_virtual_network.vnet.name}"
}


resource "azurerm_kubernetes_cluster" "k8s" {
    name                = var.cluster_name
    location            = var.location
    node_resource_group = var.node_resource_group
    resource_group_name = var.resource_group
    dns_prefix          = var.dns_prefix
    depends_on =[azurerm_subnet.aks-subnet]
    tags = var.tags
    #private_cluster_enabled = var.private_cluster_enabled

    network_profile {
        network_plugin      =   "azure"
        load_balancer_sku   =   "Standard"
    }
    
    role_based_access_control {
        enabled = true
    }
    
    linux_profile {
        admin_username = "ubuntu"

        ssh_key {
            key_data = var.ssh_public_key_data
        }
    }

    default_node_pool {
        name            = "agentpool"
        node_count      = var.agent_count
        vm_size         = var.vm_size
        vnet_subnet_id  = azurerm_subnet.aks-subnet.id
        os_disk_size_gb = var.os_disk_size_gb 
    }

     
   service_principal {
        client_id     = var.client_id
        client_secret = var.client_secret
    }
    
    
}

Finally at the kubernetes level we should use an ingress that exposes a private ip address and use this IP address in our application gateway to route connections towards our cluster.

And voila! We have deployed our Kubernetes Cluster using a Hub an Spoke Architecture, the advantage is that when we have to deploy other Clusters we can reuse the code of our spoke and only change parameters (for example the environment type from dev to pre prod) and then we will have a main entry point for all our services defined at the hub level protected by the Web Application Firewall. Furthermore we will delegate all SSL offloading to the Application Gateway.