AzureFixes Logo
AZUREFIXES
DEBUG FASTER. DEPLOY SMARTER.
Published on
4 min read

ExpressRoute Design Changes: Scaling for Segmentation & HA

Goal

This document outlines a recent shift in ExpressRoute architecture to improve scalability, segmentation, and high availability (HA). The previous design had several technical limitations that became bottlenecks as the cloud footprint expanded.


Previous Topology

The legacy ExpressRoute configuration was built using:

  • Standard SKU ExpressRoute circuit
  • A single Virtual Network Gateway (VNG) in the Hub VNet
  • Spoke VNets peered to the hub for ER access
  • No Global Reach or cross-region BCDR

Limitations Observed:

  • Prefix limits exceeded during routing updates (max 10,000 for Standard SKU)
  • Lack of regional segmentation – one blast radius for all apps
  • No Global Reach – cross-geo traffic routed inefficiently via VPN/Internet
  • Throughput constraints due to shared VNG and circuit

New Design Highlights

The revised architecture leverages:

  • Premium SKU ExpressRoute Circuit
  • Dedicated VNG per region (or per functional Hub)
  • Enabled ExpressRoute Global Reach for east-west BCDR
  • Segmented VNets by workload/region via multiple peering domains

Architecture Diagrams

Previous Design

graph TD
    A[On-Prem Router] -->|BGP/Private Peering| B[ER Circuit (Standard)]
    B --> C[ER Gateway (Hub VNet)]
    C --> D1[Spoke VNet 1]
    C --> D2[Spoke VNet 2]
    C --> D3[Spoke VNet 3]

New Design

graph TD
    A1[On-Prem DC - South Central] -->|BGP| B1[ER Circuit - South Central (Premium)]
    A2[On-Prem DC - East] -->|BGP| B2[ER Circuit - East (Premium)]

    B1 --> C1[ER GW - Hub South Central US]
    B2 --> C2[ER GW - Hub East US]

    C1 --> D1[Spoke VNet 1 - South Central]
    C1 --> D2[Spoke VNet 2 - South Central]
    C2 --> E1[Spoke VNet 1 - East]
    C2 --> E2[Spoke VNet 2 - East]

    B1 <--> B2[Global Reach]

Technical Changes Implemented

ChangeDescription
Circuit UpgradeUpgraded ER from Standard to Premium to lift prefix limits and allow Global Reach
VNG RedesignCreated multiple ER Gateways in separate Hubs (regional or functional)
Peering ModelImplemented dual-peering strategy per environment
Route FilteringEnabled to control learned prefixes
Traffic SegmentationUsed UDRs and NVAs to route regionally and isolate workloads

Sample Change Log Entry

Change ID: CHG-AZR-ER-2025-07
Date: 2025-07-15
Summary: ER Circuit upgrade + Hub redesign for segmentation and HA
Action:
- Upgraded ER circuit to Premium
- Deployed new ER GW in Central US
- Connected 3 spoke VNets using regional hub
- Enabled Global Reach with East US circuit
Impact: Improved BCDR and removed prefix limit error observed during routing updates

Risk Mitigation Steps

  • Staged Deployment: Implemented regional hub design incrementally
  • Backout Plan: Preserved previous ER circuit until routing validation completed
  • Prefix Review: Analyzed BGP route tables before rollout to avoid blackholing
  • Monitoring: Enabled NPM, Log Analytics for route and latency tracking

PowerShell Validation Script

# Get ExpressRoute Circuit Details
$circuit = Get-AzExpressRouteCircuit -ResourceGroupName "rg-network" -Name "er-circuit-premium"
$circuit | Format-List Name,ServiceProviderProperties,BandwidthInMbps,PeeringLocation,Sku

# List all Peerings
Get-AzExpressRouteCircuitPeering -ExpressRouteCircuit $circuit

# Validate Peering Connection Status
$circuit.Peerings | ForEach-Object {
    Write-Host "Peering: $($_.PeeringType) | State: $($_.State) | AzureASN: $($_.AzureASN)"
}

# Get associated VNet Gateway and Connections
$vng = Get-AzVirtualNetworkGateway -ResourceGroupName "rg-network" -Name "er-gw-central"
Get-AzVirtualNetworkGatewayConnection -VirtualNetworkGateway1 $vng | Format-Table Name,ConnectionType,ConnectionStatus

Terraform Validation Snippet

data "azurerm_express_route_circuit" "er_circuit" {
  name                = "er-circuit-premium"
  resource_group_name = "rg-network"
}

output "er_circuit_details" {
  value = {
    service_provider_name = data.azurerm_express_route_circuit.er_circuit.service_provider_name
    peering_location      = data.azurerm_express_route_circuit.er_circuit.peering_location
    bandwidth             = data.azurerm_express_route_circuit.er_circuit.bandwidth_in_mbps
    sku                   = data.azurerm_express_route_circuit.er_circuit.sku[0].name
  }
}

data "azurerm_virtual_network_gateway" "er_gateway" {
  name                = "er-gw-central"
  resource_group_name = "rg-network"
}

output "gateway_bgp_settings" {
  value = data.azurerm_virtual_network_gateway.er_gateway.bgp_settings
}

Related Articles

Fixing Azure WAF False Positives: A 7-Phase Diagnostic Guide

If Azure WAF is blocking legitimate traffic after a policy update, new deployment, or rule set upgrade, false positives are the most common cause. This guide walks through 7 diagnostic phases to identify, reproduce, and safely exclude the offending rule — without disabling WAF protection.

Fixing Azure VPN Gateway High CPU: BGP Route Explosion Diagnosis and Fix

If your VPN Gateway CPU is above 85% while traffic volume looks normal, BGP route bloat is almost certainly the cause. This guide covers the exact commands to diagnose it, route summarization on the CE router, an Azure-side prefix filter for defense in depth, and when to upgrade the gateway SKU.

Fixing Enterprise SSO and OAuth Failures in Entra ID: An 8-Step Diagnostic Guide

If your enterprise SSO logins are returning AADSTS errors, redirect_uri mismatches, or silent token refresh failures, this guide walks through 8 diagnostic steps to locate and fix the root cause — covering OAuth 2.0, OIDC, Entra ID app registration, and high-availability token caching.

Diagnosing Kubernetes DNS Resolution Failures: CoreDNS Upstream Forwarding Broken by resolv.conf Drift

If some pods in your AKS cluster resolve external DNS names fine and others fail — from the same Deployment, same image, same config — the variable is the node. This guide covers the exact test sequence to isolate CoreDNS upstream forwarding failures, identify /etc/resolv.conf drift on worker nodes, and fix it without reprovisioning.

How to Diagnose Azure ExpressRoute Gateway Control-Plane Failure

If your ExpressRoute Gateway shows BGP Connected but cross-premises traffic is down and effective routes on spoke VNet NICs are empty, the gateway has a control-plane failure. This guide covers the diagnostic commands to confirm it, a regional failover procedure, and the zone-redundant HA architecture to prevent recurrence.