AWS DX Auto-Failover

Context

Tech Mahindra runs cloud operations for Indonesia's largest telco, production workloads spread across 20+ AWS accounts and 10+ Kubernetes clusters, reaching AWS over dedicated Direct Connect circuits rather than the public internet.

Dedicated circuits are physical. They run through real fiber, in real ground, and anything that can happen to real ground can happen to the link: road work, construction digging where it shouldn't, even a rat chewing through a cable. Over the preceding period the link had been severed repeatedly by exactly these kinds of causes, each cut an unplanned outage on a path that revenue-generating traffic depended on.

The hybrid setup had a redundant path for exactly this reason, but redundancy on paper isn't redundancy in production. If the primary Direct Connect link went down and traffic wasn't moved to the healthy path fast enough, it kept trying to route over the same troubled link, and every transaction riding on it was a transaction that could fail.

Challenge

A redundant path existed. What did not exist was anything that moved traffic onto it without a person deciding to.

The switchover was a documented manual procedure: notice the incident, confirm it was physical rather than transient, then execute the cutover by hand. That sequence is slow at 2 PM and slower at 2 AM, and every minute of it was billable downtime on a link carrying production traffic.

Detecting the fault early didn't skip any of that. Cutting over still needed sign-off before anyone touched the link, which in practice meant an actual phone call to the client's representative, even if the fault was caught at 3 AM. The bottleneck was never how fast the incident could be noticed. It was how fast a human on the other end could be reached and could say yes.

Action

Turned into a requirement, the challenge was simple to state and hard to build: detect a physical-layer fault the moment it happens, confirm it's genuinely a break rather than a transient blip, and cut over to the healthy path immediately, all without a phone call standing between the fault and the fix.

1. Detect

   Continuously watch the Direct Connect link's health rather than waiting for a downstream service alarm to surface it second-hand.

2. Validate

   Distinguish a real physical-layer failure from transient noise, so the system doesn't flap traffic on every momentary blip.

3. Switch

   Execute the cutover to the redundant path automatically, the same steps the runbook prescribed, minus the wait for a human to run them.

4. Notify

   Tell the operations team what happened and what state the link is now in, so the human involvement moves from executing to reviewing.

My approach was to treat detection, validation, and execution as one pipeline instead of three separate fixes. A fast detector paired with a slow, manual validation step would just move the bottleneck instead of removing it, so the whole path from fault to cutover had to run without a human in the middle of it, not just at the start.

Tech Stack

AWS Direct Connect provides the redundant transit VIFs this system watches over, fed by health metrics from a monitoring EC2 instance provisioned through an AWS Systems Manager Automation runbook. Amazon CloudWatch rolls those metrics into one composite alarm per link, and when it fires, it invokes the AWS Lambda orchestrator, which calls the Direct Connect API to bring BGP down on the affected VIF. Amazon S3 holds the last-known VIF state between invocations, so a re-trigger can be told apart from a first run. Once the BGP failover test's own wait period ends, an Amazon EventBridge rule re-invokes the Lambda to recheck whether the link is still unhealthy: if it is, the failover reruns; if it has recovered, the rule just passes. Amazon SNS sends the operator a notification once the action is taken. A second Lambda renders a CloudWatch dashboard widget using the diagrams library, so the topology and its live BGP status stay visible without leaving the console.

Results

The manual switchover procedure was eliminated. Failover stopped depending on who was on shift and how quickly they could confirm what had happened.

Measured against the recurrent fiber-optic outage pattern that prompted the work, the automation accounts for IDR 79B+ in yearly revenue loss prevented, the difference between an outage that ends when a human reacts and one that ends when a system does.

On the operations side, the bigger change is what stopped happening. 30+ of these link failures had occurred by the last time I had visibility into the count, and not one of them required someone to be pulled out of bed, confirm what was going on, and approve a cutover by hand. The team's role shifted from executing the switchover to reviewing what the system already did, an SNS notification instead of a page demanding action. That consistency matters as much as the speed: every failover now runs the same validated sequence, instead of depending on which engineer picked up the phone and how sharp they were at 3 AM.