Keeping Up Your Infra 24/7: Machine Perspective

Overview

A guest session for RevoU. The audience was people learning infrastructure rather than running it yet, so the framing had to start with why availability matters commercially before it could get to how you engineer for it.

The opening argument: with the internet embedded in daily life, slight latency becomes outrage, an application error becomes a social media post, and downtime becomes a one-star rating. It impacts reputation, and reputation always impacts business.

The counterweight came immediately after: systems are destined to fail. Software and distributed systems eventually break because something can always go wrong. You accept that and design for resilience, test accordingly, and think through the edge cases. To run 24/7 you prepare for the worst case, not the expected one.

Key Takeaway

• The error budget makes reliability a decision, not an aspiration.
  Once you have 52.6 minutes a year written down, arguments about whether to ship stop being philosophical and start being arithmetic.
• Build the system around the SLI you actually committed to.
  Response-time SLOs and error-rate SLOs point at completely different engineering work. Knowing which one you are on tells you where to spend.
• Redundancy beats reliability-per-component.
  Two cheap cars beat one expensive one. Removing single points of failure buys more availability than making any single component more perfect.
• Verify the manual process before you automate it.
  A machine executes a wrong instruction faster and more consistently than a human ever would.

Summary

The seven SRE core principles, each one stated first as a principle and then re-stated from the machine's point of view.

1. Embracing risk

   100% reliability is unrealistic and unnecessary, because as reliability approaches 100%, reducing the remaining risk gets exponentially harder and more expensive. Hence the error budget: 99.99% over a year still allows 52.6 minutes of downtime, and that is budget you are allowed to spend. Machine view: even the most expensive car can throw a rod, so buy two cheap cars instead. The whole job is designing out the single point of failure, via disaster recovery sites, cross-region sites, A-B failover, or HA. Redundancy is the mechanism.

2. Service level objectives

   An SLO is a measurable quality target, expressed through SLIs: commonly availability, query rate, error rate, and response time. Machine view: build the system around whichever SLI you committed to. If the SLO is about response time, engineer for throughput: geo-proximity routing, CDN, caching. If it is about error rate, engineer for stability: horizontal and vertical scaling, orchestration, automation.

3. Eliminating toil

   Toil is manual work that scales linearly, repeats, and leaves the system no better than before, plus cognitive toil, the time lost re-reading guidance for something that keeps recurring, which is what standard documentation is for. Machine view: find toil by looking for patterns. When does traffic peak? Does the issue always appear at a specific utilisation percentage? Does the same fix always work? A confirmed pattern is an automation candidate.

4. Monitoring

   Monitoring means looking at meaningful, actionable data and deciding from it. Too much data means no insight at all. Follow the four golden signals: latency, traffic, errors, saturation. Machine view: leave no stone unturned, everything gets an agent. Optimise logging by dropping the level to WARN/ERROR rather than drowning in INFO. Alert on critical metrics at defined thresholds.

5. Automation

   The point is freeing engineers from repetitive work that adds no value, so they can do the things automation cannot: creation, ideation, large-scale guidance. It also reduces human error, increases execution speed, and enforces consistency. Machine view: verify the manual task 100% first, because the machine will follow a wrong instruction faithfully. Confirm integration is actually supported rather than assuming an approach that worked on system A ports to system B. Plan logic, concurrency, and capacity deliberately.

6. Release engineering

   Built in from the beginning rather than assigned at the last minute, on four key points: automation and self-service, velocity, identical builds, and a unified standard. Machine view: rolling deployment, blue/green, or canary. Designing a system that can release autonomously is important; designing one that can roll back is mandatory. And keep a manual approval gate for production-level change.

7. Simplicity

   The least complex system that still does the job, because simpler systems are easier to monitor, repair, and improve, and that includes the tooling the team uses. Machine view: avoid duplicate tech stacks, because more tools means never-ending integration work. Compare overlapping tools and keep the one that fits. For open-source, check it is still actively maintained. Flashy tools often do not meet the business objective, so put business first.