Home Projects Architecture Case Studies AWS
Coming Soon AWS AWS DevOps Engineer Professional

Chaos Engineering with OpenTelemetry

PRJ-AWS-DOP-025

Resilience testing with distributed tracing

~8 min read Intermediate
Status Coming Soon
Last Updated Jan 16, 2026
Completion 0%
Status: Coming Soon· Last Updated: Jan 16, 2026· Completion: 0%· ~8 min read· Intermediate
Download Guide Watch Tutorial View Architecture Download Architecture

Estimated Monthly Cost

~$32/mo on minimal config
CodePipeline $10ECS $12CloudWatch $6S3 $4
Business ContextLack of visibility into complex distributed systems makes identifying root cause…

The Problem

  • Lack of visibility into complex distributed systems makes identifying root causes of failures challenging and time-consuming.
  • Traditional testing methods often fail to uncover latent vulnerabilities and unexpected behaviors under adverse conditions.
  • Unplanned outages and performance degradation directly impact customer satisfaction and business revenue.

The Solution

  • Implement OpenTelemetry for standardized, end-to-end distributed tracing across all microservices and infrastructure.
  • Utilize AWS X-Ray to visualize service maps, trace requests, and pinpoint performance bottlenecks and errors within the AWS ecosystem.
  • Conduct automated Chaos Engineering experiments using tools integrated with CloudWatch to proactively identify and remediate system weaknesses.

Business Value

  • Reduces mean time to recovery (MTTR) by 40% through enhanced observability and faster root cause analysis.
  • Increases system uptime by 15% by proactively addressing resilience gaps identified through chaos experiments.
  • Improves developer productivity by 25% by providing clear insights into application behavior and dependencies.
  • Ensures compliance with operational resilience standards, avoiding potential regulatory fines and reputational damage.

Risk Mitigation

  • Addresses the risk of undetected system vulnerabilities leading to critical service disruptions.
  • Mitigates the impact of cascading failures in distributed architectures by improving system resilience.
  • Reduces the likelihood of performance degradation and service outages due to unforeseen dependencies or resource contention.
  • Enhances the ability to recover from infrastructure failures and malicious attacks by validating recovery mechanisms.
GRC MappingNIST Cybersecurity Framework (CSF): Identify (Asset Management), Protect (Config…

Compliance Frameworks

  • NIST Cybersecurity Framework (CSF): Identify (Asset Management), Protect (Configuration Management), Detect (Continuous Monitoring), Respond (Incident Response), Recover (Recovery Planning).
  • ISO 27001: A.12.4 Logging and monitoring, A.17.1 Information security continuity management.
  • Operational Resilience (e.g., DORA, PRA SS2/21): Focus on identifying critical business services, mapping dependencies, and testing resilience.

Security Controls Implemented

  • Distributed Tracing (OpenTelemetry/AWS X-Ray): Provides granular visibility into request flows, aiding in anomaly detection and forensic analysis.
  • Centralized Logging (AWS CloudWatch Logs): Aggregates application and infrastructure logs for security monitoring and incident investigation.
  • Automated Alerting (AWS CloudWatch Alarms): Triggers notifications for deviations from baseline behavior or detected security events.
  • Configuration Management (AWS CloudFormation/Terraform): Ensures consistent and secure deployment of infrastructure and application components.
  • Chaos Engineering Experiments: Proactively tests system resilience against various failure scenarios to identify and remediate weaknesses.

Audit Evidence

  • AWS X-Ray trace data and service maps demonstrating request flow and error rates.
  • CloudWatch Logs and metrics showing system performance, resource utilization, and error patterns during chaos experiments.
  • Reports from Chaos Engineering platforms detailing experiment execution, observed impacts, and remediation actions.
  • OpenTelemetry span data and metrics collected from application components.

Regulatory Alignment

  • DORA (Digital Operational Resilience Act): Article 12 (ICT incident management), Article 13 (Digital operational resilience testing).
  • GDPR (General Data Protection Regulation): Article 32 (Security of processing) through enhanced system resilience and incident response capabilities.
  • PCI DSS (Payment Card Industry Data Security Standard): Requirement 10 (Track and monitor all access to network resources and cardholder data) via comprehensive logging and monitoring.

Video tutorial coming soon!

Subscribe to our YouTube channel to get notified when this tutorial is published.

Subscribe on YouTube

Architecture Diagram

PRJ-AWS-DOP-025 Architecture

Technology Stack

FIS
OpenTelemetry
X-Ray
CloudWatch
Chaos Engineering

Complete Documentation

Prerequisites

IAM Admin or PowerUser role
AWS CLI v2 configured
Terraform >= 1.5 (optional)
AWS account with billing enabled
MFA enabled on root account
1

Clone & Configure

Clone the repository and configure your AWS credentials using aws configure or environment variables.

aws configure --profile cloudguard
2

Review IAM Policies

Review and attach the required IAM policies to your deployment role. Ensure least-privilege access is applied.

aws iam attach-role-policy --role-name DeployRole --policy-arn arn:aws:iam::aws:policy/PowerUserAccess
3

Initialize Infrastructure

Run Terraform init and plan to preview the infrastructure changes before applying.

terraform init && terraform plan -out=tfplan
4

Deploy Resources

Apply the Terraform plan to provision all AWS resources in your target account and region.

terraform apply tfplan
5

Verify & Monitor

Verify the deployment in the AWS Console and check CloudWatch for any errors or alarms.

aws cloudwatch describe-alarms --state-value ALARM

Deployment Guide

Step-by-step instructions to deploy this project

Download Guide

Architecture Diagram

Visual representation of the system architecture

Download Architecture

Source Code

Complete source code and configuration files

View on GitHub

Video Tutorial

Watch the complete walkthrough video

Watch Now