Document EventBridge routing architecture and Lambda message processing

[drernie]

Summary

Comprehensive documentation of EventBridge → SNS → SQS → Lambda routing architecture, replacing hypothesis-focused analysis with an architectural guide focused on the core problem.

Key Changes:

• Renamed 10-input-transformer-hypothesis.md → 10-eventbridge-routing.md
• Restructured content to focus on the core problem (SNS message wrapping) rather than one solution approach
• Added detailed Lambda processing patterns from code review
• Expanded testing strategy with isolation requirements and common mistakes

Core Documentation Sections

Message Flow Architecture

• EventBridge → SNS → SQS → Lambda transformation chain
• Example message structures at each layer
• Required unwrapping steps (SQS → SNS → Event payload)

Lambda Processing Patterns

Analyzed 4 Lambda types from actual code:

• ❌ ManifestIndexer (≤1.65): Missing SNS unwrapping → crashes
• ✅ SearchHandler: Correct SNS unwrapping → works with S3 Records
• ⚠️ EsIngest: EventBridge variant format → untested
• ⚠️ Iceberg: EventBridge format → untested

The Two Event Sources Problem

Why testing was flaky: production environments may have both:

1. Direct S3 Notifications (instant, S3 Records format)
2. EventBridge Route (3-5 min delay, EventBridge format)

Critical insight: Files appeared in search via direct S3 notifications, masking EventBridge routing failures.

Solutions & Approaches

Solution 1: Fix Lambda Code (Recommended)

• Add SNS unwrapping to ManifestIndexer
• Minimal code change, maintains backward compatibility
• Platform 1.66+

Solution 2: Input Transformers (Limited Use)

• Converts EventBridge → S3 Records format
• Helps SearchHandler process EventBridge events
• ❌ Cannot solve SNS unwrapping issue (transforms BEFORE SNS wrapping)

Solution 3: Dual Format Support (Comprehensive)

• Enhanced Lambdas detect and handle multiple formats
• Handles all event sources gracefully
• Platform 1.66+

Testing Strategy

Critical Testing Principle: ALWAYS isolate event sources to avoid false positives

4 Test Scenarios:

1. Baseline - Direct S3 Notifications
2. EventBridge WITHOUT Input Transformer
3. EventBridge WITH Input Transformer
4. Platform 1.66 with SNS Unwrap Fix

Common Testing Mistakes:

• ❌ Relying on intermediate metrics (EventBridge triggered ≠ Lambda success)
• ❌ Not isolating event sources (success via S3, not EventBridge)
• ❌ Not waiting for CloudTrail (3-5 min delay)
• ❌ Assuming success from invocation count (may be crashing)

Production Deployment

Version-Specific Recommendations:

• Platform ≤1.65: ManifestIndexer will NOT work with EventBridge routing
• Platform 1.66: ManifestIndexer works, SearchHandler needs Input Transformer
• Platform 1.66+: All Lambdas work with dual format support

Impact

This documentation:

• ✅ Explains why EventBridge routing breaks certain Lambdas
• ✅ Provides clear testing methodology to avoid false positives
• ✅ Documents three solution approaches with version guidance
• ✅ Includes S3 Event Notification management commands
• ✅ Focuses on architectural understanding, not one failed solution

Related Issues

• Resolves confusion about Input Transformer effectiveness
• Documents the "two event sources" testing trap
• Provides version-specific migration path

🤖 Generated with Claude Code