⌘K
Event Pipeline
Workstream 2 · Data Model Unification & Unified Event Streaming Architecture.
Pipeline Health Score
End-to-end SLOs across translator, registry, brokers and consumers.
Overall Pipeline Health
96%
Data Contract Compliance
98%
Translation Success Rate
99.2%
Event Delivery Rate
99.8%
DLQ Rate
0.03%
Pipeline flow
Event volume analytics
Throughput, hot event types and delivery outcomes.
Events per hour
24h
Top event types
ShipmentEvent
38%
ETA_UPDATED
22%
WORKFLOW_TRIGGERED
14%
EXCEPTION_RAISED
11%
TASK_ASSIGNED
9%
ACTION_COMPLETED
6%
Translation volume trend (14d)
Success · 99.2%
Failure · 0.8%
14.2M events · last 24h
Canonical schema registry
Confluent Schema Registry · 7 canonical models
| Schema | Version | Status | Validation | Actions |
|---|---|---|---|---|
| ShipmentEvent | v2.1 | Active | 98% | |
| ETA_UPDATED | v1.8 | Active | 99% | |
| SHIPMENT_DELAYED | v1.3 | Active | 97% | |
| EXCEPTION_RAISED | v1.2 | Active | 96% | |
| WORKFLOW_TRIGGERED | v2.0 | Active | 99% | |
| TASK_ASSIGNED | v1.1 | Active | 98% | |
| ACTION_COMPLETED | v1.0 | Active | 99% |
Event translation layer
JVM translator · Avro ↔ canonical · 218 mapping rules
Events Translated
14.2M
Translation Success
99.2%
Transformation Errors
118
Avg Processing Time
38ms
Most Common Error
Schema mismatch
Translation volume (14d)
Kafka topics
| Topic | Throughput | Consumers | Lag | Status | Actions |
|---|---|---|---|---|---|
| company-a.shipments.v1 | 12.4k/s | 8 | 120ms | Healthy | |
| company-a.tracking.v1 | 28.1k/s | 12 | 85ms | Healthy | |
| company-b.workflows.v1 | 6.2k/s | 6 | 240ms | Degraded | |
| canonical.eta.v2 | 9.8k/s | 10 | 62ms | Healthy | |
| canonical.alerts.v1 | 2.1k/s | 4 | 510ms | Degraded | |
| canonical.shipment.v1 | 18.9k/s | 11 | 56ms | Healthy | |
| port.dnd.events.v1 | 640/s | 3 | 12ms | Healthy | |
| dlq.shipments | 12/s | 2 | — | DLQ |
Failed events
| Event ID | Type | Error | Retry |
|---|---|---|---|
| evt_8a21f4 | shipment.update | Schema mismatch: missing carrier_scac | Retrying |
| evt_8a21d1 | eta.update | Timeout writing to Company B workflow svc | Retrying |
| evt_8a2199 | alert.create | 401 on downstream Keycloak token | Failed |
| evt_8a2174 | tender.book | Validation: pickup_window required | DLQ |
| evt_8a2150 | shipment.update | Network reset | Resolved |
Consumer group monitoring
| Consumer Group | Topic | Members | Lag | Last Commit | Status | Actions |
|---|---|---|---|---|---|---|
| workflow-engine | canonical.eta.v2 | 6 | 124 | 2s ago | Healthy | |
| shipment-processor | canonical.shipment.v1 | 8 | 38 | 1s ago | Healthy | |
| alert-engine | canonical.alerts.v1 | 4 | 2,104 | 12s ago | Degraded | |
| notification-service | canonical.eta.v2 | 5 | 56 | 1s ago | Healthy | |
| audit-collector | canonical.shipment.v1 | 2 | 12,450 | 4m ago | Failed | |
| bigquery-sink | canonical.shipment.v1 | 3 | 412 | 5s ago | Healthy |
Dead letter queue management
Quarantined messages awaiting triage
DLQ Events
36
Resolved
412
Pending
12
Retry Queue
8
| Event ID | Topic | Reason | Age | Status | Actions |
|---|---|---|---|---|---|
| dlq_4a91 | dlq.shipments | Schema validation failed: carrier_scac | 4m | Planned | |
| dlq_4a90 | dlq.eta | Downstream 504 timeout | 12m | retry | |
| dlq_4a8f | dlq.alerts | 401 Keycloak token expired | 1h | Planned | |
| dlq_4a8e | dlq.shipments | Consumer offset out of range | 2h | archived | |
| dlq_4a8d | dlq.workflows | Translator unavailable | 3h | Resolved |
Data contract compliance
Schema evolution governance · last audit 14m ago
Schema Validation Success
99.6%
Contract Violations
12
Breaking Changes
2
Compatible Changes
34
BigQuery unified analytics
Unified shipment + workflow data warehouse · dbt models · v2
Rows Processed (24h)
284M
Analytics Queries (24h)
12,418
Data Freshness
42s
Pipeline Sync Status
Healthy
Event pipeline roadmap
Recent activity