Ops Briefing Surface

Production Reliability Dashboard

Generated 2026-07-14 01:29 for 2026-06-28 07:00 to 2026-07-05 07:00 from Pingdom checks, Slack #_alerts_prod, and AWS SNS alerts.

All sources Pingdom customer checks Slack alert families AWS alarm emails
Email-confirmed customer incidents0Pingdom down/slow events confirmed by inbox alertsObserved in Pingdom: 0
Impacted services24Mapped from Slack and Pingdom evidence
AWS alarms in ALARM0Still alarming at window end
Latest observed signal2026-07-05 03:19Most recent cross-source activity
Run notes

Data source availability

Some enrichments were unavailable in this run; drilldowns below stay focused on captured evidence.

Executive summary

What needs attention

Bottom line: application-level critical paths are present.

Pingdom customer impact

External signal
No criticalActive: 0Total seen: 0

No Pingdom checks were available in this window.

No active issue listed in this category.

Slack impacted services

Application signal
5 criticalActive: 24Total seen: 24

5 critical, 19 non-critical active item(s).

AWS alarms

Infrastructure signal
No criticalActive: 0Total seen: 0

No AWS alarm emails were captured in this window.

No active issue listed in this category.

What to do next

  1. NextTreat critical services web-80, grafana, library-api, admission-api, social-api as the primary application investigation set

    Treat critical services web-80, grafana, library-api, admission-api, social-api as the primary application investigation set. Use TraefikServiceHighErrorRate as the leading category, reproduce the failing paths on web-80, grafana, library-api, admission-api, social-api, compare against the latest deploy or config change, and do not close it until both error rate and latency flatten.

    Executive recommendation
  2. ThenReduce the operational drag on minicrm-sync by separating repeated symptom alerts from the underlying workload failure

    Reduce the operational drag on minicrm-sync by separating repeated symptom alerts from the underlying workload failure. Either eliminate the recurrent fault or retune the alert once the failure mode is understood.

    Executive recommendation
Global Evidence Explorer

Global Evidence Explorer

Report-wide charts and tables stay here, separate from the active investigation scope.

Application + Infrastructure Alerts by Day

Pingdom latency + downtime by source

Customer View

Pingdom Checks

Pingdom CheckStatusEventsDowntimeLast SeenLikely ServicesCorrelated Evidence

Pingdom rows show externally visible signal first. The correlated evidence column helps tie the failing check back to services, Slack alert families, or AWS alarms when those links exist.

Application View

Slack Impacted Service / Resource View

This view attributes alerts to the workload or resource named in the alert text. Grafana, Loki, and Tempo are treated as observability components and are excluded when a more specific impacted target is also present.

Impacted Service / ResourceHighest SeverityCountLast SeenStatusTop Alert TypesDiscussion SignalLatest Thread Note
grafanaCritical282026-07-03 13:43Recent (72h)Platform5xxLowVolumeCritical100 (3)KubeContainerWaiting (15)Platform5xxLowVolumeWarning20 (5)KubeHpaMaxedOut (2)KubeContainerMemoryHigh (1)Release / migration issue
this was due to recent deployment on production , the 5xx errors . should be gone now , as the social migration
library-api
Grouped 3 variantsVariant mentions 8Active variants 3
Critical82026-07-03 09:22Recent (72h)TraefikServiceHighErrorRate (2)KubePodContainerRestartingFrequently (6)NoneNo thread note
web-80Critical52026-07-03 00:55Recent (72h)TraefikServiceHighErrorRate (3)TraefikServiceHighLatency (2)NoneNo thread note
admission-apiCritical42026-07-02 11:28Recent (72h)TraefikServiceHighErrorRate (2)TraefikServiceHighLatency (2)Release / migration issue
admission-api is single-replica (replicas=1). During the rev 271 deploy the one pod was replaced, so there was a brief window with no ready…
social-apiCritical12026-06-29 10:18Seen this weekTraefikServiceHighErrorRate (1)Release / migration issue
this was due to recent deployment on production , the 5xx errors . should be gone now , as the social migration
minicrm-syncWarning242026-07-05 03:19Seen todayKubeJobFailed (24)NoneNo thread note
colecteaza-sms-note-abs
Grouped 10 variantsVariant mentions 40Active variants 10
Warning362026-07-03 09:25Recent (72h)KubeJobFailed (29)KubePodContainerRestartingFrequently (7)NoneNo thread note
download-album
Grouped 10 variantsVariant mentions 39Active variants 10
Warning342026-07-03 09:25Recent (72h)KubeJobFailed (27)KubePodContainerRestartingFrequently (7)NoneNo thread note
rezumat
Grouped 10 variantsVariant mentions 38Active variants 10
Warning332026-07-03 09:25Recent (72h)KubeJobFailed (26)KubePodContainerRestartingFrequently (7)NoneNo thread note
core-getresponse-events-worker
Grouped 3 variantsVariant mentions 12Active variants 3
Warning152026-07-03 00:57Recent (72h)KubePodContainerRestartingFrequently (8)KubePodCrashLooping (4)KubeDeploymentReplicasMismatch (3)NoneNo thread note
access-codes-reminder
Grouped 2 variantsVariant mentions 12Active variants 2
Warning122026-07-03 09:25Recent (72h)KubeJobFailed (8)KubePodContainerRestartingFrequently (4)NoneNo thread note
wellbeing-mood-notifications
Grouped 2 variantsVariant mentions 12Active variants 2
Warning122026-07-03 09:25Recent (72h)KubeJobFailed (8)KubePodContainerRestartingFrequently (4)NoneNo thread note
notifications-event-manager
Grouped 6 variantsVariant mentions 22Active variants 6
Warning62026-07-02 15:42Recent (72h)KubePodContainerRestartingFrequently (4)KubeHpaMaxedOut (1)KubePodCrashLooping (1)NoneNo thread note
jobs-queue-emailWarning42026-07-03 00:57Recent (72h)KubePodContainerRestartingFrequently (4)NoneNo thread note
jobs-queue-smsWarning42026-07-03 00:57Recent (72h)KubePodContainerRestartingFrequently (4)NoneNo thread note
rooms-apiWarning42026-07-03 00:57Recent (72h)KubePodContainerRestartingFrequently (4)NoneNo thread note
accommodations-apiWarning32026-07-03 00:57Recent (72h)KubePodContainerRestartingFrequently (3)NoneNo thread note
core-scheduled-events-workerWarning32026-07-03 00:57Recent (72h)KubePodContainerRestartingFrequently (3)NoneNo thread note
send-codesWarning32026-07-03 00:57Recent (72h)KubePodContainerRestartingFrequently (3)NoneNo thread note
superset-workerWarning32026-07-02 17:51Recent (72h)KubeDeploymentRolloutStuck (1)KubeDeploymentReplicasMismatch (1)KubePodNotReady (1)NoneNo thread note
core-grafanaWarning32026-07-02 08:36Recent (72h)KubePodContainerRestartingFrequently (1)KubeDeploymentRolloutStuck (1)KubePodNotReady (1)General investigation
i am working on it currently .
admission-end-session
Grouped 3 variantsVariant mentions 61Active variants 3
Warning222026-07-01 10:17Seen this weekKubeJobFailed (22)NoneNo thread note
social-migrationWarning152026-06-29 13:23Seen this weekKubeJobNotCompleted (15)NoneNo thread note
lokiWarning12026-07-01 11:25Seen this weekKubeContainerMemoryHigh (1)NoneNo thread note
Evidence

Slack Alert Families

AlertSeverityCountLast SeenStatusThreadsTop Impacted ServicesDiscussion SignalLatest Thread Note
TraefikServiceHighErrorRateCritical82026-07-03 00:55Recent (72h)2web-80 (3)admission-api (2)library-api (2)social-api (1)Release / migration issue
admission-api is single-replica (replicas=1). During the rev 271 deploy the one pod was replaced, so there was a brief window with no ready…
Platform5xxLowVolumeCritical100Critical32026-07-03 00:47Recent (72h)1grafana (3)Release / migration issue
this was due to recent deployment on production , the 5xx errors . should be gone now , as the social migration
KubeJobFailedWarning542026-07-05 03:19Seen today0colecteaza-sms-note-abs (29)download-album (27)rezumat (26)minicrm-sync (24)admission-end-session (22)None
KubeContainerWaitingWarning152026-07-02 18:26Recent (72h)0grafana (15)None
KubePodContainerRestartingFrequentlyWarning112026-07-03 09:22Recent (72h)0core-getresponse-events-worker (8)colecteaza-sms-note-abs (7)download-album (7)rezumat (7)library-api (6)None
Platform5xxLowVolumeWarning20Warning52026-07-03 00:45Recent (72h)1grafana (5)Release / migration issue
this was due to recent deployment on production , the 5xx errors . should be gone now , as the social migration
KubePodCrashLoopingWarning42026-07-03 00:55Recent (72h)0core-getresponse-events-worker (4)notifications-event-manager (1)None
KubeDeploymentReplicasMismatchWarning42026-07-03 00:55Recent (72h)0core-getresponse-events-worker (3)superset-worker (1)None
TraefikServiceHighLatencyWarning42026-07-03 00:42Recent (72h)0web-80 (2)admission-api (2)None
KubeHpaMaxedOutWarning32026-07-03 13:43Recent (72h)0grafana (2)notifications-event-manager (1)None
KubeDeploymentRolloutStuckWarning22026-07-02 17:51Recent (72h)1core-grafana (1)superset-worker (1)General investigation
i am working on it currently .
KubePodNotReadyWarning22026-07-02 17:41Recent (72h)0core-grafana (1)superset-worker (1)None
KubeJobNotCompletedWarning152026-06-29 13:23Seen this week0social-migration (15)None
KubeContainerMemoryHighWarning12026-07-01 11:25Seen this week0grafana (1)loki (1)None
PlatformLatencyP99Warning5sWarning12026-06-29 22:56Seen this week0grafana (1)None
AlertmanagerFailedToSendAlertsWarning12026-06-29 10:34Seen this week0grafana (1)None

Status is heuristic. Slack rarely posts explicit resolutions, so “Seen today” or “Recent” means the alert family still appeared in production recently, not that it is definitely unresolved.

AWS Email Alarm Families

AWS AlarmEmailsALARMOKState FlipsFirst SeenLast SeenLatest StateStatus

“Flapping, latest OK” means the most recent email was an OK, but the alarm toggled repeatedly and is still a reliability concern.

Global Discussion-Derived Signal

Thread DateAlertSeverityServicesSignalKey Notes
2026-07-02 08:28KubeDeploymentRolloutStuckWarningcore-grafanaGeneral investigation
i am working on it currently .
2026-06-29 10:32TraefikServiceHighErrorRateCriticaladmission-apiRelease / migration issue
admission-api is single-replica (replicas=1). During the rev 271 deploy the one pod was replaced, so there was a brief window with no ready…
2026-06-29 10:23Platform5xxLowVolumeCritical100CriticalgrafanaRelease / migration issue
this was due to recent deployment on production , the 5xx errors . should be gone now , as the social migration
2026-06-29 10:23Platform5xxLowVolumeWarning20WarninggrafanaRelease / migration issue
this was due to recent deployment on production , the 5xx errors . should be gone now , as the social migration
2026-06-29 10:18TraefikServiceHighErrorRateCriticalsocial-apiRelease / migration issue
this was due to recent deployment on production , the 5xx errors . should be gone now , as the social migration