Ops Briefing Surface

Production Reliability Dashboard

Generated 2026-06-21 22:54 for 2026-06-14 07:00 to 2026-06-21 07:00 from Pingdom checks, Slack #_alerts_prod, and AWS SNS alerts.

All sources Pingdom customer checks Slack alert families AWS alarm emails
Email-confirmed customer incidents0Pingdom down/slow events confirmed by inbox alertsObserved in Pingdom: 0
Impacted services35Mapped from Slack and Pingdom evidence
AWS alarms in ALARM2Still alarming at window end
Latest observed signal2026-06-21 05:43Most recent cross-source activity
Run notes

Data source availability

Some enrichments were unavailable in this run; drilldowns below stay focused on captured evidence.

Executive summary

What needs attention

Bottom line: application-level critical paths are present and catalog database alarms still look active.

Pingdom customer impact

External signal
No criticalActive: 0Total seen: 0

No Pingdom checks were available in this window.

No active issue listed in this category.

Slack impacted services

Application signal
5 criticalActive: 35Total seen: 35

5 critical, 30 non-critical active item(s).

AWS alarms

Infrastructure signal
2 criticalActive: 2Total seen: 12

2 critical, 0 non-critical active item(s).

What to do next

  2. NextTreat critical services web-80, grafana, uni-api, core-grafana-80 as the primary application investigation set

    Treat critical services web-80, grafana, uni-api, core-grafana-80 as the primary application investigation set. Use TraefikServiceHighErrorRate as the leading category, reproduce the failing paths on web-80, grafana, uni-api, core-grafana-80, compare against the latest deploy or config change, and do not close it until both error rate and latency flatten.

    Executive recommendation
  3. ThenReduce the operational drag on download-album by separating repeated symptom alerts from the underlying workload failure

    Reduce the operational drag on download-album by separating repeated symptom alerts from the underlying workload failure. Either eliminate the recurrent fault or retune the alert once the failure mode is understood.

    Executive recommendation
Global Evidence Explorer

Global Evidence Explorer

Report-wide charts and tables stay here, separate from the active investigation scope.

Application + Infrastructure Alerts by Day

Pingdom latency + downtime by source

Customer View

Pingdom Checks

Pingdom CheckStatusEventsDowntimeLast SeenLikely ServicesCorrelated Evidence

Pingdom rows show externally visible signal first. The correlated evidence column helps tie the failing check back to services, Slack alert families, or AWS alarms when those links exist.

Application View

Slack Impacted Service / Resource View

This view attributes alerts to the workload or resource named in the alert text. Grafana, Loki, and Tempo are treated as observability components and are excluded when a more specific impacted target is also present.

Impacted Service / ResourceHighest SeverityCountLast SeenStatusTop Alert TypesDiscussion SignalLatest Thread Note
grafana
Latest alertLatest thread
Critical672026-06-19 14:18Recent (72h)PlatformLatencyP95Critical1s (7)Platform5xxLowVolumeCritical100 (4)KubeContainerMemoryHigh (15)NodeHighNumberConntrackEntriesUsed (7)KubePersistentVolumeFillingUp (6)Observability storage
codex
*Alert:* `Platform5xxRatioCritical3pct` (critical) | *When:* 2026-06-16 06:53 Europe/Bucharest | | *Service:* `grafana` | *TL;DR:* `grafana…
ThreadAlert
web-80
Latest alertLatest thread
Critical312026-06-18 16:09Recent (72h)TraefikServiceHighErrorRate (14)TraefikServiceHighLatency (17)Observability storage
codex
*Alert:* `TraefikServiceHighErrorRate` (critical) | *When:* 2026-06-16 05:55 Europe/Bucharest | | *Service:* `web-80` | *TL;DR:* `web-80` c…
ThreadAlert
docgen2-api
Latest alert
Critical282026-06-19 09:05Recent (72h)TraefikServiceHighErrorRate (4)CPUThrottlingHigh (13)KubeHpaMaxedOut (11)NoneNo thread note
core-grafana-80
Latest alert
Critical32026-06-19 10:43Recent (72h)TraefikServiceHighErrorRate (3)NoneNo thread note
uni-api
Grouped 2 variantsVariant mentions 13Active variants 2
Latest alertLatest thread
Critical142026-06-17 18:34Seen this weekTraefikServiceHighErrorRate (4)TraefikServiceHighLatency (8)KubePodCrashLooping (1)KubeDeploymentReplicasMismatch (1)Release / migration issue
Thread summary · codex
*Alert:* `TraefikServiceHighErrorRate` (critical) | *When:* 2026-06-16 16:38 Europe/Bucharest | | *Service:* `uni-api` | *TL;DR:* `uni-api`… | *Fix now:* Check the latest deployed revision for `/api/v2/uni/student-studies/students/1882/disciplines?semester=1` and rollback or hotfix…
ThreadAlert
download-album
Grouped 3 variantsVariant mentions 20Active variants 3
Latest alertLatest thread
Warning202026-06-21 05:43Seen todayKubeJobFailed (20)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
ai-api
Latest alert
Warning172026-06-19 17:15Recent (72h)TraefikServiceHighLatency (17)NoneNo thread note
loki
Latest alert
Warning62026-06-19 14:18Recent (72h)KubePersistentVolumeFillingUp (6)NoneNo thread note
admission-api
Latest alert
Warning32026-06-18 11:26Recent (72h)TraefikServiceHighLatency (3)NoneNo thread note
subscriptions-api
Grouped 2 variantsVariant mentions 14Active variants 2
Latest alert
Warning212026-06-16 01:43Seen this weekTraefikServiceHighLatency (13)KubeDeploymentReplicasMismatch (7)KubePodCrashLooping (1)NoneNo thread note
social-api
Latest alert
Warning172026-06-15 14:11Seen this weekTraefikServiceHighLatency (17)NoneNo thread note
core-getresponse-events-worker
Latest alert
Warning132026-06-16 07:03Seen this weekKubeDeploymentReplicasMismatch (9)KubePodCrashLooping (4)NoneNo thread note
notifications-event-manager
Latest alert
Warning122026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (6)KubeHpaMaxedOut (5)KubePodCrashLooping (1)NoneNo thread note
colecteaza-sms-note-abs
Latest alertLatest thread
Warning112026-06-16 15:36Seen this weekKubeJobFailed (11)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
service-websocket
Latest alert
Warning112026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (8)KubePodCrashLooping (3)NoneNo thread note
rezumat
Latest alertLatest thread
Warning102026-06-16 15:36Seen this weekKubeJobFailed (10)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
publish-results
Latest alertLatest thread
Warning92026-06-16 15:36Seen this weekKubeJobFailed (9)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
update-recurenta
Latest alertLatest thread
Warning92026-06-16 15:36Seen this weekKubeJobFailed (9)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
attendance-register-missed-attendance
Latest alertLatest thread
Warning82026-06-16 15:36Seen this weekKubeJobFailed (8)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
stats-active-users
Latest alertLatest thread
Warning82026-06-16 15:36Seen this weekKubeJobFailed (8)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
stats-school-students
Latest alertLatest thread
Warning72026-06-16 15:36Seen this weekKubeJobFailed (7)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
library-api
Latest alert
Warning72026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (6)KubePodCrashLooping (1)NoneNo thread note
notifications-push-sender-worker
Latest alert
Warning72026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (6)KubePodCrashLooping (1)NoneNo thread note
service-av
Latest alert
Warning72026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (6)KubePodCrashLooping (1)NoneNo thread note
send-codes
Latest alertLatest thread
Warning62026-06-16 15:36Seen this weekKubeJobFailed (6)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
notifications-events-worker
Latest alert
Warning62026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (5)KubePodCrashLooping (1)NoneNo thread note
subscriptions-school-stats-worker
Latest alert
Warning62026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (5)KubePodCrashLooping (1)NoneNo thread note
generate-invoices
Latest alertLatest thread
Warning52026-06-16 15:36Seen this weekKubeJobFailed (5)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
service-fetgenerator
Grouped 2 variantsVariant mentions 2Active variants 2
Latest alert
Warning52026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (4)KubePodCrashLooping (1)NoneNo thread note
subscriptions-assign-worker
Latest alert
Warning52026-06-16 01:43Seen this weekKubeDeploymentReplicasMismatch (4)KubePodCrashLooping (1)NoneNo thread note
rooms-api
Latest alert
Warning52026-06-15 11:29Seen this weekTraefikServiceHighLatency (5)NoneNo thread note
web
Latest alert
Warning32026-06-15 11:36Seen this weekKubeDeploymentReplicasMismatch (3)NoneNo thread note
social-local-cache
Latest alert
Warning12026-06-16 15:30Seen this weekKubeJobNotCompleted (1)NoneNo thread note
metrics-server
Latest alert
Warning12026-06-15 21:00Seen this weekKubeAggregatedAPIDown (1)NoneNo thread note
billing-api
Latest alert
Warning12026-06-15 14:11Seen this weekTraefikServiceHighLatency (1)NoneNo thread note
Evidence

Slack Alert Families

AlertSeverityCountLast SeenStatusThreadsTop Impacted ServicesDiscussion SignalLatest Thread Note
TraefikServiceHighErrorRate
Latest alertLatest thread
Critical252026-06-19 10:43Recent (72h)2web-80 (14)docgen2-api (4)uni-api (4)core-grafana-80 (3)Observability storageRelease / migration issue
Thread summary · codex
*Alert:* `TraefikServiceHighErrorRate` (critical) | *When:* 2026-06-16 16:38 Europe/Bucharest | | *Service:* `uni-api` | *TL;DR:* `uni-api`… | *Fix now:* Check the latest deployed revision for `/api/v2/uni/student-studies/students/1882/disciplines?semester=1` and rollback or hotfix…
ThreadAlert
Platform5xxRatioCritical3pct
Latest alertLatest thread
Critical22026-06-18 16:18Recent (72h)1grafana (2)Observability storage
codex
*Alert:* `Platform5xxRatioCritical3pct` (critical) | *When:* 2026-06-16 06:53 Europe/Bucharest | | *Service:* `grafana` | *TL;DR:* `grafana…
ThreadAlert
PlatformLatencyP95Critical1s
Latest alert
Critical72026-06-15 14:01Seen this week0grafana (7)None
Alert
Platform5xxLowVolumeCritical100
Latest alertLatest thread
Critical42026-06-16 05:48Seen this week1grafana (4)Observability storage
codex
*Alert:* `Platform5xxLowVolumeCritical100` (critical) | *When:* 2026-06-16 05:48 Europe/Bucharest | | *Service:* `grafana` | *TL;DR:* `graf…
ThreadAlert
KubeJobFailed
Latest alertLatest thread
Warning272026-06-21 05:43Seen today1download-album (20)colecteaza-sms-note-abs (11)rezumat (10)publish-results (9)update-recurenta (9)General investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
TraefikServiceHighLatency
Latest alert
Warning262026-06-19 17:15Recent (72h)0social-api (17)web-80 (17)ai-api (17)subscriptions-api (13)uni-api (8)None
Alert
KubeHpaMaxedOut
Latest alert
Warning222026-06-18 13:21Recent (72h)0docgen2-api (11)grafana (6)notifications-event-manager (5)None
Alert
KubeContainerMemoryHigh
Latest alert
Warning152026-06-18 17:22Recent (72h)0grafana (15)None
Alert
CPUThrottlingHigh
Latest alert
Warning132026-06-19 09:05Recent (72h)0docgen2-api (13)None
Alert
NodeHighNumberConntrackEntriesUsed
Latest alert
Warning72026-06-19 09:32Recent (72h)0grafana (7)None
Alert
KubePersistentVolumeFillingUp
Latest alert
Warning62026-06-19 14:18Recent (72h)0grafana (6)loki (6)None
Alert
NodeSystemSaturation
Latest alert
Warning32026-06-19 01:24Recent (72h)0grafana (3)None
Alert
Platform5xxRatioWarning1pct
Latest alert
Warning22026-06-18 16:18Recent (72h)0grafana (2)None
Alert
NodeDiskIOSaturation
Latest alert
Warning12026-06-19 01:40Recent (72h)0grafana (1)None
Alert
KubeDeploymentReplicasMismatch
Latest alert
Warning142026-06-17 18:28Seen this week0core-getresponse-events-worker (9)service-websocket (8)subscriptions-api (7)library-api (6)notifications-event-manager (6)None
Alert
KubePodCrashLooping
Latest alert
Warning62026-06-17 18:29Seen this week0core-getresponse-events-worker (4)service-websocket (3)library-api (1)notifications-event-manager (1)notifications-events-worker (1)None
Alert
PlatformLatencyP95Warning400ms
Latest alert
Warning62026-06-15 14:01Seen this week0grafana (6)None
Alert
PlatformLatencyP99Warning5s
Latest alert
Warning52026-06-15 14:07Seen this week0grafana (5)None
Alert
TargetDown
Latest alert
Warning22026-06-15 09:31Seen this week0grafana (2)None
Alert
KubeJobNotCompleted
Latest alert
Warning12026-06-16 15:30Seen this week0social-local-cache (1)None
Alert
Platform5xxLowVolumeWarning20
Latest alert
Warning12026-06-16 04:11Seen this week0grafana (1)None
Alert
KubeAggregatedAPIDown
Latest alert
Warning12026-06-15 21:00Seen this week0metrics-server (1)None
Alert

Status is heuristic. Slack rarely posts explicit resolutions, so “Seen today” or “Recent” means the alert family still appeared in production recently, not that it is definitely unresolved.

AWS Email Alarm Families

AWS AlarmEmailsALARMOKState FlipsFirst SeenLast SeenLatest StateStatus
adservio-rds-mysql-catalog2-memory-low321715302026-06-15 06:222026-06-21 03:59ALARMStill alarming
adservio-rds-mysql-catalog2-swap-high32122026-06-15 06:332026-06-17 06:42ALARMStill alarming
adservio-root-account-usage42232026-06-16 02:472026-06-16 15:07OKLatest OK
adservio-rds-mysql-catalog2-storage-low42232026-06-15 23:552026-06-16 06:00OKLatest OK
adservio-rds-mysql-catalog3-cpu-high10102026-06-15 20:052026-06-15 20:05OKLatest OK
adservio-rds-mysql-catalog3-read-latency-high10102026-06-15 20:052026-06-15 20:05OKLatest OK
adservio-rds-mysql-catalog3-memory-low10102026-06-15 20:052026-06-15 20:05OKLatest OK
adservio-rds-mysql-catalog3-storage-low10102026-06-15 20:052026-06-15 20:05OKLatest OK
adservio-rds-mysql-catalog3-write-latency-high10102026-06-15 20:052026-06-15 20:05OKLatest OK
adservio-rds-mysql-catalog3-connections-high10102026-06-15 20:042026-06-15 20:04OKLatest OK
adservio-rds-mysql-catalog3-disk-queue-high10102026-06-15 20:042026-06-15 20:04OKLatest OK
adservio-rds-mysql-catalog3-swap-high10102026-06-15 20:042026-06-15 20:04OKLatest OK

“Flapping, latest OK” means the most recent email was an OK, but the alarm toggled repeatedly and is still a reliability concern.

Global Discussion-Derived Signal

Thread DateAlertSeverityServicesSignalKey Notes
2026-06-16 16:38TraefikServiceHighErrorRate
ThreadAlert
Criticaluni-apiRelease / migration issue
Thread summary · codex
*Alert:* `TraefikServiceHighErrorRate` (critical) | *When:* 2026-06-16 16:38 Europe/Bucharest | | *Service:* `uni-api` | *TL;DR:* `uni-api`… | *Fix now:* Check the latest deployed revision for `/api/v2/uni/student-studies/students/1882/disciplines?semester=1` and rollback or hotfix…
ThreadAlert
2026-06-16 11:31KubeJobFailed
ThreadAlert
Warningattendance-register-missed-attendance, colecteaza-sms-note-abs, download-album, generate-invoicesGeneral investigation
Rojan Shrestha
These are failed jobs which can be cleaned up from cluster , removed these failed jobs .
ThreadAlert
2026-06-16 06:53Platform5xxRatioCritical3pct
ThreadAlert
CriticalgrafanaObservability storage
codex
*Alert:* `Platform5xxRatioCritical3pct` (critical) | *When:* 2026-06-16 06:53 Europe/Bucharest | | *Service:* `grafana` | *TL;DR:* `grafana…
ThreadAlert
2026-06-16 05:55TraefikServiceHighErrorRate
ThreadAlert
Criticalweb-80Observability storage
codex
*Alert:* `TraefikServiceHighErrorRate` (critical) | *When:* 2026-06-16 05:55 Europe/Bucharest | | *Service:* `web-80` | *TL;DR:* `web-80` c…
ThreadAlert
2026-06-16 05:48Platform5xxLowVolumeCritical100
ThreadAlert
CriticalgrafanaObservability storage
codex
*Alert:* `Platform5xxLowVolumeCritical100` (critical) | *When:* 2026-06-16 05:48 Europe/Bucharest | | *Service:* `grafana` | *TL;DR:* `graf…
ThreadAlert