Engineering Council Test Reliability Report

Scope aligned with Slack channel #dezvoltare, covering 2026-05-09 07:00 to 2026-05-16 07:00. Metrics and timings are sourced from GitLab pipelines, jobs, and test-report artifacts for the daily 6 PM regression suite and the production smoke suite. Trend charts use daily buckets across this window.

Executive Snapshot

4
Daily Runs
0/4
Daily Green
20m 05s
Avg Daily Runtime
12
Smoke Attempts
11/12
Smoke Green
3m 04s
Avg Smoke Runtime
3m 07s
Median Smoke Time
0
Current Green Streak

Executive Analysis

Bottom line: the regression system is informative but not calm. The data suggest repeatable problem areas rather than random breakage, which means focused ownership should move the needle quickly.

What Matters

  • Daily regression passed 0 of 4 runs (0.0%), with a current green streak of 0 and a best streak of 0 in this window. The latest daily run (156175) failed, so the system is ending the week under tension rather than in a clean state.
  • Smoke passed 11 of 12 attempts (91.7%) across 8 production pipelines. 1 pipeline(s) recovered on rerun, which is useful for continuity but also a sign that first-pass deploy signal is noisier than it should be.
  • Failure concentration is not random: Frontend has the highest strict failure ratio at 9.25%, while Frontend has the broadest non-pass footprint at 27.14%.
  • University is the weakest smoke surface in this window at 3/4 green (75.0%).
  • Daily-suite runtime averaged 20m 05s.

Engineering Analysis

  • A release gate should fail loudly for product regressions and quietly for infrastructure noise. Rerun recoveries plus incomplete daily or smoke attempts suggest those two failure modes are still partially mixed together.
  • The failure profile is concentrated enough to act on. Frontend and Frontend are carrying the strongest signal, which means reliability work should be assigned by category ownership instead of treating the suite as one undifferentiated problem.
  • The broader daily suite is carrying more instability than smoke, which usually means product regressions are escaping into wider coverage areas even when the narrow deploy gate looks acceptable.

Recommended Actions

  • Assign one owner to Frontend for the next cycle and expect a short written burn-down: top failing tests, suspected root causes, flake versus regression breakdown, and what gets fixed or quarantined first.
  • Treat the daily regression suite like an operations queue until it is calm again: triage failures after each red run, close known-noise items fast, and avoid letting multiple unrelated red signals pile up between runs.
  • Put University smoke under closer guardrails for the next release cycle. It is the best place to improve first-pass deploy confidence quickly.

Improvement Ideas

  • Introduce a small reliability budget for tests: every flaky or quarantined case needs an owner and an expiry, and the team should review that budget weekly the same way it reviews bugs or incidents.
  • Track first-fail to root-cause time as a core metric. Fast diagnosis is as important as raw pass rate because the practical value of a test gate depends on how quickly it helps the team recover.
  • Define a runtime budget per suite and require justification when test count or duration grows. Reliable feedback systems stay trusted when they remain both stable and proportionate.

Category Execution Ratios

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

Daily Daily Suite Status0000105-1205-1305-1405-15
Daily Smoke Attempts0123505-1205-1305-1405-15
Daily Average Daily Suite Runtime8m 13s12m 24s16m 36s20m 47s24m 58s05-1205-1305-1405-15
Daily Average Smoke Runtime0m 00s0m 51s1m 43s2m 34s3m 26s05-1205-1305-1405-15
Daily Suite Total Test Growth (Recent 4 Runs)1255125512551255125605-1205-1305-1405-15
Smoke Suite Total Test Growth (Latest Run Per Day)
FrontendUniversity
60728597110Frontend 05-10: 110Frontend 05-11: 110Frontend 05-12: 110Frontend 05-14: 110Frontend 05-15: 110University 05-12: 60University 05-14: 6005-1005-1105-1205-1405-15

Category Aggregate Table

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

CategoryTotalFailedPendingSkippedFailure RatioNon-pass RatioRuns With Failures
Billing432100982.31%25.00%1
Web312013906414.46%25.00%1
Frontend112410402019.25%27.14%4
Library344230636.69%25.00%1
CatFailF%NP%Tot
Billing
Pend 0Skip 98Runs 1
10
2.31%
25.00%
432
Web
Pend 0Skip 641Runs 1
139
4.46%
25.00%
3120
Frontend
Pend 0Skip 201Runs 4
104
9.25%
27.14%
1124
Library
Pend 0Skip 63Runs 1
23
6.69%
25.00%
344

Recent Runs

Recent Daily Suite Runs

DatePipelineSuitesStatusSummary
2026-05-12 18:26155661BillingWebFrontendLibraryFAILEDTotal 1255 | Passed 1244 | Failed 11
2026-05-13 18:27155895BillingWebFrontendLibraryFAILEDTotal 1255 | Passed 1244 | Failed 11
2026-05-14 18:28155995BillingWebFrontendLibraryFAILEDTotal 1255 | Passed 1253 | Failed 2
2026-05-15 18:11156175BillingWebFrontendLibraryFAILEDTotal 1255 | Passed 0 | Failed 252
2026-05-12 18:26Pipeline 155661BillingWebFrontendLibrary
FAILED
T 1255 | P 1244 | F 11 | Pend 0
2026-05-13 18:27Pipeline 155895BillingWebFrontendLibrary
FAILED
T 1255 | P 1244 | F 11 | Pend 0
2026-05-14 18:28Pipeline 155995BillingWebFrontendLibrary
FAILED
T 1255 | P 1253 | F 2 | Pend 0
2026-05-15 18:11Pipeline 156175BillingWebFrontendLibrary
FAILED
T 1255 | P 0 | F 252 | Pend 0

Recent Smoke Attempts

DateSuitePipelineJobStatusPassedFailedDuration
2026-05-10 15:16Frontend155320Frontend smokePASSED11003m 59s
2026-05-10 15:44Frontend155324Frontend smokePASSED11003m 40s
2026-05-11 11:13Frontend155377Frontend smokePASSED11003m 04s
2026-05-11 16:34Frontend155489Frontend smokePASSED11003m 10s
2026-05-12 16:02University155636University smokePASSED6002m 14s
2026-05-12 16:07Frontend155636Frontend smokePASSED11003m 07s
2026-05-14 07:48University155903University smokePASSED6002m 13s
2026-05-14 07:53Frontend155903Frontend smokePASSED11003m 07s
2026-05-14 14:26University155955University smokeFAILED5733m 11s
2026-05-14 14:30Frontend155955Frontend smokePASSED11003m 11s
2026-05-14 16:32University155955University smokePASSED6002m 25s
2026-05-15 12:20Frontend156034Frontend smokePASSED11003m 26s

Smoke Suite Breakdown

Frontend
8 attempts across 8 pipelines
100% green
Passed8
Failed0
Incomplete0
Avg runtime3m 20s
Median passing runtime3m 11s
Pipelines8
University
4 attempts across 3 pipelines
75% green
Passed3
Failed1
Incomplete0
Avg runtime2m 31s
Median passing runtime2m 14s
Pipelines3
Generated from GitLab project adservio/helm2. Times are shown in Europe/Bucharest. Daily-suite runtime is measured from GitLab pipeline and job timestamps. Category counts come from GitLab test-report JSON artifacts, with job-trace fallback when older artifacts have expired.