Engineering Council Test Reliability Report

Scope aligned with Slack channel #dezvoltare, covering 2026-08-22 07:00 to 2026-08-29 07:00. Metrics and timings are sourced from GitLab pipelines, jobs, and test-report artifacts for the daily 6 PM regression suite and the production smoke suite. Trend charts use daily buckets across this window.

Executive Snapshot

6
Daily Runs
0/6
Daily Green
18m 29s
Avg Daily Runtime
5
Smoke Attempts
4/5
Smoke Green
4m 16s
Avg Smoke Runtime
5m 20s
Median Smoke Time
0
Current Green Streak

Executive Analysis

Bottom line: the weakest link is smoke reliability, not test speed. The suite can still provide signal, but deploy confidence is being taxed by failed or noisy smoke attempts.

What Matters

  • Daily regression passed 0 of 6 runs (0.0%), with a current green streak of 0 and a best streak of 0 in this window. The latest daily run (168126) failed, so the system is ending the week under tension rather than in a clean state. 6 failed run(s) never reached complete daily-suite counts, which points to some infrastructure or setup noise mixed into the product signal.
  • Smoke passed 4 of 5 attempts (80.0%) across 3 production pipelines. 1 pipeline(s) recovered on rerun, which is useful for continuity but also a sign that first-pass deploy signal is noisier than it should be. 1 failed attempt(s) never reached test execution counts at all.
  • Failure concentration is not random: Billing has the highest strict failure ratio at 0.46%, while Billing has the broadest non-pass footprint at 0.46%.
  • Frontend is the weakest smoke surface in this window at 3/4 green (75.0%).
  • Daily-suite runtime averaged 18m 29s.

Engineering Analysis

  • A release gate should fail loudly for product regressions and quietly for infrastructure noise. Rerun recoveries plus incomplete daily or smoke attempts suggest those two failure modes are still partially mixed together.
  • The failure profile is concentrated enough to act on. Billing and Billing are carrying the strongest signal, which means reliability work should be assigned by category ownership instead of treating the suite as one undifferentiated problem.
  • The broader daily suite is carrying more instability than smoke, which usually means product regressions are escaping into wider coverage areas even when the narrow deploy gate looks acceptable.

Recommended Actions

  • Split incomplete execution failures from real assertion failures in the report narrative. Setup breakage should stay visible, but it should not look identical to a product regression in the executive readout.
  • Assign one owner to Billing for the next cycle and expect a short written burn-down: top failing tests, suspected root causes, flake versus regression breakdown, and what gets fixed or quarantined first.
  • Treat the daily regression suite like an operations queue until it is calm again: triage failures after each red run, close known-noise items fast, and avoid letting multiple unrelated red signals pile up between runs.
  • Put Frontend smoke under closer guardrails for the next release cycle. It is the best place to improve first-pass deploy confidence quickly.

Improvement Ideas

  • Introduce a small reliability budget for tests: every flaky or quarantined case needs an owner and an expiry, and the team should review that budget weekly the same way it reviews bugs or incidents.
  • Track first-fail to root-cause time as a core metric. Fast diagnosis is as important as raw pass rate because the practical value of a test gate depends on how quickly it helps the team recover.
  • Define a runtime budget per suite and require justification when test count or duration grows. Reliable feedback systems stay trusted when they remain both stable and proportionate.

Category Execution Ratios

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

Daily Daily Suite Status0000108-2308-2408-2508-2608-2708-28
Daily Smoke Attempts0123408-2308-2408-2508-2608-2708-28
Daily Average Daily Suite Runtime16m 01s18m 07s20m 13s22m 19s24m 26s08-2308-2408-2508-2608-2708-28
Daily Average Smoke Runtime0m 00s1m 11s2m 23s3m 34s4m 46s08-2308-2408-2508-2608-2708-28
Daily Suite Total Test Growth (Recent 6 Runs)23223223223223308-2308-2408-2508-2608-2708-28
Smoke Suite Total Test Growth (Latest Run Per Day)
FrontendUniversity
60728597110Frontend 08-26: 110Frontend 08-27: 110University 08-27: 6008-2608-27

Category Aggregate Table

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

CategoryTotalFailedPendingSkippedFailure RatioNon-pass RatioRuns With Failures
Billing8764000.46%0.46%2
Web00000.00%0.00%6
Frontend00000.00%0.00%6
Library5160000.00%0.00%0
CatFailF%NP%Tot
Billing
Pend 0Skip 0Runs 2
4
0.46%
0.46%
876
Web
Pend 0Skip 0Runs 6
0
0.00%
0.00%
0
Frontend
Pend 0Skip 0Runs 6
0
0.00%
0.00%
0
Library
Pend 0Skip 0Runs 0
0
0.00%
0.00%
516

Recent Runs

Recent Daily Suite Runs

DatePipelineSuitesStatusSummary
2026-08-23 18:19167456BillingWebFrontendLibraryFAILEDTotal 232 | Passed 232 | Failed 0 | Incomplete suite counts
2026-08-24 18:21167600BillingWebFrontendLibraryFAILEDTotal 232 | Passed 232 | Failed 0 | Incomplete suite counts
2026-08-25 18:20167721BillingWebFrontendLibraryFAILEDTotal 232 | Passed 231 | Failed 1 | Incomplete suite counts
2026-08-26 18:21167871BillingWebFrontendLibraryFAILEDTotal 232 | Passed 229 | Failed 3 | Incomplete suite counts
2026-08-27 18:27168026BillingWebFrontendLibraryFAILEDTotal 232 | Passed 232 | Failed 0 | Incomplete suite counts
2026-08-28 18:19168126BillingWebFrontendLibraryFAILEDTotal 232 | Passed 232 | Failed 0 | Incomplete suite counts
2026-08-23 18:19Pipeline 167456BillingWebFrontendLibrary
FAILED
T 232 | P 232 | F 0 | Pend 0 | Incomplete
2026-08-24 18:21Pipeline 167600BillingWebFrontendLibrary
FAILED
T 232 | P 232 | F 0 | Pend 0 | Incomplete
2026-08-25 18:20Pipeline 167721BillingWebFrontendLibrary
FAILED
T 232 | P 231 | F 1 | Pend 0 | Incomplete
2026-08-26 18:21Pipeline 167871BillingWebFrontendLibrary
FAILED
T 232 | P 229 | F 3 | Pend 0 | Incomplete
2026-08-27 18:27Pipeline 168026BillingWebFrontendLibrary
FAILED
T 232 | P 232 | F 0 | Pend 0 | Incomplete
2026-08-28 18:19Pipeline 168126BillingWebFrontendLibrary
FAILED
T 232 | P 232 | F 0 | Pend 0 | Incomplete

Recent Smoke Attempts

DateSuitePipelineJobStatusPassedFailedDuration
2026-08-26 11:31Frontend167761Frontend smokePASSED11004m 45s
2026-08-27 15:29Frontend168001Frontend smokeFAILEDn/an/a0m 13s
2026-08-27 16:02Frontend168001Frontend smokePASSED11006m 34s
2026-08-27 16:15University168001University smokePASSED6003m 51s
2026-08-27 17:05Frontend168022Frontend smokePASSED11005m 55s

Smoke Suite Breakdown

Frontend
4 attempts across 3 pipelines
75% green
Passed3
Failed1
Incomplete1
Avg runtime4m 22s
Median passing runtime5m 55s
Pipelines3
University
1 attempts across 1 pipeline
100% green
Passed1
Failed0
Incomplete0
Avg runtime3m 51s
Median passing runtime3m 51s
Pipelines1
Generated from GitLab project adservio/helm2. Times are shown in Europe/Bucharest. Daily-suite runtime is measured from GitLab pipeline and job timestamps. Category counts come from GitLab test-report JSON artifacts, with job-trace fallback when older artifacts have expired.