Engineering Council Test Reliability Report

Executive Snapshot

7

Daily Runs

0/7

Daily Green

14m 13s

Avg Daily Runtime

18

Smoke Attempts

17/18

Smoke Green

4m 01s

Avg Smoke Runtime

4m 21s

Median Smoke Time

0

Current Green Streak

Executive Analysis

Bottom line: the regression system is informative but not calm. The data suggest repeatable problem areas rather than random breakage, which means focused ownership should move the needle quickly.

What Matters

Daily regression passed 0 of 7 runs (0.0%), with a current green streak of 0 and a best streak of 0 in this window. The latest daily run (163673) failed, so the system is ending the week under tension rather than in a clean state. 7 failed run(s) never reached complete daily-suite counts, which points to some infrastructure or setup noise mixed into the product signal.
Smoke passed 17 of 18 attempts (94.4%) across 11 production pipelines. 1 pipeline(s) recovered on rerun, which is useful for continuity but also a sign that first-pass deploy signal is noisier than it should be. 1 failed attempt(s) never reached test execution counts at all.
Failure concentration is not random: Library has the highest strict failure ratio at 5.65%, while Library has the broadest non-pass footprint at 8.97%.
Frontend is the weakest smoke surface in this window at 10/11 green (90.9%).
Daily-suite runtime averaged 14m 13s.

Engineering Analysis

A release gate should fail loudly for product regressions and quietly for infrastructure noise. Rerun recoveries plus incomplete daily or smoke attempts suggest those two failure modes are still partially mixed together.
The failure profile is concentrated enough to act on. Library and Library are carrying the strongest signal, which means reliability work should be assigned by category ownership instead of treating the suite as one undifferentiated problem.
The broader daily suite is carrying more instability than smoke, which usually means product regressions are escaping into wider coverage areas even when the narrow deploy gate looks acceptable.

Recommended Actions

Split incomplete execution failures from real assertion failures in the report narrative. Setup breakage should stay visible, but it should not look identical to a product regression in the executive readout.
Assign one owner to Library for the next cycle and expect a short written burn-down: top failing tests, suspected root causes, flake versus regression breakdown, and what gets fixed or quarantined first.
Treat the daily regression suite like an operations queue until it is calm again: triage failures after each red run, close known-noise items fast, and avoid letting multiple unrelated red signals pile up between runs.
Put Frontend smoke under closer guardrails for the next release cycle. It is the best place to improve first-pass deploy confidence quickly.

Improvement Ideas

Introduce a small reliability budget for tests: every flaky or quarantined case needs an owner and an expiry, and the team should review that budget weekly the same way it reviews bugs or incidents.
Track first-fail to root-cause time as a core metric. Fast diagnosis is as important as raw pass rate because the practical value of a test gate depends on how quickly it helps the team recover.
Define a runtime budget per suite and require justification when test count or duration grows. Reliable feedback systems stay trusted when they remain both stable and proportionate.

Category Execution Ratios

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

Strict Failure Ratio

Share of category executions that ended in failed across all daily runs in this window.

Billing0.00%

Web0.00%

Frontend0.00%

Library5.65%

Non-pass Ratio

Share of category executions that ended in failed, pending, or skipped across all daily runs in this window.

Billing0.00%

Web0.00%

Frontend0.00%

Library8.97%

Category Aggregate Table

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

How computed

Category total executions means the sum of that category's observed test executions across every daily-suite run in the selected window.

Strict Failure Ratio = failed executions for that category divided by total executions for that category across the window.

Non-pass Ratio = (failed + pending + skipped) executions for that category divided by total executions for that category across the window.

Example: if Billing executed 800 times across the week and 2 of those executions failed, Billing strict failure ratio is 0.25%. That does not mean 0.25% of pipelines failed; it means 0.25% of observed Billing executions ended in failed.

Category	Total	Failed	Skipped	Failure Ratio	Non-pass Ratio	Runs With Failures
Billing	896	0	0	0.00%	0.00%	0
Web	0	0	0	0.00%	0.00%	7
Frontend	0	0	0	0.00%	0.00%	7
Library	602	34	20	5.65%	8.97%	2

Billing

Pend 0Skip 0Runs 0

0

0.00%

896

Web

Pend 0Skip 0Runs 7

0

0.00%

0

Frontend

Pend 0Skip 0Runs 7

0

0.00%

0

Library

Pend 0Skip 20Runs 2

34

5.65%

8.97%

602

Recent Runs

Recent Daily Suite Runs

Date	Pipeline	Suites	Status	Summary
2026-07-04 18:16	162871	BillingWebFrontendLibrary	FAILED	Total 214 \| Passed 214 \| Failed 0 \| Incomplete suite counts
2026-07-05 18:17	162873	BillingWebFrontendLibrary	FAILED	Total 214 \| Passed 214 \| Failed 0 \| Incomplete suite counts
2026-07-06 18:17	163047	BillingWebFrontendLibrary	FAILED	Total 214 \| Passed 214 \| Failed 0 \| Incomplete suite counts
2026-07-07 18:24	163228	BillingWebFrontendLibrary	FAILED	Total 214 \| Passed 187 \| Failed 17 \| Incomplete suite counts
2026-07-08 18:16	163391	BillingWebFrontendLibrary	FAILED	Total 214 \| Passed 187 \| Failed 17 \| Incomplete suite counts
2026-07-09 18:15	163540	BillingWebFrontendLibrary	FAILED	Total 214 \| Passed 214 \| Failed 0 \| Incomplete suite counts
2026-07-10 18:16	163673	BillingWebFrontendLibrary	FAILED	Total 214 \| Passed 214 \| Failed 0 \| Incomplete suite counts

2026-07-04 18:16Pipeline 162871BillingWebFrontendLibrary

FAILED

T 214 | P 214 | F 0 | Pend 0 | Incomplete

2026-07-05 18:17Pipeline 162873BillingWebFrontendLibrary

FAILED

T 214 | P 214 | F 0 | Pend 0 | Incomplete

2026-07-06 18:17Pipeline 163047BillingWebFrontendLibrary

FAILED

T 214 | P 214 | F 0 | Pend 0 | Incomplete

2026-07-07 18:24Pipeline 163228BillingWebFrontendLibrary

FAILED

T 214 | P 187 | F 17 | Pend 0 | Incomplete

2026-07-08 18:16Pipeline 163391BillingWebFrontendLibrary

FAILED

T 214 | P 187 | F 17 | Pend 0 | Incomplete

2026-07-09 18:15Pipeline 163540BillingWebFrontendLibrary

FAILED

T 214 | P 214 | F 0 | Pend 0 | Incomplete

2026-07-10 18:16Pipeline 163673BillingWebFrontendLibrary

FAILED

T 214 | P 214 | F 0 | Pend 0 | Incomplete

Recent Smoke Attempts

Date	Suite	Pipeline	Job	Status	Passed	Failed	Duration
2026-07-06 16:47	University	163038	University smoke	PASSED	60	0	4m 13s
2026-07-06 17:16	Frontend	163038	Frontend smoke	PASSED	110	0	5m 22s
2026-07-06 18:09	University	163046	University smoke	PASSED	60	0	3m 42s
2026-07-06 18:35	Frontend	163046	Frontend smoke	PASSED	110	0	5m 28s
2026-07-06 19:14	University	163052	University smoke	PASSED	60	0	3m 57s
2026-07-06 19:16	Frontend	163052	Frontend smoke	PASSED	110	0	4m 27s
2026-07-07 01:41	University	163069	University smoke	PASSED	60	0	3m 22s
2026-07-07 01:42	Frontend	163069	Frontend smoke	PASSED	110	0	4m 23s
2026-07-08 10:50	University	163259	University smoke	PASSED	60	0	3m 35s
2026-07-08 10:55	Frontend	163259	Frontend smoke	PASSED	110	0	4m 51s
2026-07-08 16:10	Frontend	163381	Frontend smoke	PASSED	110	0	4m 39s
2026-07-08 16:17	University	163381	University smoke	PASSED	60	0	3m 19s
2026-07-08 17:24	Frontend	163388	Frontend smoke	PASSED	110	0	4m 21s
2026-07-09 08:44	Frontend	163417	Frontend smoke	PASSED	110	0	4m 27s
2026-07-09 09:31	Frontend	163424	Frontend smoke	PASSED	110	0	4m 12s
2026-07-10 16:40	Frontend	163659	Frontend smoke	FAILED	n/a	n/a	0m 08s
2026-07-10 16:45	University	163659	University smoke	PASSED	60	0	3m 29s
2026-07-10 19:01	Frontend	163681	Frontend smoke	PASSED	110	0	4m 24s

2026-07-06 16:47UniversityPipeline 163038Job University smoke

PASSED

P 60 | F 0 | 4m 13s

2026-07-06 17:16FrontendPipeline 163038Job Frontend smoke

PASSED

P 110 | F 0 | 5m 22s

2026-07-06 18:09UniversityPipeline 163046Job University smoke

PASSED

P 60 | F 0 | 3m 42s

2026-07-06 18:35FrontendPipeline 163046Job Frontend smoke

PASSED

P 110 | F 0 | 5m 28s

2026-07-06 19:14UniversityPipeline 163052Job University smoke

PASSED

P 60 | F 0 | 3m 57s

2026-07-06 19:16FrontendPipeline 163052Job Frontend smoke

PASSED

P 110 | F 0 | 4m 27s

2026-07-07 01:41UniversityPipeline 163069Job University smoke

PASSED

P 60 | F 0 | 3m 22s

2026-07-07 01:42FrontendPipeline 163069Job Frontend smoke

PASSED

P 110 | F 0 | 4m 23s

2026-07-08 10:50UniversityPipeline 163259Job University smoke

PASSED

P 60 | F 0 | 3m 35s

2026-07-08 10:55FrontendPipeline 163259Job Frontend smoke

PASSED

P 110 | F 0 | 4m 51s

2026-07-08 16:10FrontendPipeline 163381Job Frontend smoke

PASSED

P 110 | F 0 | 4m 39s

2026-07-08 16:17UniversityPipeline 163381Job University smoke

PASSED

P 60 | F 0 | 3m 19s

2026-07-08 17:24FrontendPipeline 163388Job Frontend smoke

PASSED

P 110 | F 0 | 4m 21s

2026-07-09 08:44FrontendPipeline 163417Job Frontend smoke

PASSED

P 110 | F 0 | 4m 27s

2026-07-09 09:31FrontendPipeline 163424Job Frontend smoke

PASSED

P 110 | F 0 | 4m 12s

2026-07-10 16:40FrontendPipeline 163659Job Frontend smoke

FAILED

P n/a | F n/a | 0m 08s

2026-07-10 16:45UniversityPipeline 163659Job University smoke

PASSED

P 60 | F 0 | 3m 29s

2026-07-10 19:01FrontendPipeline 163681Job Frontend smoke

PASSED

P 110 | F 0 | 4m 24s

Smoke Suite Breakdown

Frontend

11 attempts across 11 pipelines

91% green

Passed10

Failed1

Incomplete1

Avg runtime4m 15s

Median passing runtime4m 27s

Pipelines11

University

7 attempts across 7 pipelines

100% green

Passed7

Failed0

Incomplete0

Avg runtime3m 40s

Median passing runtime3m 35s

Pipelines7