Skip to main content

Flaky Test Detection

TrendGate's flaky test detection helps identify unreliable tests that impact CI/CD efficiency and developer productivity.

🎲 What Are Flaky Tests?​

Flaky tests are tests that:

  • Pass and fail without code changes
  • Show inconsistent behavior
  • Reduce confidence in test suite
  • Waste CI/CD resources

🔍 Detection Methods​

Statistical Analysis​

TrendGate uses multiple signals:

flaky_detection:
  methods:
    - failure_rate_analysis    # >5% random failures
    - pattern_recognition      # Time-based failures
    - environmental_variance   # Platform-specific issues
    - execution_time_variance  # Inconsistent duration

Confidence Scoring​

Each test gets a flakiness score:

  • 🟢 0-20%: Reliable
  • 🟡 20-50%: Potentially flaky
  • 🟠 50-80%: Likely flaky
  • 🔴 80-100%: Confirmed flaky

📊 Flaky Test Dashboard​

Overview Metrics​

  • Total flaky tests identified
  • Flakiness trend over time
  • CI time wasted on reruns
  • Team/module breakdown

Detailed Analysis​

For each flaky test:

{
  "test": "UserService.login should handle timeout",
  "flakiness_score": 67,
  "failure_pattern": "time_based",
  "environments_affected": ["linux", "windows"],
  "first_detected": "2024-01-15",
  "failure_rate": "12%",
  "suggested_action": "Add retry logic or increase timeout"
}

🛠️ Management Strategies​

Quarantine Process​

quarantine_policy:
  auto_quarantine:
    threshold: 50  # Flakiness score
    grace_period: 3  # Days before quarantine
    
  quarantine_actions:
    - exclude_from_ci: true
    - create_ticket: true
    - notify_owner: true

Fix Prioritization​

TrendGate prioritizes based on:

  1. Impact: Tests blocking critical paths
  2. Frequency: How often they fail
  3. Cost: CI/CD time wasted
  4. Coverage: Important code areas

🔧 Common Causes & Fixes​

1. Timing Issues​

// ❌ Flaky
await sleep(1000);
expect(element).toBeVisible();

// ✅ Reliable
await waitFor(() => {
  expect(element).toBeVisible();
}, { timeout: 5000 });

2. Test Interdependencies​

// ❌ Flaky - depends on test order
test('should have user', () => {
  expect(globalUser).toBeDefined();
});

// ✅ Reliable - self-contained
test('should have user', () => {
  const user = await createTestUser();
  expect(user).toBeDefined();
});

3. External Dependencies​

// ❌ Flaky - real API call
const data = await fetch('https://api.external.com');

// ✅ Reliable - mocked
const data = await mockApi.getData();

📈 Tracking Progress​

Flakiness Reduction Metrics​

  • Weekly flaky test count
  • Mean time to fix
  • Reintroduction rate
  • CI stability improvement

Success Stories​

Track and share wins:

  • "Reduced flaky tests by 75% in Q3"
  • "Saved 10 hours/week of CI time"
  • "Improved developer confidence"

🚀 Prevention Best Practices​

Code Review Checklist​

  • No hardcoded delays
  • Proper test isolation
  • Deterministic assertions
  • Resource cleanup
  • Mock external services

Testing Guidelines​

  1. Use proper wait strategies
  2. Isolate test data
  3. Mock time-dependent code
  4. Handle async operations correctly
  5. Clean up after tests

🤖 Automation​

Auto-Retry Configuration​

retry_policy:
  flaky_tests:
    max_retries: 3
    only_on_ci: true
    
  known_flaky:
    skip_in_pr: true
    run_in_scheduled: true

AI-Powered Suggestions​

TrendGate analyzes patterns and suggests fixes:

  • "This test fails 80% of the time on Windows - check file path handling"
  • "Failures correlate with high CPU load - add resource wait"
  • "Similar tests in this file are stable - compare implementations"

Start eliminating flaky tests and stabilize your CI/CD pipeline today!