Coding DebuggingIntermediate30 minSaves 30 minutes

Diagnosing Flaky CI Tests in Parallel Execution Environments

Engineers battling inconsistent CI failures can use this playbook to diagnose and fix timing, shared-state, and ordering issues in parallel test suites, improving build reliability.

This prompt helps engineers debug flaky CI tests that fail inconsistently under parallel execution. It generates a comprehensive playbook outlining common hypotheses for timing, shared-state, and test ordering issues, along with specific checks, commands, and fixes to stabilize your test suite.

ChatGPT Gemini Claude

READY-TO-USE PROMPT

Copy Prompt

prompt.txt

As an experienced software engineer and CI/CD specialist, your task is to assist in debugging flaky test suites that exhibit inconsistent failures when run in parallel within a continuous integration environment.

### Context
I am experiencing intermittent failures in my test suite when it runs on our CI system. These failures do not consistently reproduce locally or when tests are run sequentially. The primary suspect is parallelism, introducing race conditions, shared state contamination, or non-deterministic test ordering. The goal is to generate a comprehensive diagnostic playbook to systematically identify and resolve these issues.

My test suite is described as: `{{test_suite_description}}`
My CI environment details are: `{{ci_environment_details}}`

### Task
Generate a structured debug playbook focused on identifying and resolving flakiness due to parallel execution. The playbook should cover the following common categories of issues: timing dependencies, shared state contamination, and non-deterministic test ordering. For each category, provide a symptom, a list of hypotheses, specific checks (including shell commands or code snippets where applicable), likely fixes, and verification steps.

### Constraints
*   Focus exclusively on issues related to parallel test execution. Do not address unrelated test failures (e.g., incorrect logic in a single test).
*   For 'Checks', provide actionable commands or specific code patterns to look for.
*   For 'Likely Fixes', suggest concrete strategies or code modifications.
*   The output must follow the specified structure precisely.
*   Assume a Linux-based CI environment for shell commands.

### Output Format
Provide the playbook structured as follows:

**Debug Playbook for Flaky CI Tests in Parallel Environments**

#### 1. Timing Dependencies
*   **Symptom**: Tests intermittently fail with timeouts, assertion failures on eventual consistency, or resource unavailability.
*   **Hypotheses**:
    *   Hypothesis A: Description of timing-related hypothesis.
    *   Hypothesis B: Description of timing-related hypothesis.
*   **Checks**:
    *   Check 1: `command or code snippet` (Explanation of what this checks).
    *   Check 2: `command or code snippet` (Explanation).
*   **Likely Fixes**:
    *   Fix 1: Description of solution.
    *   Fix 2: Description of solution.
*   **Verification**: How to confirm the fix.

#### 2. Shared State Contamination
*   **Symptom**: Tests pass in isolation but fail when run with others, often due to data corruption or unexpected side effects.
*   **Hypotheses**:
    *   Hypothesis A: Description of shared state hypothesis.
    *   Hypothesis B: Description of shared state hypothesis.
*   **Checks**:
    *   Check 1: `command or code snippet` (Explanation).
    *   Check 2: `command or code snippet` (Explanation).
*   **Likely Fixes**:
    *   Fix 1: Description of solution.
    *   Fix 2: Description of solution.
*   **Verification**: How to confirm the fix.

#### 3. Non-Deterministic Test Ordering
*   **Symptom**: Tests fail only when run in a specific, often random, order, indicating implicit dependencies.
*   **Hypotheses**:
    *   Hypothesis A: Description of ordering hypothesis.
    *   Hypothesis B: Description of ordering hypothesis.
*   **Checks**:
    *   Check 1: `command or code snippet` (Explanation).
    *   Check 2: `command or code snippet` (Explanation).
*   **Likely Fixes**:
    *   Fix 1: Description of solution.
    *   Fix 2: Description of solution.
*   **Verification**: How to confirm the fix.

Estimated results

DifficultyIntermediate

Setup time30 min

Time saved30 minutes

Best modelsChatGPT, Gemini, Claude

Best audiencesoftware-development, devops

Editor's note

Why this prompt matters

Intermittent CI failures, often termed "flaky tests," are a common source of frustration for engineering teams. These issues can be particularly insidious when they only manifest under the specific conditions of a parallelized continuous integration environment, passing reliably when tests are run sequentially or locally. Debugging such failures without a structured approach often devolves into guesswork, leading to wasted time and delayed deployments.

This workflow is designed for engineers who find themselves repeatedly battling CI builds that fail inconsistently due to timing issues, shared resource contention, or unpredictable test execution order. It provides a systematic framework to move beyond anecdotal observations, offering a diagnostic playbook to identify the root causes of parallelism-induced flakiness. When your team's velocity is hindered by unreliable CI results, and you suspect that concurrent test execution is the culprit, this guide offers a targeted methodology to pinpoint and rectify those elusive failures, restoring confidence in your automated testing pipeline.

Anatomy

Prompt engineering breakdown

Role

As an experienced software engineer and CI/CD specialist, your task is to assist in debugging flaky test suites that exhibit inconsistent failures when run in parallel within a continuous integration environment.

Context

I am experiencing intermittent failures in my test suite when it runs on our CI system. These failures do not consistently reproduce locally or when tests are run sequentially. The primary suspect is parallelism, introducing race conditions, shared state contamination, or non-deterministic test ordering. The goal is to generate a comprehensive diagnostic playbook to systematically identify and resolve these issues. My test suite is described as: `{{test_suite_description}}` My CI environment details are: `{{ci_environment_details}}`

Goal

Generate a structured debug playbook focused on identifying and resolving flakiness due to parallel execution. The playbook should cover the following common categories of issues: timing dependencies, shared state contamination, and non-deterministic test ordering. For each category, provide a symptom, a list of hypotheses, specific checks (including shell commands or code snippets where applicable), likely fixes, and verification steps.

Constraints

Focus exclusively on issues related to parallel test execution. Do not address unrelated test failures (e.g., incorrect logic in a single test). For 'Checks', provide actionable commands or specific code patterns to look for. For 'Likely Fixes', suggest concrete strategies or code modifications. The output must follow the specified structure precisely. Assume a Linux-based CI environment for shell commands.

Output format

Provide the playbook structured as follows: **Debug Playbook for Flaky CI Tests in Parallel Environments** #### 1. Timing Dependencies * **Symptom**: Tests intermittently fail with timeouts, assertion failures on eventual consistency, or resource unavailability. * **Hypotheses**: * Hypothesis A: Description of timing-related hypothesis. * Hypothesis B: Description of timing-related hypothesis. * **Checks**: * Check 1: `command or code snippet` (Explanation of what this checks). * Check 2: `command or code snippet` (Explanation). * **Likely Fixes**: * Fix 1: Description of solution. * Fix 2: Description of solution. * **Verification**: How to confirm the fix. #### 2. Shared State Contamination * **Symptom**: Tests pass in isolation but fail when run with others, often due to data corruption or unexpected side effects. * **Hypotheses**: * Hypothesis A: Description of shared state hypothesis. * Hypothesis B: Description of shared state hypothesis. * **Checks**: * Check 1: `command or code snippet` (Explanation). * Check 2: `command or code snippet` (Explanation). * **Likely Fixes**: * Fix 1: Description of solution. * Fix 2: Description of solution. * **Verification**: How to confirm the fix. #### 3. Non-Deterministic Test Ordering * **Symptom**: Tests fail only when run in a specific, often random, order, indicating implicit dependencies. * **Hypotheses**: * Hypothesis A: Description of ordering hypothesis. * Hypothesis B: Description of ordering hypothesis. * **Checks**: * Check 1: `command or code snippet` (Explanation). * Check 2: `command or code snippet` (Explanation). * **Likely Fixes**: * Fix 1: Description of solution. * Fix 2: Description of solution. * **Verification**: How to confirm the fix.

Why this structure works

The prompt establishes expertise through role priming, designating the model as an 'experienced software engineer and CI/CD specialist.' Explicit constraints guide the model to focus specifically on parallel execution issues and to provide actionable commands and concrete fixes. The highly structured output format ensures the generated diagnostic playbook is consistently organized and directly usable by an engineer.

Pick your version

Prompt variations

BeginnerWorks with any model

For new engineers or those less familiar with CI/CD debugging, requiring a simpler, more guided approach to common issues.

prompt.txt

You are a helpful CI/CD assistant. Your job is to help me figure out why my tests fail sometimes on our build server, but not on my computer. I think it's because they run at the same time. I need a simple plan to find and fix these problems. My tests are: `{{test_description}}`. Our build system is: `{{build_system_info}}`. Give me a playbook with common problems like tests waiting for each other (timing), tests changing shared data (shared state), or tests running in different orders (ordering). For each problem, tell me what to look for, ideas to fix it, and how to check if it's fixed. Keep it focused only on issues from tests running in parallel.

ProfessionalBest with claude

When a detailed, structured, and expert-level diagnostic plan is required for complex parallel execution issues.

prompt.txt

Act as an experienced software engineer specializing in CI/CD reliability. Your objective is to guide the diagnosis of flaky test suites exhibiting inconsistent failures when executed in parallel within a continuous integration system.

### Context
Our CI system shows intermittent test failures that do not reproduce reliably locally or in sequential runs. Parallel execution is suspected, introducing race conditions, shared state contamination, or non-deterministic test ordering. The goal is to develop a structured diagnostic playbook to systematically identify and resolve these issues.

My test suite configuration is: `{{test_suite_description}}`
Our CI environment specifics are: `{{ci_environment_details}}`

### Task
Construct a detailed debug playbook tailored to pinpointing and resolving flakiness stemming from parallel execution. The playbook must address timing dependencies, shared state contamination, and non-deterministic test ordering. For each issue category, include symptoms, plausible hypotheses, specific checks (with shell commands or code examples), likely remediation strategies, and verification steps.

### Constraints
* Focus strictly on issues caused by parallel test execution. Do not cover general test logic errors.
* 'Checks' should offer actionable commands or identifiable code patterns.
* 'Likely Fixes' should propose concrete solutions or code modifications.
* Maintain the specified output structure precisely.
* Assume a Linux-based CI environment for command-line examples.

Short VersionWorks with any model

For quick initial hypotheses or when needing a concise overview before deep diving into specific diagnostic steps.

prompt.txt

As a CI/CD debugging expert, generate a concise playbook to diagnose flaky parallel test failures. Focus on timing dependencies, shared state contamination, and non-deterministic ordering. For my `{{test_suite_context}}` and `{{ci_environment_details}}`, provide symptoms, key hypotheses, actionable checks (with commands), concrete fixes, and verification steps for each category. Exclude non-parallel issues and ensure the output follows a structured debug playbook format with clear headings for each section. Keep the advice direct and actionable for rapid troubleshooting.

EnterpriseBest with chatgpt

In organizations where compliance, operational risk, and stakeholder communication are key considerations in the debugging process.

prompt.txt

As an experienced software engineer and CI/CD specialist with a focus on enterprise reliability, your task is to develop a comprehensive diagnostic playbook for addressing flaky test suites that exhibit inconsistent failures in parallel CI environments. This is critical for maintaining system reliability and mitigating operational risk.

### Context
We are observing intermittent failures in our CI test suite under parallel execution, which impacts release velocity and stakeholder confidence. These issues do not consistently reproduce in sequential or local runs, pointing to parallelism as the root cause (race conditions, shared state, non-deterministic ordering).
My test suite: `{{test_suite_description}}`
CI environment: `{{ci_environment_details}}`

### Task
Generate a structured debug playbook to systematically identify and resolve these parallel execution-related flakiness issues. For timing dependencies, shared state contamination, and non-deterministic test ordering, provide: symptoms, hypotheses, actionable checks (including shell commands/code), concrete fixes, and verification steps to ensure long-term stability and auditability.

### Constraints
Focus solely on parallel execution issues. Provide actionable commands/patterns and concrete fix strategies. Output must adhere strictly to the specified playbook structure to facilitate consistent application and reporting. Consider potential compliance implications of test failures and the need for clear documentation for audit purposes.

What you'll get

Expected output

Debug Playbook for Flaky CI Tests in Parallel Environments

1. Timing Dependencies

Symptom: Intermittent timeouts, assertion failures on eventual consistency, or resource unavailability, especially with external services or asynchronous operations.
Hypotheses:

* Hypothesis A: A test asserts on state before an asynchronous operation completes. * Hypothesis B: Resource contention under parallel load (e.g., port conflicts, database connection limits).

Checks:

* Check 1: grep -r "time.Sleep" . (Identifies fixed Sleep calls potentially masking race conditions.) * Check 2: docker-compose logs --tail 1000 <service_name> | grep "error\|timeout" (Monitors dependent service logs for resource-related errors during CI runs.)

Likely Fixes:

* Fix 1: Implement explicit wait-for-condition logic with timeouts instead of fixed delays. * Fix 2: Increase CI environment resource limits or use robust health checks for services before tests.

Verification: Rerun the test suite multiple times in parallel on CI. Confirm previous error patterns are absent.

2. Shared State Contamination

Symptom: Tests pass individually but fail when run in a suite, exhibiting corrupted data, unexpected environment variables, or incorrect object states.
Hypotheses:

* Hypothesis A: Tests fail to clean up or isolate their state (e.g., database entries, temporary files), affecting subsequent tests. * Hypothesis B: Parallel tests modify the same shared resource (e.g., a single test database, global cache), leading to race conditions.

Checks:

* Check 1: go test -count=1 -run "TestSpecificFlakyCase" (Run flaky test in isolation. If it passes, run with neighbors to identify interaction.) * Check 2: git grep -lE "(db\.Connect|os\.Setenv|globalVar)" . (Searches for global variables, singletons, or environment variable modifications.)

Likely Fixes:

* Fix 1: Implement robust test isolation: transaction rollbacks for database tests or dedicated temporary resources. * Fix 2: Refactor to avoid global state. Pass dependencies explicitly. Ensure TestMain or equivalent resets global state.

Verification: Run the test suite with increased parallelism (e.g., -p 8) on CI. Observe consistent passes across multiple builds.

3. Non-Deterministic Test Ordering

Symptom: Tests fail only when run in a specific, often random, order, but pass when the order changes or in isolation, indicating implicit dependencies.
Hypotheses:

* Hypothesis A: Test A implicitly relies on side effects or state set up by Test B. * Hypothesis B: A test modifies a shared resource without cleanup, and another test expects a pristine state.

Checks:

* Check 1: go test -shuffle=on -v ./... (Run tests with randomized ordering if supported to expose dependencies.) * Check 2: Analyze CI logs for execution sequence patterns in failed versus successful builds.

Likely Fixes:

* Fix 1: Make each test self-contained and independent with complete setup and teardown. * Fix 2: Refactor shared setup/teardown logic to ensure isolation, e.g., fresh database instances or in-memory databases.

Verification: Run the test suite multiple times in parallel and with randomized ordering. Confirm consistent passes regardless of execution sequence.

Under the hood

Why this prompt works

This prompt workflow is effective because it employs several targeted prompt engineering techniques. First, role priming establishes the persona of an "experienced software engineer and CI/CD specialist." This instructs the model to generate content from an authoritative, practical perspective, ensuring the advice is actionable and technically sound rather than generic.

Second, the prompt utilizes explicit constraints by specifically forbidding unrelated issues and demanding a focus on parallel execution problems. This narrows the scope, preventing the model from wandering into general debugging advice and ensuring the output is highly relevant to the problem at hand. The instruction to assume a "Linux-based CI environment" for commands is another precise constraint that improves the utility of the generated checks.

Finally, the most critical technique is structured output. By providing a detailed, multi-level outline for the "Debug Playbook," the prompt dictates the exact format for symptoms, hypotheses, checks, fixes, and verification steps. This scaffolding guides the model to produce a comprehensive, organized, and immediately usable document. Without this clear structure, a general request for debugging advice would likely yield a less coherent and less actionable list, requiring significant manual reformatting and synthesis from the user. The structured output ensures all necessary components of a diagnostic playbook are present and logically arranged.

Model fit

Best AI models for this prompt

Claude

Claude models excel at structured output and complex reasoning, making them suitable for generating detailed playbooks like this. They are strong at maintaining the specified format and generating comprehensive explanations for each section. However, they may occasionally provide overly generic code snippets if the context provided in the placeholders is not sufficiently specific. See the full Claude hub for deeper guidance.

ChatGPT

ChatGPT models are proficient at diagnostic tasks and can produce practical, actionable advice for debugging scenarios. They are good at generating specific commands and code examples relevant to common CI environments. Users might need to iterate if initial outputs are too broad, refining the context to get more targeted suggestions. See the full ChatGPT hub for deeper guidance.

Gemini

Gemini models handle multi-faceted requests well, balancing structured output with detailed technical content. They are effective at outlining distinct hypotheses and corresponding solutions for debugging. While generally strong, ensure the input for {{test_suite_description}} and {{ci_environment_details}} is thorough to avoid less specific recommendations. See the full Gemini hub for deeper guidance.

When to use

When local test runs consistently pass, but CI builds fail intermittently with no clear pattern.
When increasing the number of parallel jobs in your CI environment correlates with a higher rate of test failures.
When failures point to resource contention, such as database deadlocks, file system errors, or port conflicts, that don't occur sequentially.
When test failures depend on which other tests execute alongside them, indicating implicit dependencies or shared state issues.
When tests occasionally time out or report stale data that should have been updated by another test.

When not to use

When tests consistently fail, regardless of whether they run locally, sequentially, or in parallel; this indicates a fundamental bug.
When the CI environment itself is unstable (e.g., build agent crashes, network outages, out-of-memory errors not related to specific tests).
When the primary goal is to optimize CI build speed, not to diagnose test flakiness.
When debugging issues with the application logic itself, rather than how tests interact within a parallel environment.
When test failures are due to external service downtime that is not related to test parallelism.

Get more from it

Pro tips

1
When troubleshooting, enrich your `test_suite_description` and `ci_environment_details` inputs. Specificity helps the playbook generate more targeted, actionable diagnostic steps, preventing generic advice.
2
Implement granular logging within suspect tests, including timestamps and thread/process IDs. This helps pinpoint exact execution order and resource access, preventing misidentification of timing or shared state issues.
3
Isolate consistently failing tests and run them sequentially first. Confirming they pass in isolation prevents conflating true flakiness with persistent, underlying test failures.
4
Systematically apply one fix at a time, then re-run the CI pipeline. This isolated change approach helps confirm the efficacy of each fix and prevents introducing new regressions.
5
Before deep-diving into code, review CI logs for external resource bottlenecks. External service throttling or database connection limits can mimic internal test flakiness.

Don't ship this

Common mistakes

✗ Not distinguishing between consistent test failures and intermittent flakiness.
Fix — Always verify tests pass reliably in a sequential, isolated run before assuming a parallel execution problem.
✗ Overlooking environmental factors like network latency or external service rate limits.
Fix — Examine CI system logs for external resource warnings or connection errors before focusing solely on test code.
✗ Failing to capture enough diagnostic data during CI runs when flakiness occurs.
Fix — Add detailed logging, process IDs, and timestamps to tests, especially around resource access or state changes.
✗ Attempting to fix multiple potential issues simultaneously without isolation.
Fix — Address one hypothesis and implement a single fix at a time, then re-verify the CI run to confirm impact.
✗ Assuming the problem is always a race condition within the application under test.
Fix — Often, flakiness stems from test setup/teardown not being robust for parallel execution, not the application itself.

Frequently asked questions

Q.How should I prioritize the issues identified by the playbook?

Start with shared state contamination, as it often causes the most unpredictable and hard-to-diagnose failures. Timing dependencies and non-deterministic ordering might become clearer once shared state is managed.

Q.Can this playbook help with performance bottlenecks in my CI suite?

This playbook focuses on identifying and resolving flakiness, not performance optimization. While fixing flakiness might indirectly improve build times by reducing retries, its primary goal is stability and correctness.

Q.What if the recommended fixes are difficult to implement in my current test framework?

The playbook suggests common strategies. If direct implementation is hard, consider framework-specific alternatives that achieve similar outcomes, like test isolation libraries or database transaction wrappers.

Q.Is it possible for a test to be flaky even when run sequentially?

Yes, but this playbook specifically targets flakiness introduced by parallel execution. If a test is flaky sequentially, it indicates a different class of bug, likely within the test logic or application under test.

Q.Should I always aim for fully isolated tests without any shared resources?

Full isolation is ideal but not always practical. The goal is to manage shared resources safely, using locks, unique temporary resources, or robust setup/teardown to prevent contention and contamination.

Version 1.0Last reviewed July 20, 2026

Reviewed by PromptInFlow Editorial Team