Exploration, Confirmation, and Reproducibility

Explore to find the next good question

Exploration is useful when you need to understand an unfamiliar data set before making a narrow inferential claim. Exploratory analysis is for discovering patterns and anomalies that refine later questions. It can reveal an unexpected subgroup, a nonlinear relationship, an extreme observation, or a violated assumption.

Plots and numerical summaries play different roles. Using both helps the analyst notice structure without treating every visible pattern as established.

Discovery is the beginning of reasoning, not the end. A pattern first noticed in a plot is an exploratory result that sharpens the question. It does not automatically confirm that the same pattern exists in the target population.

Imagine a retention chart in which one small region appears to behave differently. The responsible next step is to describe the pattern, quantify its uncertainty, ask whether the region was planned in advance, and design a later check if the claim matters. Visual clarity cannot substitute for sampling quality, model assumptions, or independent confirmation.

Separate discovery from confirmation

Readers need to know when an analytical choice entered the process. The critical reporting distinction is between a priori objectives and post hoc objectives. A priori objectives are declared before the relevant results are inspected; post hoc objectives arise after the data suggest a direction.

This distinction does not make post hoc work worthless. It tells readers how much evidential weight to give it. Confirmation begins with a declared purpose, target population or process, estimand, assumptions, and analysis rule. Exploration can generate those elements for later work, but it cannot retroactively make them prespecified.

An analysis introduced after inspecting the results must be labeled post hoc rather than planned. Reports can also name a third group: analyses added while the results were still blinded. The label protects the chronology of the evidence. It allows a useful follow-up to be reported without presenting discovery as prior confirmation.

For example, if an analyst tests twelve customer segments only after the overall result disappoints, the segment comparison belongs beside the planned result, not in its place. Report an unplanned subgroup effect as exploratory, show its uncertainty, and seek new confirmation. Also disclose the broader search: showing only the favorable segment hides how the result was found.

Confirm a declared target, not a convenient pattern

Confirmation serves a decision only when the target is explicit. State the population or process, observational unit, variables, estimand, and rule that connects the sample to the claim. Then state the assumptions under which the rule is valid.

A confirmatory result remains conditional. A low p-value or narrow interval does not turn a sample result into certainty about every future observation. The conclusion should name both what was estimated and the conditions supporting it.

Suppose an experiment estimates an average treatment effect for eligible customers during one campaign. Even a well-executed analysis does not guarantee the same effect for excluded customers, every individual, or a later campaign. The declared estimand and population determine the scope.

A practical confirmation record therefore connects five elements:

Preserve a stable beginning

Reproducibility starts before modeling. Begin a reproducible analysis by preserving raw data and provenance before validating inputs. Provenance records where the observations came from, when they were obtained, and which version entered the analysis.

Do not transform the only copy and then attempt to reconstruct its origin. Keep the preserved input immutable, and make cleaning steps produce new, traceable states. Validation should check expected columns, types, ranges, missingness, and identifiers before transformations or models run. This sequence creates a stable starting point for comparison when results later change.

A simple raw-to-analysis chain might be:

1. Register the input file and its version.
2. Validate its schema and critical values.
3. Apply scripted cleaning and transformations.
4. Fit the declared analysis.
5. Run diagnostics and tests.
6. Produce the report from the same pipeline.

If a refreshed file changes a category label or drops a column, validation should stop the run before the change silently reaches the model. Preserving the prior input then lets the team explain why the new output differs.

Make every consequential change reviewable

Manual operations are dangerous when they are invisible. An undocumented manual filter means the analyzed records cannot be reconstructed reliably. The problem is not that every manual action is forbidden; it is that another analyst cannot know exactly which rows changed, why they changed, or when the action occurred.

Necessary exceptions should record the affected records, rule, rationale, author, and pipeline stage. Better still, encode repeatable transformations in versioned code. A scripted filter is inspectable, testable, and rerunnable; a remembered point-and-click sequence is none of those things.

Versioned code, validated inputs, and execution logs create an auditable analytical path. The history connects a result to the data state and code version that produced it, while tests and logs show whether each step succeeded.

Missing-data choices illustrate why this matters. Silent row deletion and imputation can produce different estimates from the same raw file. Take three customers with weekly sales of $10, $20, and one value missing. Dropping the incomplete row leaves two customers and a mean of $15. Replacing the missing sale with $0 leaves three customers and a mean of $10. Same file, same target, two different numbers.

Version both missing-data rules and compare their resulting estimates. The comparison makes the analytical choice visible and reveals whether the conclusion depends on it; selecting whichever estimate is preferred would instead make the process outcome-driven.

Carry the evidence all the way to the claim

A reproducible report is not merely code that runs twice. It connects the stated question to the estimate, uncertainty, diagnostics, deviations, and limits. A reviewer should be able to follow the declared steps from preserved input to reported output without guessing at hidden choices.

The strongest workflow preserves inputs, explores, prespecifies inference, checks assumptions, and reports limits. Exploration remains visible as discovery, and confirmation stays tied to its declared target. The executable path preserves how observations became evidence. Deviations can be justified, but each one should be dated, explained, and interpreted.

The complete workflow is a chain of responsibilities:

• Preserve raw observations and provenance.
• Explore patterns, anomalies, and assumption failures.
• Declare the target and confirmatory rule before inspecting the answer.
• Execute validation, transformation, analysis, and diagnostics in a reviewable pipeline.
• Report uncertainty, post hoc work, deviations, and scope.

Breaking any link weakens a different part of trust. Hidden exploration confuses evidential roles; hidden processing breaks reproducibility; ignored diagnostics weaken validity; and an unqualified conclusion exceeds what the data support.