Exploration, Confirmation, and Reproducibility: 10 practice questions

Question 1 of 10

What is the primary role of exploratory analysis?

1. ATo replace uncertainty with a deterministic conclusion
2. BTo prove a population claim from the observed sample alone
3. CTo certify that every discovered pattern was prespecified
4. DDiscover patterns and anomalies that refine later questions

Question 2 of 10

Which reporting distinction protects readers from confusing discovery with confirmation?

1. APrimary outcomes versus secondary outcomes, regardless of when they were chosen
2. BA priori objectives versus post hoc objectives
3. CAnalyses that use confidence intervals versus analyses that use p-values
4. DResults produced by scripted code versus results produced by a spreadsheet

Question 3 of 10

A pattern first noticed in a plot should initially be described as what?

1. AA confirmed effect in the target population
2. BAn exploratory result that sharpens the question
3. CA guarantee that the same pattern will recur
4. DA prespecified result regardless of the analysis plan

Question 4 of 10

Why should a report label an analysis introduced after inspecting the outcome results?

1. ASo uncertainty no longer needs to be reported
2. BSo readers recognize it as post hoc rather than planned
3. CSo the analysis can be omitted from the methods
4. DSo the result becomes confirmatory by disclosure alone

Question 5 of 10

Which sequence best begins a reproducible analysis?

1. AFit the final model before checking data types
2. BWrite the conclusion before recording the question
3. CTransform the only copy, then reconstruct its origin later
4. DPreserve raw data and provenance before validating inputs

Question 6 of 10

What is the main reproducibility risk of an undocumented manual filter?

1. AThe analyzed records cannot be reconstructed reliably
2. BSaving the filtered output is sufficient even when the filtering rule is absent
3. CThe filter is reproducible as long as the analyst remembers which rows were removed
4. DA screenshot of the filter menu reconstructs the operation without recording selected values

Question 7 of 10

Which combination creates an auditable analytical path?

1. AA final chart without the code that produced it
2. BA spreadsheet whose manual steps are remembered by one analyst
3. CA narrative that omits deviations from the plan
4. DVersioned code, validated inputs, and execution logs

Question 8 of 10

An analyst finds a subgroup effect while exploring the same data intended for the final test. The subgroup was not in the plan, and the sample is small. What is the most defensible report?

1. AReport it as exploratory with uncertainty and seek new confirmation
2. BSuppress the original plan and report only the smallest p-value
3. CClaim the effect for every subgroup because the plot is visually clear
4. DPresent it as the planned primary result because it is interesting

Question 9 of 10

Two analysts start from the same raw file but obtain different estimates. One silently removed missing rows and the other imputed them. Which repair best restores reproducibility and interpretability?

1. ADelete the raw file so only the selected table remains
2. BAverage the two estimates without documenting either transformation
3. CKeep whichever estimate is closer to the desired conclusion
4. DVersion both missing-data rules and compare their resulting estimates

Question 10 of 10

A team discovered an unplanned subgroup pattern in its first dataset. A second dataset exists and has not been inspected. Which workflow keeps discovery and confirmation separate?

1. ACombine both datasets, pick the subgroup with the clearest effect, and report that test as prespecified.
2. BSpecify the follow-up analysis and test it again on the first dataset, calling that repeat an independent confirmation.
3. CInspect the subgroup results in both datasets, then register the most consistent subgroup as the planned target.
4. DReport the first finding as exploratory, then plan the follow-up analysis before inspecting the second dataset.