Multiple comparisons problem - Wikipedia

In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously ... Multiplecomparisonsproblem FromWikipedia,thefreeencyclopedia Jumptonavigation Jumptosearch Problemwhereoneconsidersasetofinferencessimultaneouslybasedontheobservedvalues Anexampleofacoincidenceproducedbydatadredging(showingacorrelationbetweenthenumberoflettersinaspellingbee'swinningwordandthenumberofpeopleintheUnitedStateskilledbyvenomousspiders).Givenalargeenoughpoolofvariablesforthesametimeperiod,itispossibletofindapairofgraphsthatshowacorrelationwithnocausation. Instatistics,themultiplecomparisons,multiplicityormultipletestingproblemoccurswhenoneconsidersasetofstatisticalinferencessimultaneously[1]orinfersasubsetofparametersselectedbasedontheobservedvalues.[2] Themoreinferencesaremade,themorelikelyerroneousinferencesbecome.Severalstatisticaltechniqueshavebeendevelopedtoaddressthatproblem,typicallybyrequiringastrictersignificancethresholdforindividualcomparisons,soastocompensateforthenumberofinferencesbeingmade. Contents 1History 2Definition 2.1Classificationofmultiplehypothesistests 3Controllingprocedures 3.1Multipletestingcorrection 4Large-scalemultipletesting 4.1Assessingwhetheranyalternativehypothesesaretrue 5Seealso 6References 7Furtherreading History[edit] Theproblemofmultiplecomparisonsreceivedincreasedattentioninthe1950swiththeworkofstatisticianssuchasTukeyandScheffé.Overtheensuingdecades,manyproceduresweredevelopedtoaddresstheproblem.In1996,thefirstinternationalconferenceonmultiplecomparisonprocedurestookplaceinIsrael.[3] Definition[edit] Multiplecomparisonsarisewhenastatisticalanalysisinvolvesmultiplesimultaneousstatisticaltests,eachofwhichhasapotentialtoproducea"discovery."Astatedconfidencelevelgenerallyappliesonlytoeachtestconsideredindividually,butoftenitisdesirabletohaveaconfidencelevelforthewholefamilyofsimultaneoustests.[4]Failuretocompensateformultiplecomparisonscanhaveimportantreal-worldconsequences,asillustratedbythefollowingexamples: Supposethetreatmentisanewwayofteachingwritingtostudents,andthecontrolisthestandardwayofteachingwriting.Studentsinthetwogroupscanbecomparedintermsofgrammar,spelling,organization,content,andsoon.Asmoreattributesarecompared,itbecomesincreasinglylikelythatthetreatmentandcontrolgroupswillappeartodifferonatleastoneattributeduetorandomsamplingerroralone. Supposeweconsidertheefficacyofadrugintermsofthereductionofanyoneofanumberofdiseasesymptoms.Asmoresymptomsareconsidered,itbecomesincreasinglylikelythatthedrugwillappeartobeanimprovementoverexistingdrugsintermsofatleastonesymptom. Inbothexamples,asthenumberofcomparisonsincreases,itbecomesmorelikelythatthegroupsbeingcomparedwillappeartodifferintermsofatleastoneattribute.Ourconfidencethataresultwillgeneralizetoindependentdatashouldgenerallybeweakerifitisobservedaspartofananalysisthatinvolvesmultiplecomparisons,ratherthanananalysisthatinvolvesonlyasinglecomparison. Forexample,ifonetestisperformedatthe5%levelandthecorrespondingnullhypothesisistrue,thereisonlya5%riskofincorrectlyrejectingthenullhypothesis.However,if100testsareeachconductedatthe5%levelandallcorrespondingnullhypothesesaretrue,theexpectednumberofincorrectrejections(alsoknownasfalsepositivesorTypeIerrors)is5.Ifthetestsarestatisticallyindependentfromeachother,theprobabilityofatleastoneincorrectrejectionisapproximately99.4%. Themultiplecomparisonsproblemalsoappliestoconfidenceintervals.Asingleconfidenceintervalwitha95%coverageprobabilitylevelwillcontainthetruevalueoftheparameterin95%ofsamples.However,ifoneconsiders100confidenceintervalssimultaneously,eachwith95%coverageprobability,theexpectednumberofnon-coveringintervalsis5.Iftheintervalsarestatisticallyindependentfromeachother,theprobabilitythatatleastoneintervaldoesnotcontainthepopulationparameteris99.4%. Techniqueshavebeendevelopedtopreventtheinflationoffalsepositiveratesandnon-coverageratesthatoccurwithmultiplestatisticaltests. Classificationofmultiplehypothesistests[edit] Thefollowingtabledefinesthepossibleoutcomeswhentestingmultiplenullhypotheses. Supposewehaveanumbermofnullhypotheses,denotedby:H1, H2, ..., Hm. Usingastatisticaltest,werejectthenullhypothesisifthetestisdeclaredsignificant.Wedonotrejectthenullhypothesisifthetestisnon-significant. SummingeachtypeofoutcomeoverallHi yieldsthefollowingrandomvariables: Nullhypothesisistrue(H0) Alternativehypothesisistrue(HA) Total Testisdeclaredsignificant V S R Testisdeclarednon-significant U T m − R {\displaystylem-R} Total m 0 {\displaystylem_{0}} m − m 0 {\displaystylem-m_{0}} m misthetotalnumberhypothesestested m 0 {\displaystylem_{0}} isthenumberoftruenullhypotheses,anunknownparameter m − m 0 {\displaystylem-m_{0}} isthenumberoftruealternativehypotheses Visthenumberoffalsepositives(TypeIerror)(alsocalled"falsediscoveries") Sisthenumberoftruepositives(alsocalled"truediscoveries") Tisthenumberoffalsenegatives(TypeIIerror) Uisthenumberoftruenegatives R = V + S {\displaystyleR=V+S} isthenumberofrejectednullhypotheses(alsocalled"discoveries",eithertrueorfalse) Inmhypothesistestsofwhich m 0 {\displaystylem_{0}} aretruenullhypotheses,Risanobservablerandomvariable,andS,T,U,andVareunobservablerandomvariables. Controllingprocedures[edit] Furtherinformation:Family-wiseerrorrate§ Controllingprocedures Seealso:Falsecoveragerate§ Controllingprocedures,andFalsediscoveryrate§ Controllingprocedures Ifmindependentcomparisonsareperformed,thefamily-wiseerrorrate(FWER),isgivenby α ¯ = 1 − ( 1 − α { percomparison } ) m . {\displaystyle{\bar{\alpha}}=1-\left(1-\alpha_{\{{\text{percomparison}}\}}\right)^{m}.} Hence,unlessthetestsareperfectlypositivelydependent(i.e.,identical), α ¯ {\displaystyle{\bar{\alpha}}} increasesasthenumberofcomparisonsincreases. Ifwedonotassumethatthecomparisonsareindependent,thenwecanstillsay: α ¯ ≤ m ⋅ α { percomparison } , {\displaystyle{\bar{\alpha}}\leqm\cdot\alpha_{\{{\text{percomparison}}\}},} whichfollowsfromBoole'sinequality.Example: 0.2649 = 1 − ( 1 − .05 ) 6 ≤ .05 × 6 = 0.3 {\displaystyle0.2649=1-(1-.05)^{6}\leq.05\times6=0.3} Therearedifferentwaystoassurethatthefamily-wiseerrorrateisatmost α ¯ {\displaystyle{\bar{\alpha}}} .Themostconservativemethod,whichisfreeofdependenceanddistributionalassumptions,istheBonferronicorrection α { p e r   c o m p a r i s o n } = α / m {\displaystyle\alpha_{\mathrm{\{per\comparison\}}}={\alpha}/m} .Amarginallylessconservativecorrectioncanbeobtainedbysolvingtheequationforthefamily-wiseerrorrateof m {\displaystylem} independentcomparisonsfor α { p e r   c o m p a r i s o n } {\displaystyle\alpha_{\mathrm{\{per\comparison\}}}} .Thisyields α { percomparison } = 1 − ( 1 − α ) 1 / m {\displaystyle\alpha_{\{{\text{percomparison}}\}}=1-{(1-{\alpha})}^{1/m}} ,whichisknownastheŠidákcorrection.AnotherprocedureistheHolm–Bonferronimethod,whichuniformlydeliversmorepowerthanthesimpleBonferronicorrection,bytestingonlythelowestp-value( i = 1 {\displaystylei=1} )againstthestrictestcriterion,andthehigherp-values( i > 1 {\displaystylei>1} )againstprogressivelylessstrictcriteria.[5] α { p e r   c o m p a r i s o n } = α / ( m − i + 1 ) {\displaystyle\alpha_{\mathrm{\{per\comparison\}}}={\alpha}/(m-i+1)} . Forcontinuousproblems,onecanemployBayesianlogictocompute m {\displaystylem} fromtheprior-to-posteriorvolumeratio.ContinuousgeneralizationsoftheBonferroniandŠidákcorrectionarepresentedin.[6] Multipletestingcorrection[edit] Thissectionmayneedtobecleanedup.IthasbeenmergedfromMultipletestingcorrection. Multipletestingcorrectionreferstomakingstatisticaltestsmorestringentinordertocounteracttheproblemofmultipletesting.ThebestknownsuchadjustmentistheBonferronicorrection,butothermethodshavebeendeveloped.Suchmethodsaretypicallydesignedtocontrolthefamilywiseerrorrateorthefalsediscoveryrate. Large-scalemultipletesting[edit] Traditionalmethodsformultiplecomparisonsadjustmentsfocusoncorrectingformodestnumbersofcomparisons,ofteninananalysisofvariance.Adifferentsetoftechniqueshavebeendevelopedfor"large-scalemultipletesting",inwhichthousandsorevengreaternumbersoftestsareperformed.Forexample,ingenomics,whenusingtechnologiessuchasmicroarrays,expressionlevelsoftensofthousandsofgenescanbemeasured,andgenotypesformillionsofgeneticmarkerscanbemeasured.Particularlyinthefieldofgeneticassociationstudies,therehasbeenaseriousproblemwithnon-replication—aresultbeingstronglystatisticallysignificantinonestudybutfailingtobereplicatedinafollow-upstudy.Suchnon-replicationcanhavemanycauses,butitiswidelyconsideredthatfailuretofullyaccountfortheconsequencesofmakingmultiplecomparisonsisoneofthecauses.[7]Ithasbeenarguedthatadvancesinmeasurementandinformationtechnologyhavemadeitfareasiertogeneratelargedatasetsforexploratoryanalysis,oftenleadingtothetestingoflargenumbersofhypotheseswithnopriorbasisforexpectingmanyofthehypothesestobetrue.Inthissituation,veryhighfalsepositiveratesareexpectedunlessmultiplecomparisonsadjustmentsaremade. Forlarge-scaletestingproblemswherethegoalistoprovidedefinitiveresults,thefamilywiseerrorrateremainsthemostacceptedparameterforascribingsignificancelevelstostatisticaltests.Alternatively,ifastudyisviewedasexploratory,orifsignificantresultscanbeeasilyre-testedinanindependentstudy,controlofthefalsediscoveryrate(FDR)[8][9][10]isoftenpreferred.TheFDR,looselydefinedastheexpectedproportionoffalsepositivesamongallsignificanttests,allowsresearcherstoidentifyasetof"candidatepositives"thatcanbemorerigorouslyevaluatedinafollow-upstudy.[11] Thepracticeoftryingmanyunadjustedcomparisonsinthehopeoffindingasignificantoneisaknownproblem,whetherappliedunintentionallyordeliberately,issometimescalled"p-hacking."[12][13] Assessingwhetheranyalternativehypothesesaretrue[edit] AnormalquantileplotforasimulatedsetofteststatisticsthathavebeenstandardizedtobeZ-scoresunderthenullhypothesis.Thedepartureoftheuppertailofthedistributionfromtheexpectedtrendalongthediagonalisduetothepresenceofsubstantiallymorelargeteststatisticvaluesthanwouldbeexpectedifallnullhypothesesweretrue.Theredpointcorrespondstothefourthlargestobservedteststatistic,whichis3.13,versusanexpectedvalueof2.06.Thebluepointcorrespondstothefifthsmallestteststatistic,whichis-1.75,versusanexpectedvalueof-1.96.Thegraphsuggeststhatitisunlikelythatallthenullhypothesesaretrue,andthatmostorallinstancesofatruealternativehypothesisresultfromdeviationsinthepositivedirection. Abasicquestionfacedattheoutsetofanalyzingalargesetoftestingresultsiswhetherthereisevidencethatanyofthealternativehypothesesaretrue.Onesimplemeta-testthatcanbeappliedwhenitisassumedthatthetestsareindependentofeachotheristousethePoissondistributionasamodelforthenumberofsignificantresultsatagivenlevelαthatwouldbefoundwhenallnullhypothesesaretrue.[citationneeded]Iftheobservednumberofpositivesissubstantiallygreaterthanwhatshouldbeexpected,thissuggeststhattherearelikelytobesometruepositivesamongthesignificantresults.Forexample,if1000independenttestsareperformed,eachatlevelα = 0.05,weexpect0.05×1000=50significantteststooccurwhenallnullhypothesesaretrue.BasedonthePoissondistributionwithmean50,theprobabilityofobservingmorethan61significanttestsislessthan0.05,soifmorethan61significantresultsareobserved,itisverylikelythatsomeofthemcorrespondtosituationswherethealternativehypothesisholds.Adrawbackofthisapproachisthatitoverstatestheevidencethatsomeofthealternativehypothesesaretruewhentheteststatisticsarepositivelycorrelated,whichcommonlyoccursinpractice.[citationneeded].Ontheotherhand,theapproachremainsvalideveninthepresenceofcorrelationamongtheteststatistics,aslongasthePoissondistributioncanbeshowntoprovideagoodapproximationforthenumberofsignificantresults.Thisscenarioarises,forinstance,whenminingsignificantfrequentitemsetsfromtransactionaldatasets.Furthermore,acarefultwostageanalysiscanboundtheFDRatapre-specifiedlevel.[14] AnothercommonapproachthatcanbeusedinsituationswheretheteststatisticscanbestandardizedtoZ-scoresistomakeanormalquantileplotoftheteststatistics.Iftheobservedquantilesaremarkedlymoredispersedthanthenormalquantiles,thissuggeststhatsomeofthesignificantresultsmaybetruepositives.[citationneeded] Seealso[edit] q-value Keyconcepts Familywiseerrorrate Falsepositiverate Falsediscoveryrate(FDR) Falsecoveragerate(FCR) Intervalestimation Post-hocanalysis Experimentwiseerrorrate Statisticalhypothesistesting Generalmethodsofalphaadjustmentformultiplecomparisons Closedtestingprocedure Bonferronicorrection Boole–Bonferronibound Duncan'snewmultiplerangetest Holm–Bonferronimethod Harmonicmeanp-valueprocedure Relatedconcepts Testinghypothesessuggestedbythedata Texassharpshooterfallacy Modelselection Look-elsewhereeffect Datadredging References[edit] ^Miller,R.G.(1981).SimultaneousStatisticalInference2ndEd.SpringerVerlagNewYork.ISBN 978-0-387-90548-8. ^Benjamini,Y.(2010)."Simultaneousandselectiveinference:Currentsuccessesandfuturechallenges".BiometricalJournal.52(6):708–721.doi:10.1002/bimj.200900299.PMID 21154895.S2CID 8806192. ^"Home".mcp-conference.org. ^Kutner,Michael;Nachtsheim,Christopher;Neter,John;Li,William(2005).AppliedLinearStatisticalModels.pp. 744–745.ISBN 9780072386882. ^Aickin,M;Gensler,H(May1996)."Adjustingformultipletestingwhenreportingresearchresults:theBonferronivsHolmmethods".AmJPublicHealth.86(5):726–728.doi:10.2105/ajph.86.5.726.PMC 1380484.PMID 8629727. ^Bayer,AdrianE.;Seljak,Uroš(2020)."Thelook-elsewhereeffectfromaunifiedBayesianandfrequentistperspective".JournalofCosmologyandAstroparticlePhysics.2020(10):009.arXiv:2007.13821.Bibcode:2020JCAP...10..009B.doi:10.1088/1475-7516/2020/10/009.S2CID 220830693. ^Qu,Hui-Qi;Tien,Matthew;Polychronakos,Constantin(2010-10-01)."Statisticalsignificanceingeneticassociationstudies".ClinicalandInvestigativeMedicine.33(5):E266–E270.ISSN 0147-958X.PMC 3270946.PMID 20926032. ^Benjamini,Yoav;Hochberg,Yosef(1995)."Controllingthefalsediscoveryrate:apracticalandpowerfulapproachtomultipletesting".JournaloftheRoyalStatisticalSociety,SeriesB.57(1):125–133.JSTOR 2346101. ^Storey,JD;Tibshirani,Robert(2003)."Statisticalsignificanceforgenome-widestudies".PNAS.100(16):9440–9445.Bibcode:2003PNAS..100.9440S.doi:10.1073/pnas.1530509100.JSTOR 3144228.PMC 170937.PMID 12883005. ^Efron,Bradley;Tibshirani,Robert;Storey,JohnD.;Tusher,Virginia(2001)."EmpiricalBayesanalysisofamicroarrayexperiment".JournaloftheAmericanStatisticalAssociation.96(456):1151–1160.doi:10.1198/016214501753382129.JSTOR 3085878.S2CID 9076863. ^Noble,WilliamS.(2009-12-01)."Howdoesmultipletestingcorrectionwork?".NatureBiotechnology.27(12):1135–1137.doi:10.1038/nbt1209-1135.ISSN 1087-0156.PMC 2907892.PMID 20010596. ^Young,S.S.,Karr,A.(2011)."Deming,dataandobservationalstudies"(PDF).Significance.8(3):116–120.doi:10.1111/j.1740-9713.2011.00506.x.{{citejournal}}:CS1maint:multiplenames:authorslist(link) ^ Smith,G.D.,Shah,E.(2002)."Datadredging,bias,orconfounding".BMJ.325(7378):1437–1438.doi:10.1136/bmj.325.7378.1437.PMC 1124898.PMID 12493654.{{citejournal}}:CS1maint:multiplenames:authorslist(link) ^Kirsch,A;Mitzenmacher,M;Pietracaprina,A;Pucci,G;Upfal,E;Vandin,F(June2012)."AnEfficientRigorousApproachforIdentifyingStatisticallySignificantFrequentItemsets".JournaloftheACM.59(3):12:1–12:22.arXiv:1002.1104.doi:10.1145/2220357.2220359. Furtherreading[edit] F.Betz,T.Hothorn,P.Westfall(2010),MultipleComparisonsUsingR,CRCPress S.DudoitandM.J.vanderLaan(2008),MultipleTestingProcedureswithApplicationtoGenomics,Springer Farcomeni,A.(2008)."AReviewofModernMultipleHypothesisTesting,withparticularattentiontothefalsediscoveryproportion".StatisticalMethodsinMedicalResearch.17(4):347–388.doi:10.1177/0962280206079046.hdl:11573/142139.PMID 17698936.S2CID 12777404. Phipson,B.;Smyth,G.K.(2010)."PermutationP-valuesShouldNeverBeZero:CalculatingExactP-valueswhenPermutationsareRandomlyDrawn".StatisticalApplicationsinGeneticsandMolecularBiology.9:Article39.arXiv:1603.05766.doi:10.2202/1544-6115.1585.PMID 21044043.S2CID 10735784. P.H.WestfallandS.S.Young(1993),Resampling-basedMultipleTesting:ExamplesandMethodsforp-ValueAdjustment,Wiley P.Westfall,R.Tobias,R.Wolfinger(2011)MultiplecomparisonsandmultipletestingusingSAS,2ndedn,SASInstitute Agalleryofexamplesofimplausiblecorrelationssourcedbydatadredging vteDesignofexperimentsScientificmethod Scientificexperiment Statisticaldesign Control Internalandexternalvalidity Experimentalunit Blinding Optimaldesign:Bayesian Randomassignment Randomization Restrictedrandomization Replicationversussubsampling Samplesize Treatmentandblocking Treatment Effectsize Contrast Interaction Confounding Orthogonality Blocking Covariate Nuisancevariable Modelsandinference Linearregression Ordinaryleastsquares Bayesian Randomeffect Mixedmodel Hierarchicalmodel:Bayesian Analysisofvariance(Anova) Cochran'stheorem Manova(multivariate) Ancova(covariance) Comparemeans Multiplecomparison DesignsCompletelyrandomized Factorial Fractionalfactorial Plackett-Burman Taguchi Responsesurfacemethodology Polynomialandrationalmodeling Box-Behnken Centralcomposite Block Generalizedrandomizedblockdesign(GRBD) Latinsquare Graeco-Latinsquare Orthogonalarray LatinhypercubeRepeatedmeasuresdesign Crossoverstudy Randomizedcontrolledtrial Sequentialanalysis Sequentialprobabilityratiotest Glossary Category  Mathematicsportal Statisticaloutline Statisticaltopics vteStatistics Outline Index DescriptivestatisticsContinuousdataCenter Mean Arithmetic Cubic Generalized/power Geometric Harmonic Heinz Lehmer Median Mode Dispersion Averageabsolutedeviation Coefficientofvariation Interquartilerange Percentile Range Standarddeviation Variance Shape Centrallimittheorem Moments Kurtosis L-moments Skewness Countdata Indexofdispersion Summarytables Contingencytable Frequencydistribution Groupeddata Dependence Partialcorrelation Pearsonproduct-momentcorrelation Rankcorrelation Kendall'sτ Spearman'sρ Scatterplot Graphics Barchart Biplot Boxplot Controlchart Correlogram Fanchart Forestplot Histogram Piechart Q–Qplot Radarchart Runchart Scatterplot Stem-and-leafdisplay Violinplot DatacollectionStudydesign Effectsize Missingdata Optimaldesign Population Replication Samplesizedetermination Statistic Statisticalpower Surveymethodology Sampling Cluster Stratified Opinionpoll Questionnaire Standarderror Controlledexperiments Blocking Factorialexperiment Interaction Randomassignment Randomizedcontrolledtrial Randomizedexperiment Scientificcontrol Adaptivedesigns Adaptiveclinicaltrial Stochasticapproximation Up-and-downdesigns Observationalstudies Cohortstudy Cross-sectionalstudy Naturalexperiment Quasi-experiment StatisticalinferenceStatisticaltheory Population Statistic Probabilitydistribution Samplingdistribution Orderstatistic Empiricaldistribution Densityestimation Statisticalmodel Modelspecification Lpspace Parameter location scale shape Parametricfamily Likelihood (monotone) Location–scalefamily Exponentialfamily Completeness Sufficiency Statisticalfunctional Bootstrap U V Optimaldecision lossfunction Efficiency Statisticaldistance divergence Asymptotics Robustness FrequentistinferencePointestimation Estimatingequations Maximumlikelihood Methodofmoments M-estimator Minimumdistance Unbiasedestimators Mean-unbiasedminimum-variance Rao–Blackwellization Lehmann–Scheffétheorem Medianunbiased Plug-in Intervalestimation Confidenceinterval Pivot Likelihoodinterval Predictioninterval Toleranceinterval Resampling Bootstrap Jackknife Testinghypotheses 1-&2-tails Power Uniformlymostpowerfultest Permutationtest Randomizationtest Multiplecomparisons Parametrictests Likelihood-ratio Score/Lagrangemultiplier Wald Specifictests Z-test(normal) Student'st-test F-test Goodnessoffit Chi-squared G-test Kolmogorov–Smirnov Anderson–Darling Lilliefors Jarque–Bera Normality(Shapiro–Wilk) Likelihood-ratiotest Modelselection Crossvalidation AIC BIC Rankstatistics Sign Samplemedian Signedrank(Wilcoxon) Hodges–Lehmannestimator Ranksum(Mann–Whitney) Nonparametricanova 1-way(Kruskal–Wallis) 2-way(Friedman) Orderedalternative(Jonckheere–Terpstra) VanderWaerdentest Bayesianinference Bayesianprobability prior posterior Credibleinterval Bayesfactor Bayesianestimator Maximumposteriorestimator CorrelationRegressionanalysisCorrelation Pearsonproduct-moment Partialcorrelation Confoundingvariable Coefficientofdetermination Regressionanalysis Errorsandresiduals Regressionvalidation Mixedeffectsmodels Simultaneousequationsmodels Multivariateadaptiveregressionsplines(MARS) Linearregression Simplelinearregression Ordinaryleastsquares Generallinearmodel Bayesianregression Non-standardpredictors Nonlinearregression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity Generalizedlinearmodel Exponentialfamilies Logistic(Bernoulli) /Binomial /Poissonregressions Partitionofvariance Analysisofvariance(ANOVA,anova) Analysisofcovariance MultivariateANOVA Degreesoffreedom Categorical /Multivariate /Time-series /SurvivalanalysisCategorical Cohen'skappa Contingencytable Graphicalmodel Log-linearmodel McNemar'stest Cochran-Mantel-Haenszelstatistics Multivariate Regression Manova Principalcomponents Canonicalcorrelation Discriminantanalysis Clusteranalysis Classification Structuralequationmodel Factoranalysis Multivariatedistributions Ellipticaldistributions Normal Time-seriesGeneral Decomposition Trend Stationarity Seasonaladjustment Exponentialsmoothing Cointegration Structuralbreak Grangercausality Specifictests Dickey–Fuller Johansen Q-statistic(Ljung–Box) Durbin–Watson Breusch–Godfrey Timedomain Autocorrelation(ACF) partial(PACF) Cross-correlation(XCF) ARMAmodel ARIMAmodel(Box–Jenkins) Autoregressiveconditionalheteroskedasticity(ARCH) Vectorautoregression(VAR) Frequencydomain Spectraldensityestimation Fourieranalysis Least-squaresspectralanalysis Wavelet Whittlelikelihood SurvivalSurvivalfunction Kaplan–Meierestimator(productlimit) Proportionalhazardsmodels Acceleratedfailuretime(AFT)model Firsthittingtime Hazardfunction Nelson–Aalenestimator Test Log-ranktest ApplicationsBiostatistics Bioinformatics Clinicaltrials /studies Epidemiology Medicalstatistics Engineeringstatistics Chemometrics Methodsengineering Probabilisticdesign Process /qualitycontrol Reliability Systemidentification Socialstatistics Actuarialscience Census Crimestatistics Demography Econometrics Jurimetrics Nationalaccounts Officialstatistics Populationstatistics Psychometrics Spatialstatistics Cartography Environmentalstatistics Geographicinformationsystem Geostatistics Kriging Category  Mathematicsportal Commons WikiProject Retrievedfrom"https://en.wikipedia.org/w/index.php?title=Multiple_comparisons_problem&oldid=1102900429" Categories:StatisticalhypothesistestingMultiplecomparisonsHiddencategories:CS1maint:multiplenames:authorslistArticleswithshortdescriptionShortdescriptionmatchesWikidataAllarticleswithunsourcedstatementsArticleswithunsourcedstatementsfromJune2016ArticleswithunsourcedstatementsfromAugust2012ArticleswithunsourcedstatementsfromJanuary2012 Navigationmenu Personaltools NotloggedinTalkContributionsCreateaccountLogin Namespaces ArticleTalk English Views ReadEditViewhistory More Search Navigation MainpageContentsCurrenteventsRandomarticleAboutWikipediaContactusDonate Contribute HelpLearntoeditCommunityportalRecentchangesUploadfile Tools WhatlinkshereRelatedchangesUploadfileSpecialpagesPermanentlinkPageinformationCitethispageWikidataitem Print/export DownloadasPDFPrintableversion Languages Deutschفارسی한국어עבריתNederlandsPolskiРусскийSvenska粵語中文 Editlinks