10.3 - Multiple Comparisons | STAT 500

Multiple comparisons conducts an analysis of all possible pairwise means. For example, with three brands of cigarettes, A, B, and C, if the ANOVA test was ... Skiptomaincontent Breadcrumb Home 10 10.3 10.3-MultipleComparisons Ifourtestofthenullhypothesisisrejected,weconcludethatnotallthemeansareequal:thatis,atleastonemeanisdifferentfromtheothermeans.TheANOVAtestitselfprovidesonlystatisticalevidenceofadifference,butnotanystatisticalevidenceastowhichmeanormeansarestatisticallydifferent. Forinstance,usingthepreviousexamplefortarcontent,iftheANOVAtestresultsinasignificantdifferenceinaveragetarcontentbetweenthecigarettebrands,afollowupanalysiswouldbeneededtodeterminewhichbrandmeanormeansdifferintarcontent.Pluswewouldwanttoknowifonebrandormultiplebrandswerebetter/worsethananotherbrandinaveragetarcontent.Tocompletethisanalysisweuseamethodcalledmultiplecomparisons. Multiplecomparisonsconductsananalysisofallpossiblepairwisemeans.Forexample,withthreebrandsofcigarettes,A,B,andC,iftheANOVAtestwassignificant,thenmultiplecomparisonmethodswouldcomparethethreepossiblepairwisecomparisons: BrandAtoBrandB BrandAtoBrandC BrandBtoBrandC Theseareessentiallytestsoftwomeanssimilartowhatwelearnedpreviouslyinourlessonforcomparingtwomeans.However,themethodshereuseanadjustmenttoaccountforthenumberofcomparisonstakingplace.Minitabprovidesthreeadjustmentchoices.WewillusetheTukeyadjustmentwhichisanadjustmentonthet-multiplierbasedonthenumberofcomparisons. Note!Wedon’tgointhetheorybehindtheTukeymethod.JustnotethatweonlyuseamultiplecomparisontechniqueinANOVAwhenwehaveasignificantresult. Inthenextsection,wepresentanexampletowalkthroughtheANOVAresults. Minitab® UsingMinitabtoPerformOne-WayANOVA Section  IfthedataenteredinMinitabareindifferentcolumns,theninMinitabweuse: Stat>ANOVA>One-Way Selecttheformatstructureofthedataintheworksheet. Iftheresponsesareinonecolumnandthefactorsareintheirowncolumn,thenselectthedropdownof'Responsedataareinonecolumnforallfactorlevels.' Iftheresponsesareintheirowncolumnforeachfactorlevel,thenselect'Responsedataareinaseparatecolumnforeachfactorlevel.' Next,incasewehaveasignificantANOVAresult,andwewanttoconductamultiplecomparisonanalysis,wepreemptivelyclick'Comparisons',theboxforTukey,andverifythattheboxesfor'Intervalplotfordifferencesofmeans'and'GroupingInformation'arealsochecked. ClickOKandOKagain. Example:TarContent(ANOVA) Section  Testthehypothesisthatthemeansarethesamevs.atleastoneisdifferentforbothlabs.Comparethetwolabsandcomment. Answer LabPrecise Wearetestingthefollowinghypotheses: \(H_0\colon\mu_1=\mu_2=\mu_3\)vs\(H_a\colon\text{atleastonemeanisdifferent}\) Theassumptionswerediscussedinthepreviousexample. Thefollowingistheoutputforone-wayANOVAforLabPrecise: One-wayANOVA:PreciseA,PreciseB,PreciseC Method NullHypothesis Allmeansareequal AlternativeHypothesis Notallmeansareequal SignificanceLevel \(\alpha\)=0.05 Equalvarianceswereassumedfortheanalysis. FactorInformation Factor Levels Values Factor 3 PreciseA,PreciseB,PreciseC AnalysisofVariance Source DF AdjSS AdjMS F-Value P-Value Factor 2 12.000 6.00000 65.46 0.000 Error 15 1.375 0.09165     Total 17 13.375       ModelSummary S R-sq R-sq(adj) R-sq(pred) 0.302743 89.72% 88.35% 85.20% Thep-valueforthistestislessthan0.0001.Atanyreasonablesignificancelevel,wewouldrejectthenullhypothesisandconcludethereisenoughevidenceinthedatatosuggestatleastonemeantarcontentisdifferent. Butwhichonesaredifferent?Thenextstepistoexaminethemultiplecomparisons.Minitabprovidesthefollowingoutput: Means Factor N Mean StDev 95%CI PreciseA 6 10.000 0.257 (9.737,10.263) PreciseB 6 11.000 0.365 (10.737,11.263) PreciseC 6 12.000 0.276 (11.737,12.263) PooledStDev=0.302743 TukeyPairwiseComparisons GroupingInformationUsingtheTukeyMethodand95%Confidence Factor N Mean Grouping PreciseC 6 12.000 A PreciseB 6 11.000 B PreciseA 6 10.000 C Meansthatdonotsharealetteraresignificantlydifferent. TheTukeypairwisecomparisonssuggestthatallthemeansaredifferent.Therefore,BrandChasthehighesttarcontentandBrandAhasthelowest. LabSloppy WearetestingthesamehypothesesforLabSloppyasLabPrecise,andtheassumptionswerechecked.TheANOVAoutputforLabSloppyis: One-wayANOVA:SloppyA,SloppyB,SloppyC Method NullHypothesis Allmeansareequal AlternativeHypothesis Notallmeansareequal SignificanceLevel \(\alpha\)=0.05 Equalvarianceswereassumedfortheanalysis. FactorInformation Factor Levels Values Factor 3 SloppyA,SloppyB,SloppyC AnalysisofVariance Source DF AdjSS AdjMS F-Value P-Value Factor 2 12.00 6.000 1.96 0.176 Error 15 45.98 3.065     Total 17 57.98       ModelSummary S R-sq R-sq(adj) R-sq(pred) 1.75073 20.70% 10.12% 0.00% Thep-valueforthistestisratherlarge,0.176.Withasignificancelevelof,say5%,wewouldfailtorejectthenullhypothesisandconcludethatthereisnotenoughevidenceinthedatatosuggestthemeantarcontentforthethreebrandsaredifferent. Comparison Theone-wayANOVAshowedstatisticallysignificantresultsforLabPrecisebutnotforLabSloppy.RecallthatANOVAcomparesthewithinvariationandthebetweenvariation.ForLabPrecise,thewithinvariationwassmallcomparedtothebetweenvariation.ThisresultedinalargeF-statistic(65.46)andthusasmallp-value.ForLabSloppy,thisratiowassmall(1.96),resultinginalargep-value. Tryit! Section  20youngpigsareassignedatrandomamong4experimentalgroups.Eachgroupisfedadifferentdiet.(Thisdesignisacompletelyrandomizeddesign.)Thedataarethepig'sweight,inkilograms,afterbeingraisedonthesedietsfor10months(pig_weights.txt).Wewishtodeterminewhetherthemeanpigweightsarethesameforall4diets. Answer First,wesetupourhypothesistest: \(H_0\colon\mu_1=\mu_2=\mu_3=\mu_4\) \(H_a\colon\text{atleastonemeanweightisdifferent}\) Herearethedatathatwereobtainedfromthefourexperimentalgroups,aswellas,theirsummarystatistics: Feed1 Feed2 Feed3 Feed4 60.8 68.3 102.6 87.9 57.1 67.7 102.2 84.7 65.0 74.0 100.5 83.2 58.7 66.3 97.5 85.8 61.8 69.9 98.9 90.3 OutputfromMinitab: DescriptiveStatistics:Feed1,Feed2,Feed3,Feed4 Statistics Variable N N* Mean StDev Minimum Maximum Feed1 5 0 60.68 3.03 57.10 65.00 Feed2 5 0 69.24 2.96 66.30 74.00 Feed3 5 0 100.34 2.16 97.50 102.60 Feed4 5 0 86.38 2.78 83.20 90.30 Thesmalleststandarddeviationis2.16,andthelargestis3.03.Sincetheruleofthumbissatisfiedhere,wecansaytheequalvarianceassumptionisnotviolated.Thedescriptionsuggeststhatthesamplesareindependent.Thereisnothinginthedescriptiontosuggesttheweightscomefromanormaldistribution.Thenormalprobabilityplotsare: Therearenoobviousviolationsfromthenormalassumption,butweshouldproceedwithcautionasthesamplesizesareverysmall. TheANOVAoutputis: One-wayANOVA:Feed1,Feed2,Feed3,Feed4 Method NullHypothesis Allmeansareequal AlternativeHypothesis Notallmeansareequal SignificanceLevel \(\alpha\)=0.05 Equalvarianceswereassumedfortheanalysis. FactorInformation Factor Levels Values Factor 4 Feed1,Feed2,Feed3,Feed4 AnalysisofVariance Source DF AdjSS AdjMS F-Value P-Value Factor 3 4703.2 1567.73 206.72 0.000 Error 16 121.3 7.58     Total 19 4824.5       ModelSummary S R-sq R-sq(adj) R-sq(pred) 2.75386 97.48% 97.01% 96.07% Thep-valueforthetestislessthan0.001.Withasignificancelevelof5%,werejectthenullhypothesis.Thedataprovidesufficientevidencetoconcludethatthemeanweightsofpigsfromthefourfeedsarenotallthesame. Witharejectionofthenullhypothesisleadingustoconcludethatnotallthemeansareequal(i.e.,atleastthemeanpigweightoronedietdiffersfromthemeanpigweightfromtheotherdiets)somefollowupquestionsare: "Whichdiettyperesultsindifferentaveragepigweights?",and "Isthereoneparticulardiettypethatproducesthelargest/smallestmeanweight?" Toanswerthesequestionsweanalyzethemultiplecomparisonoutput(thegroupinginformation)andtheintervalgraph. Means Factor N Mean StDev 95%CI Feed1 5 60.68 3.03 (58.07,63.29) Feed2 5 69.24 2.96 (66.63,71.85) Feed3 5 100.340 2.164 (97.729,102.951) Feed4 5 86.38 2.78 (83.77,88.99) PooledStDev=2.75386 TukeyPairwiseComparisons GroupingInformationUsingtheTukeyMethodand95%Confidence Factor N Mean Grouping Feed3 5 100.340 A       Feed4 5 86.38   B     Feed2 5 69.24     C   Feed1 5 60.68       D Meansthatdonotsharealetteraresignificantlydifferent. Eachofthesefactorlevelsareassociatedwithagroupingletter.Ifanyfactorlevelshavethesameletter,thenthemultiplecomparisonmethoddidnotdetermineasignificantdifferencebetweenthemeanresponse.Foranyfactorlevelthatdoesnotsharealetter,asignificantmeandifferencewasidentified.FromtheletteringweseeeachDietTypehasadifferentletter,i.e.notwogroupssharealetter.Therefore,wecanconcludethatallfourdietsresultedinstatisticallysignificantdifferentmeanpigweights.Furthermore,withtheorderofthemeansalsoprovidedfromhighesttolowest,wecansaythatFeed3resultedinthehighestmeanweightfollowedbyFeed4,thenFeed2,thenFeed1.Thisgroupingresultissupportedbythegraphoftheintervals. Inanalyzingtheintervals,wereflectbackonourlessonincomparingtwomeans:ifanintervalcontainedzero,wecouldnotconcludeadifferencebetweenthetwomeans;iftheintervaldidnotcontainzero,thenadifferencebetweenthetwomeanswassupported.Withfourfactorlevels,therearesixpossiblepairwisecomparisons.(Rememberthebinomialformulawherewehadthecounterforthenumberofpossibleoutcomes?Inthiscase\(4\choose2\)=6).Ininspectingeachofthesesixintervals,wefindthatallsixdoNOTincludezero.Therefore,thereisastatisticaldifferencebetweenallfourgroupmeans;thefourtypesofdietresultedinsignificantlydifferentmeanpigweights. «Previous10.2.2-TheANOVATable Next10.4-Two-WayANOVA» Lessons Lesson0:Overview 0.1-WhatisStatistics? 0.2-Foundations 0.3-IntroductiontoMinitab Lesson1:CollectingandSummarizingData 1.1-CollectingData 1.1.1-TypesofBias 1.1.2-StrategiesforCollectingData 1.1.3-TypesofStudies 1.1.4-Variables 1.1.5-PrinciplesofExperimentalDesign 1.2-ClassifyingData 1.3-SummarizingOneQualitativeVariable 1.4-GraphingOneQualitativeVariable 1.4.1-Minitab:GraphingOneQualitativeVariable 1.5-SummarizingOneQuantitativeVariable 1.5.1-MeasuresofCentralTendency 1.5.2-MeasuresofPosition 1.5.3-MeasuresofVariability 1.5.4-Minitab:DescriptiveStatistics 1.6-GraphingOneQuantitativeVariable 1.6.1-Dotplots,Stem-and-LeafDiagrams 1.6.2-Histograms 1.6.3-Boxplots 1.7-Lesson1Summary Lesson2:Probability 2.1-Notation 2.2-SetNotationandOperations 2.3-InterpretationsofProbability 2.4-ProbabilityProperties 2.5-ConditionalProbability 2.6-IndependentEvents 2.7-Bayes'Theorem 2.8-Lesson2Summary Lesson3:ProbabilityDistributions 3.1-RandomVariables 3.2-DiscreteProbabilityDistributions 3.2.1-ExpectedValueandVarianceofaDiscreteRandomVariable 3.2.2-BinomialRandomVariables 3.2.3-Minitab:BinomialDistributions 3.3-ContinuousProbabilityDistributions 3.3.1-TheNormalDistribution 3.3.2-TheStandardNormalDistribution 3.3.3-ProbabilitiesforNormalRandomVariables(Z-scores) 3.3.4-TheEmpiricalRule 3.3.5-OtherContinuousDistributions 3.4-Lesson3Summary Lesson4:SamplingDistributions 4.1-SamplingDistributionoftheSampleMean 4.1.1-PopulationisNormal 4.1.2-PopulationisNotNormal 4.2-SamplingDistributionoftheSampleProportion 4.2.1-NormalApproximationtotheBinomial 4.2.2-SamplingDistributionoftheSampleProportion 4.3-Lesson4Summary Lesson5:ConfidenceIntervals 5.1-IntroductiontoInferences 5.2-EstimationandConfidenceIntervals 5.3-InferenceforthePopulationProportion 5.3.1-ConstructandInterprettheCI 5.3.2-InterpretingtheCI 5.3.3-SampleSizeComputation 5.4-InferenceforthePopulationMean 5.4.1-ConstructandInterprettheCI 5.4.2-Thet-distribution 5.4.3-Example 5.4.4-CheckingNormality 5.4.5-SampleSizeComputation 5.5-Lesson5Summary Lesson6a:HypothesisTestingforOne-SampleProportion 6a.1-IntroductiontoHypothesisTesting 6a.2-StepsforHypothesisTests 6a.3-Set-UpforOne-SampleHypotheses 6a.4-HypothesisTestforOne-SampleProportion 6a.4.1-MakingaDecision 6a.4.2-MoreontheP-ValueandRejectionRegionApproach 6a.4.3-StepsinConductingaHypothesisTestfor\(p\) 6a.5-RelatingtheCItoaTwo-TailedTest 6a.6-Minitab:One-Sample\(p\)HypothesisTesting 6a.7-Lesson6aSummary Lesson6b:HypothesisTestingforOne-SampleMean 6b.1-StepsinConductingaHypothesisTestfor\(\mu\) 6b.2-Minitab:One-SampleMeanHypothesisTest 6b.3-FurtherConsiderationsforHypothesisTesting 6b.4-MoreExamples 6b.5-Lesson6bSummary Lesson7:ComparingTwoPopulationParameters 7.1-DifferenceofTwoIndependentNormalVariables 7.2-ComparingTwoPopulationProportions 7.2.1-ConfidenceIntervals 7.2.2-HypothesisTesting 7.3-ComparingTwoPopulationMeans 7.3.1-InferenceforIndependentMeans 7.3.1.1-PooledVariances 7.3.1.2-UnpooledVariances 7.3.2-InferenceforPairedMeans 7.4-ComparingTwoPopulationVariances 7.5-Lesson7Summary Lesson8:Chi-SquareTestforIndependence 8.1-TheChi-SquareTestofIndependence 8.2-The2x2Table:Testof2IndependentProportions 8.3-Risk,RelativeRiskandOdds 8.4-Lesson8Summary Lesson9:LinearRegressionFoundations 9.1-LinearRelationships 9.1.1-Scatterplots 9.1.2-Correlation 9.2-SimpleLinearRegression 9.2.1-TheSLRModel 9.2.2-InterpretingtheCoefficients 9.2.3-AssumptionsfortheSLRModel 9.2.4-InferencesaboutthePopulationSlope 9.2.5-OtherInferencesandConsiderations 9.2.6-Examples 9.3-CoefficientofDetermination 9.4-InferenceforCorrelation 9.4.1-HypothesisTestingforthePopulationCorrelation 9.4.2-ComparingCorrelationandSlope 9.5-MultipleRegressionModel 9.6-Lesson9Summary Lesson10:IntroductiontoANOVA 10.1-IntroductiontoAnalysisofVariance 10.2-AStatisticalTestforOne-WayANOVA 10.2.1-ANOVAAssumptions 10.2.2-TheANOVATable 10.3-MultipleComparisons 10.4-Two-WayANOVA 10.5-Summary Lesson11:IntroductiontoNonparametricTestsandBootstrap 11.1-InferenceforthePopulationMedian 11.1.1-TheSignTest 11.1.2-One-SampleWilcoxon 11.1.3-OtherNonparametricTests 11.2-IntroductiontoBootstrapping 11.2.1-BootstrappingMethods 11.3-Summary Lesson12:SummaryandReview 12.1-SummaryofStatisticalTechniques 12.2-ChoosetheCorrectStatisticalTechnique × Savechanges Close