Microsoft SQL Server Analysis Services Multidimensional Performance and Operations Guide

Thomas Kejser and Denny Lee

Contributors and Technical Reviewers: Peter Adshead (UBS), T.K. Anand, KaganArca, Andrew Calvett (UBS), Brad Daniels, John Desch, Marius Dumitru, WillfriedFrber (Trivadis), Alberto Ferrari (SQLBI), Marcel Franke (pmOne), Greg Galloway (Artis Consulting), Darren Gosbell (James & Monroe), DaeSeong Han, Siva Harinath, Thomas Ivarsson (Sigma AB), Alejandro Leguizamo (SolidQ), Alexei Khalyako, Edward Melomed, AkshaiMirchandani, Sanjay Nayyar (IM Group), TomislavPiasevoli, Carl Rabeler (SolidQ), Marco Russo (SQLBI), Ashvini Sharma, Didier Simon, John Sirmon, Richard Tkachuk, Andrea Uggetti, Elizabeth Vitt, Mike Vovchik, Christopher Webb (Crossjoin Consulting), SedatYogurtcuoglu, Anne Zorner

Summary: Download this book to learn about Analysis Services Multidimensional performance tuning from an operational and development perspective. This book consolidates the previously published SQL Server 2008 R2 Analysis Services Operations Guide and SQL Server 2008 R2 Analysis Services Performance Guide into a single publication that you can view on portable devices. Category: Guide Applies to: SQL Server 2005, SQL Server 2008, SQL Server 2008 R2, SQL Server 2012 Source: White paper (link to source content, link to source content) E-book publication date: May 2012 200 pages

This page intentionally left blank

Copyright 2012 by Microsoft Corporation All rights reserved. No part of the contents of this book may be reproduced or transmitted in any form or by any means without the written permission of the publisher.

Microsoft and the trademarks listed at http://www.microsoft.com/about/legal/en/us/IntellectualProperty/Trademarks/EN-US.aspx are trademarks of the Microsoft group of companies. All other marks are property of their respective owners. The example companies, organizations, products, domain names, email addresses, logos, people, places, and events depicted herein are fictitious. No association with any real company, organization, product, domain name, email address, logo, person, place, or event is intended or should be inferred. This book expresses the authors views and opinions. The information contained in this book is provided without any express, statutory, or implied warranties. Neither the authors, Microsoft Corporation, nor its resellers, or distributors will be held liable for any damages caused or alleged to be caused either directly or indirectly by this book.

Contents
1 2 Introduction..........................................................................................................................................5 Part1:BuildingaHighPerformanceCube...........................................................................................6 2.1 2.2 2.3 2.4 2.5 3 DesignPatternsforScalableCubes...........................................................................................6 TestingAnalysisServicesCubes..............................................................................................32 TuningQueryPerformance.....................................................................................................39 TuningProcessingPerformance..............................................................................................76 SpecialConsiderations............................................................................................................93

Part2:RunningaCubeinProduction...............................................................................................105 3.1 3.2 3.3 3.4 3.5 3.6 3.7 ConfiguringtheServer ..........................................................................................................106 . MonitoringandTuningtheServer........................................................................................133 SecurityandAuditing............................................................................................................143 HighAvailabilityandDisasterRecovery................................................................................147 DiagnosingandOptimizing....................................................................................................150 ServerMaintenance..............................................................................................................189 SpecialConsiderations..........................................................................................................192

Conclusion.........................................................................................................................................200

Sendfeedback...........................................................................................................................................200

1 Introduction
ThisbookconsolidatestwopreviouslypublishedguidesintooneessentialresourceforAnalysisServices developersandoperationspersonnel.AlthoughthetitlesoftheoriginalpublicationsindicateSQLServer 2008R2,mostoftheknowledgethatyougainfromthisbookiseasilytransferredtootherversionsof AnalysisServices,includingmultidimensionalmodelsbuiltusingSQLServer2012. Part1isfromtheSQLServer2008R2AnalysisServicesPerformanceGuide.PublishedinOctober 2011,thisguidewascreatedfordevelopersandcubedesignerswhowanttobuildhighperformance cubesusingbestpracticesandinsightslearnedfromrealworlddevelopmentprojects.InPart1,youll learnproventechniquesforbuildingsolutionsthatarefastertoprocessandquery,minimizingtheneed forfurthertuningdowntheroad. Part2isfromtheSQLServer2008R2AnalysisServicesOperationsGuide.Thisguide,publishedinJune 2011,isintendedfordevelopersandoperationsspecialistswhomanagesolutionsthatarealreadyin production.Part2showsyouhowtoextractperformancegainsfromaproductioncube,including changingserverandsystemproperties,andperformingsystemmaintenancethathelpyouavoid problemsbeforetheystart. Whileeachguidetargetsadifferentpartofasolutionlifecycle,havingbothinasingleportableformat givesyouanintellectualtoolkitthatyoucanaccessonmobiledeviceswhereveryoumaybe.Wehope youfindthisbookhelpfulandeasytouse,butitisonlyoneofseveralformatsavailableforthiscontent. YoucanalsogetprintableversionsofbothguidesbydownloadingthemfromtheMicrosoftwebsite.

2 Part 1: Building a High-Performance Cube

ThissectionprovidesinformationaboutbuildingandtuningAnalysisServicescubesforthebestpossible performance.Itisprimarilyaimedatbusinessintelligence(BI)developerswhoarebuildinganewcube fromscratchoroptimizinganexistingcubeforbetterperformance. Thegoalofthissectionistoprovideyouwiththenecessarybackgroundtounderstanddesigntradeoffs andwithtechniquesanddesignpatternsthatwillhelpyouachievethebestpossibleperformanceof evenlargecubes. Cubeperformancecanbedividedintotwotypesofworkload:queryperformanceandprocessing performance.Becausetheseworkloadsareverydifferent,thissectionisorganizedintofourmain groups. DesignPatternsforScalableCubesNoamountofquerytuningandoptimizationcanbeatthebenefits ofawelldesigneddatamodel.Thissectioncontainsguidancetohelpyougetthedesignrightthefirst time.Ingeneral,goodcubedesignfollowsKimballmodelingtechniques,andifyouavoidsometypical designmistakes,youareinverygoodshape. TestingAnalysisServicesCubesIneveryITproject,preproductiontestingisacrucialpartofthe developmentanddeploymentcycle.Evenwiththemostcarefuldesign,testingwillstillbeabletoshake outerrorsandavoidproductionissues.Designingandrunningatestrunofanenterprisecubeistime wellinvested.Hence,thissectionincludesadescriptionofthetestmethodsavailabletoyou. TuningQueryPerformanceQueryperformancedirectlyimpactsthequalityoftheenduser experience.Assuch,itistheprimarybenchmarkusedtoevaluatethesuccessofanonlineanalytical processing(OLAP)implementation.AnalysisServicesprovidesavarietyofmechanismstoaccelerate queryperformance,includingaggregations,caching,andindexeddataretrieval.Thissectionalso providesguidanceonwritingefficientMultidimensionalExpressions(MDX)calculationscripts. TuningProcessingPerformanceProcessingistheoperationthatrefreshesdatainanAnalysisServices database.Thefastertheprocessingperformance,thesooneruserscanaccessrefresheddata.Analysis Servicesprovidesavarietyofmechanismsthatyoucanusetoinfluenceprocessingperformance, includingparallelizedprocessingdesigns,relationaltuning,andaneconomicalprocessingstrategy(for example,incrementalversusfullrefreshversusproactivecaching). SpecialConsiderationsSomefeaturesofAnalysisServicessuchasdistinctcountmeasuresandmany tomanydimensionsrequiremorecarefulattentiontothecubedesignthanothers.AttheendofPart1, youwillfindasectionthatdescribesthespecialtechniquesyoushouldapplywhenusingthesefeatures.

2.1 Design Patterns for Scalable Cubes

CubespresentauniquechallengetotheBIdeveloper:theyareadhocdatabasesthatareexpectedto respondtomostqueriesinshorttime.Thefreedomoftheenduserislimitedonlybythedatamodel youimplement.Achievingabalancebetweenuserfreedomandscalabledesignwilldeterminethe 6

successofacube.Eachindustryhasspecificdesignpatternsthatlendthemselveswelltovalueadding reportingandadetailedtreatmentofoptimal,industryspecificdatamodelisoutsidethescopeofthis book.However,therearealotofcommondesignpatternsyoucanapplyacrossallindustriesthis sectiondealswiththesepatternsandhowyoucanleveragethemforincreasedscalabilityinyourcube design.

2.1.1 Building Optimal Dimensions

AwelltuneddimensiondesignisoneofthemostcriticalsuccessfactorsofahighperformingAnalysis Servicessolution.Thedimensionsofthecubearethefirststopfordataanalysisandtheirdesignhasa deepimpactontheperformanceofallmeasuresinthecube. Dimensionsarecomposedofattributes,whicharerelatedtoeachotherthroughhierarchies.Efficient useofattributesisakeydesignskilltomaster,andstudyingandimplementingtheattribute relationshipsavailableinthebusinessmodelcanhelpimprovecubeperformance. Inthissection,youwillfindguidanceonbuildingoptimizeddimensionsandproperlyusingboth attributesandhierarchies.

2.1.1.1 Using the KeyColumns, ValueColumn, and NameColumn Properties Effectively

Whenyouaddanewattributetoadimension,threepropertiesareusedtodefinetheattribute.The KeyColumnspropertyspecifiesoneormoresourcefieldsthatuniquelyidentifyeachinstanceofthe attribute. TheNameColumnpropertyspecifiesthesourcefieldthatwillbedisplayedtoendusers.Ifyoudonot specifyavaluefortheNameColumnproperty,itisautomaticallysettothevalueoftheKeyColumns property. ValueColumnallowsyoutocarryfurtherinformationabouttheattributetypicallyusedfor calculations.Unlikememberproperties,thispropertyofanattributeisstronglytypedproviding increasedperformancewhenitisusedincalculations.Thecontentsofthispropertycanbeaccessed throughtheMemberValueMDXfunction. UsingbothValueColumnandNameColumntocarryinformationeliminatestheneedforextraneous attributes.Thisreducesthetotalnumberofattributesinyourdesign,makingitmoreefficient. Itisabestpracticetoassignanumericsourcefield,ifavailable,totheKeyColumnspropertyratherthan astringproperty.Furthermore,useasinglecolumnkeyinsteadofacomposite,multicolumnkey.Not onlydothesepracticesthisreduceprocessingtime,theyalsoreducethesizeofthedimensionandthe likelihoodofusererrors.Thisisespeciallytrueforattributesthathavealargenumberofmembers,that is,greaterthanonemillionmembers.

2.1.1.2 Hiding Attribute Hierarchies

Formanydimensions,youwillwanttheusertonavigatehierarchiescreatedforeaseofaccess.For example,acustomerdimensioncouldbenavigatedbydrillingintocountryandcitybeforereachingthe customername,orbydrillingthroughagegroupsorincomelevels.Suchhierarchies,coveredinmore detaillater,makenavigationofthecubeeasierandmakequeriesmoreefficient. Inadditiontouserhierarchies,AnalysisServicesbydefaultcreatesaflathierarchyforeveryattributein adimensiontheseareattributehierarchies.Hidingattributehierarchiesisoftenagoodidea,because alotofhierarchiesinasingledimensionwilltypicallyconfuseusersandmakeclientqueriesless efficient.ConsidersettingAttributeHierarchyVisible=falseformostattributehierarchiesanduseuser hierarchiesinstead. 2.1.1.2.1 Hiding the Surrogate Key Itisoftenagoodideatohidethesurrogatekeyattributeinthedimension.Ifyouexposethesurrogate keytotheclienttoolsasaValueColumn,thosetoolsmayrefertothekeyvaluesinreports.The surrogatekeyinaKimballstarschemadesignholdsnobusinessinformation,andmayevenchangeif youremodeltype2history.Afteryoucreateadependencytothekeyintheclienttools,youcannot changethekeywithoutbreakingreports.Becauseofthis,youdontwantenduserreportsreferringto thesurrogatekeydirectlyandthisiswhywerecommendhidingit. Thebestdesignforasurrogatekeyistohideitfromusersinthedimensiondesignbysettingthe AttributeHierarchyVisible=falseandbynotincludingtheattributeinanyuserhierarchies.This preventsendusertoolsfromreferencingthesurrogatekey,leavingyoufreetochangethekeyvalueif requirementschange.

2.1.1.3 Setting or Disabling Ordering of Attributes

Inmostcases,youwantanattributetohaveanexplicitordering.Forexample,youwillwantaCity attributetobesortedalphabetically.YoushouldexplicitlysettheOrderByorOrderByAttribute propertyoftheattributetoexplicitlycontrolthisordering.Typically,thisorderingisbyattributename orkey,butitmayalsobeanotherattribute.Ifyouincludeanattributeonlyforthepurposeofordering anotherattribute,makesureyousetAttributeHierarchyEnabled=falseand AttributeHierarchyOptimizedState=NotOptimizedtosaveonprocessingoperations. Therearefewcaseswhereyoudontcareabouttheorderingofanattribute,yetthesurrogatekeyis onesuchcase.Forsuchhiddenattributethatyouusedonlyforimplementationpurposes,youcanset AttributeHierarchyOrdered=falsetosavetimeduringprocessingofthedimension.

2.1.1.4 Setting Default Attribute Members

Anyquerythatdoesnotexplicitlyreferenceahierarchywillusethecurrentmemberofthat hierarchy.ThedefaultbehaviorofAnalysisServicesistoassigntheAllmemberofadimensionasthe defaultmember,whichisnormallythedesiredbehavior.Butforsomeattributes,suchasthecurrent

dayinadatedimension,itsometimesmakessensetoexplicitlyassignadefaultmember.Forexample, youmaysetadefaultdateintheAdventureWorkscubelikethis.
ALTERCUBE [Adventure Works]UPDATE DIMENSION [Date], DEFAULT_MEMBER='[Date].[Date].&[2000]'

However,defaultmembersmaycauseissuesintheclienttool.Forexample,MicrosoftExcel2010will notprovideavisualindicationthatadefaultmemberiscurrentlyselectedandhenceimplicitlyinfluence thequeryresult.ThismayconfuseuserswhoexpecttheAllleveltobethecurrentmemberwhenno othermembersareimpliedbythequery.Also,ifyousetadefaultmemberinadimensionwithmultiple hierarchies,youwilltypicallygetresultsthatarehardforuserstointerpret. Ingeneral,preferexplicitlydefaultmembersonlyondimensionswithsinglehierarchiesorinhierarchies thatdonothaveanAlllevel.

2.1.1.5 Removing the All Level

MostdimensionsrolluptoacommonAlllevel,whichistheaggregationofalldescendants.Butthere aresomeexceptionswhereisdoesnotmakesensetoqueryattheAlllevel.Forexample,youmayhave acurrencydimensioninthecubeandaskingforthesumofallcurrenciesisameaninglessquestion. ItcanevenbeexpensivetoaskfortheAlllevelofdimensionifthereisnotgoodaggregatetorespondto thequery.Forexample,ifyouhaveacubepartitionedbycurrency,askingfortheAlllevelofcurrency willcauseascanofallpartitions,whichcouldbeexpensiveandleadtoauselessresult. InordertopreventusersfromqueryingmeaninglessAlllevels,youcandisabletheAllmemberina hierarchy.YoudothisbysettingtheIsAggregateable=falseontheattributeatthetopofthehierarchy. NotethatifyoudisabletheAlllevel,youshouldalsosetadefaultmemberasdescribedintheprevious sectionifyoudont,AnalysisServiceswillchooseoneforyou.

2.1.1.6 Identifying Attribute Relationships

Attributerelationshipsdefinehierarchicaldependenciesbetweenattributes.Inotherwords,ifAhasa relatedattributeB,writtenAB,thereisonememberinBforeverymemberinA,andmanymembers inAforagivenmemberinB.Forexample,givenanattributerelationshipCityState,ifthecurrent cityisSeattle,weknowtheStatemustbeWashington. Often,therearerelationshipsbetweenattributesthatmightormightnotbemanifestedintheoriginal dimensiontablethatcanbeusedbytheAnalysisServicesenginetooptimizeperformance.Bydefault, allattributesarerelatedtothekey,andtheattributerelationshipdiagramrepresentsabushwhere relationshipsallstemfromthekeyattributeandendateachothersattribute.

Figure 1 Bushy attribute relationship 11: a ps

Youcano optimizeperfo ormancebyd defininghierarchicalrelatio onshipssupportedbythedata.Inthisc case, amodeln nameidentifie estheproduc ctlineandsubcategory,an ndthesubcat tegoryidentif fiesacategor ry.In otherwor rds,asingles subcategoryis snotfoundin nmorethano onecategory.Ifyouredefi inethe relationsh hipsintheatt tributerelatio onshipeditor, ,therelationshipsareclea arer.

Figure 2 Redefin 22: ned attribu relation ute nships

Attributerelationships shelpperform manceinthre eesignificantways: sbetweenlev velsinthehie erarchydono otneedtogothroughthekeyattribute.This Crossproducts avesCPUtime eduringquer ries. sa Aggregationsb builtonattrib butescanber reusedforqu ueriesonrelat tedattributes s.Thissaves esourcesduringprocessing gandforque eries. re AutoExistcanmoreefficientlyeliminate eattributeco ombinationst thatdonotex xistinthedat ta.

Considert thecrosspro oductbetweenSubcategor ryandCatego oryinthetwo ofigures.Inthefirst,wher reno attributerelationshipshavebeenexplicitlydefin ned,theengin nemustfirstf findwhichpr roductsarein n 10

eachsubcategoryandthendeterminewhichcategorieseachoftheseproductsbelongsto.Forlarge dimensions,thiscantakealongtime.Iftheattributerelationshipisdefined,theAnalysisServices engineknowsbeforehandwhichcategoryeachsubcategorybelongstoviaindexesbuiltatprocesstime. 2.1.1.6.1 Flexible vs. Rigid Relationships Whenanattributerelationshipisdefined,therelationcaneitherbeflexibleorrigid.Aflexibleattribute relationshipisonewherememberscanmovearoundduringdimensionupdates,andarigidattribute relationshipisonewherethememberrelationshipsareguaranteedtobefixed.Forexample,the relationshipbetweenmonthandyearisfixedbecauseaparticularmonthisntgoingtochangeitsyear whenthedimensionisreprocessed.However,therelationshipbetweencustomerandcitymaybe flexibleascustomersmove. Whenachangeisdetectedduringprocessinaflexiblerelationship,allindexesforpartitionsreferencing theaffecteddimension(includingtheindexesforattributethatarenotaffected)mustbeinvalidated. ThisisanexpensiveoperationandmaycauseProcessUpdateoperationstotakeaverylongtime. IndexesinvalidatedbychangesinflexiblerelationshipsmustberebuiltafteraProcessUpdateoperation withaProcessIndexontheaffectedpartitions;thisaddsevenmoretimetocubeprocessing. Flexiblerelationshipsarethedefaultsetting.Carefullyconsidertheadvantagesofrigidrelationshipsand changethedefaultwherethedesignallowsit.

2.1.1.7 Using Hierarchies Effectively

AnalysisServicesenablesyoutobuildtwotypesofuserhierarchies:naturalandunnaturalhierarchies. Eachtypehasdifferentdesignandperformancecharacteristics. Inanaturalhierarchy,allattributesparticipatingaslevelsinthehierarchyhavedirectorindirect attributerelationshipsfromthebottomofthehierarchytothetopofthehierarchy. Inanunnaturalhierarchy,thehierarchyconsistsofatleasttwoconsecutivelevelsthathavenoattribute relationships.Typicallythesehierarchiesareusedtocreatedrilldownpathsofcommonlyviewed attributesthatdonotfollowanynaturalhierarchy.Forexample,usersmaywanttoviewahierarchyof GenderandEducation.

Figure 33: Natural and unnatural hierarchies

11

Fromaperformanceperspective,naturalhierarchiesbehaveverydifferentlythanunnaturalhierarchies do.Innaturalhierarchies,thehierarchytreeismaterializedondiskinhierarchystores.Inaddition,all attributesparticipatinginnaturalhierarchiesareautomaticallyconsideredtobeaggregationcandidates. Unnaturalhierarchiesarenotmaterializedondisk,andtheattributesparticipatinginunnatural hierarchiesarenotautomaticallyconsideredasaggregationcandidates.Rather,theysimplyprovide userswitheasytousedrilldownpathsforcommonlyviewedattributesthatdonothavenatural relationships.Byassemblingtheseattributesintohierarchies,youcanalsouseavarietyofMDX navigationfunctionstoeasilyperformcalculationslikepercentofparent. Totakeadvantageofnaturalhierarchies,definecascadingattributerelationshipsforallattributesthat participateinthehierarchy.

2.1.1.8 Turning Off the Attribute Hierarchy

Memberpropertiesprovideadifferentmechanismtoexposedimensioninformation.Foragiven attribute,memberpropertiesareautomaticallycreatedforeverydirectattributerelationship.Forthe primarykeyattribute,thismeansthateveryattributethatisdirectlyrelatedtotheprimarykeyis availableasamemberpropertyoftheprimarykeyattribute. Ifyouonlywanttoaccessanattributeasmemberproperty,afteryouverifythatthecorrectrelationship isinplace,youcandisabletheattributeshierarchybysettingtheAttributeHierarchyEnabledproperty toFalse.Fromaprocessingperspective,disablingtheattributehierarchycanimproveperformanceand decreasecubesizebecausetheattributewillnolongerbeindexedoraggregated.Thiscanbeespecially usefulforhighcardinalityattributesthathaveaonetoonerelationshipwiththeprimarykey.High cardinalityattributessuchasphonenumbersandaddressestypicallydonotrequiresliceanddice analysis.Bydisablingthehierarchiesfortheseattributesandaccessingthemviamemberproperties, youcansaveprocessingtimeandreducecubesize. Decidingwhethertodisabletheattributeshierarchyrequiresthatyouconsiderboththequeryingand processingimpactsofusingmemberproperties.Memberpropertiescannotbeplacedonaqueryaxisin anMDXqueryinthesamemannerasattributehierarchiesanduserhierarchies.Toqueryamember property,youmustquerytheattributethatcontainsthatmemberproperty. Forexample,ifyourequiretheworkphonenumberforacustomer,youmustquerythepropertiesof customerandthenrequestthephonenumberproperty.Asaconvenience,mostfrontendtoolseasily displaymemberpropertiesintheiruserinterfaces. Ingeneral,filteringmeasuresusingmemberpropertiesisslowerthanfilteringusingattribute hierarchies,becausememberpropertiesarenotindexedanddonotparticipateinaggregations.The actualimpacttoqueryperformancedependsonhowyouusetheattribute. Forexample,ifyouruserswanttosliceanddicedatabybothaccountnumberandaccountdescription, fromaqueryingperspectiveyoumaybebetteroffhavingtheattributehierarchiesinplaceand removingthebitmapindexesifprocessingperformanceisanissue. 12

2.1.1.9 Reference Dimensions

Referencedimensionsallowyoutobuildadimensionalmodelontopofasnowflakerelationaldesign. Whilethisisapowerfulfeature,youshouldunderstandtheimplicationsofusingit. Bydefault,areferencedimensionisnonmaterialized.Thismeansthatquerieshavetoperformthejoin betweenthereferenceandtheouterdimensiontableatquerytime.Also,filtersdefinedonattributesin theouterdimensiontablearenotdrivenintothemeasuregroupwhenthebitmapstherearescanned. Thismayresultinreadingtoomuchdatafromdisktoansweruserqueries.Leavingadimensionasnon materializedprioritizesmodelingflexibilityoverqueryperformance.Considercarefullywhetheryoucan affordthistradeoff:cubesaretypicallyintendedtobefastadhocstructures,andputtingthe performanceburdenontheenduserisrarelyagoodidea. AnalysisServiceshastheabilitytomaterializethereferencesdimension.Whenyouenablethisoption, memoryanddiskstructuresarecreatedthatmakethedimensionbehavejustlikeadenormalizedstar schema.Thismeansthatyouwillretainalltheperformancebenefitsofaregular,nonreference dimension.However,becarefulwithmaterializedreferencedimensionifyourunaprocessupdateon theintermediatedimension,anychangesintherelationshipsbetweentheouterdimensionandthe referencewillnotbereflectedinthecube.Instead,theoriginalrelationshipbetweentheouter dimensionandthemeasuregroupisretainedwhichismostlikelynotthedesiredresult.Inaway,you canconsiderthereferencetabletobearigidrelationshiptoattributesintheouterattributes.Theonly waytoreflectchangesinthereferencetableistofullyprocessthedimension.

2.1.1.10

Fast-Changing Attributes

Somedatamodelscontainattributesthatchangeveryfast.Dependingonwhichtypeofhistorytracking youneed,youmayfacedifferentchallenges. Type2FastChangingAttributesIfyoutrackeverychangetoafastchangingattribute,thismaycause thedimensioncontainingtheattributetogrowverylarge.Type2attributesaretypicallyaddedtoa dimensionwithaProcessAddcommand.Atsomepoint,runningProcessAddonalargedimensionand runningalltheconsistencycheckswilltakealongtime.Also,havingahugedimensionisunwieldy becauseuserswillhavetroublequeryingitandtheserverwillhavetroublekeepingitinmemory.A goodexampleofsuchamodelingchallengeistheageofacustomerthiswillchangeeveryyearand causethecustomerdimensiontogrowdramatically. Type1FastChangingAttributesEvenifyoudonottrackeverychangetotheattribute,youmaystill runintoissueswithfastchangingattributes.Toreflectachangeinthedatasourcetothecube,you havetorunProcessUpdateonthechangeddimension.Asthecubeanddimensiongrowslarger, runningProcessUpdatebecomesexpensive.Anexampleofsuchamodelingchallengeistotrackthe statusattributeofaserverinahostingenvironment(Running,Shutdown,Overloadedandsoon). Astatusattributelikethismaychangeseveraltimesperdayorevenperhour.Runningfrequent ProcessUpdatesonsuchadimensiontoreflectchangescanbeanexpensiveoperation,anditmaynot befeasiblewiththelockingimplementationofAnalysisServicesinaproductionenvironment. 13

Inthefollowingsections,wewilllookatsomemodelingoptionsyoucanusetoaddresstheseproblems. 2.1.1.10.1 Type 2 Fast-Changing Attributes Ifhistorytrackingisarequirementofafastchangingattribute,thebestoptionisoftentousethefact tabletotrackhistory.Thisisbestillustratedwithanexample.Consideragainthecustomerdimension withtheageattribute.ModelingtheAgeattributedirectlyinthecustomerdimensionproducesadesign likethis.

Figure 44: Age in customer dimension

NoticethateverytimeThomashasabirthday,anewrowisaddedinthedimensiontable.The alternativedesignapproachsplitsthecustomerdimensionintotwodimensionslikethis.

14


Figure 55: Age in its own dimension

Notethattherearesomerestrictionsonthesituationwherethisdesigncanbeapplied.Itworksbest whenthechangingattributetakesonasmall,distinctsetofvalues.Italsoaddscomplexitytothe design;byaddingmoredimensionstothemodel,itcreatesmoreworkfortheETLdeveloperswhenthe facttableisloaded.Also,considerthestorageimpactonthefacttable:Withthealternativedesign,the facttablebecomeswider,andmorebyteshavetobestoredperrow. 2.1.1.10.2 Type 1 Fast-Changing Attributes Yourbusinessrequirementmaybeupdatinganattributeofadimensionathighfrequency,daily,or evenhourly.Forasmallcube,runningProcessUpdatewillhelpyouaddressthisissue.Butasthecube growslarger,theruntimeofProcessUpdatecanbecometoolongforthebatchwindoworthereal timerequirementsofthecube(youcanreadmoreabouttuningprocessupdateintheprocessing section). Consideragaintheserverhostingexample:Youmaywanttotrackthestatus,whichchangesfrequently, ofallservers.Fortheexample,letussaythattheserverdimensionisusedbyafacttabletracking performancecounters.Assumeyouhavemodeledlikethis.

15


Figure 66: Status column in server dimension

TheproblemwiththismodelistheStatuscolumn.IftheFactCounterislargeandstatuschangesalot, ProcessUpdatewilltakeaverylongtimetorun.Tooptimize,considerthisdesigninstead.

Figure 77: Status column in its own dimension

IfyouimplementDimServerastheintermediatereferencetabletoDimServerStatus,AnalysisServices nolongerhastokeeptrackofthemetadataintheFactCounterwhenyourunProcessUpdateon DimServerStatus.Butasdescribedearlier,thismeansthatthejointoDimServerStatuswillhappenat runtime,increasingCPUcostandquerytimes.Italsomeansthatyoucannotindexattributesin DimServerbecausetheintermediatedimensionisnotmaterialized.Youhavetocarefullybalancethe tradeoffbetweenprocessingtimeandqueryspeeds.

16

2.1.1.11

Large Dimensions

InSQLServer2005,SQLServer2008,andSQLServer2008R2,AnalysisServiceshassomebuiltin limitationsthatlimitthesizeofthedimensionsyoucancreate.Firstofall,ittakestimetoupdatea dimensionthisisexpensivebecauseallindexesonfacttableshavetobeconsideredforinvalidation whenanattributechanges.Second,stringvaluesindimensionattributesarestoredonadiskstructure calledthestringstore.Thisstructurehasasizelimitationof4GB.Ifadimensioncontainsattributes wherethetotalsizeofthestringvalues(thisincludestranslations)exceeds4GB,youwillgetanerror duringprocessing.ThenextversionofSQLServerAnalysisServices,codenamedDenali,isexpectedto removethislimitation. Considerforamomentadimensionwithtensorevenhundredsofmillionsofmembers.Sucha dimensioncanbebuiltandaddedtoacube,evenonSQLServer2005,SQLServer2008,andSQLServer 2008R2.Butwhatdoessuchadimensionmeantoanadhocuser?Howwilltheusernavigateit?Which hierarchieswillgroupthemembersofthisdimensionintoreasonablesizesthatcanberenderedona screen?Whileitmaymakesenseforsomereportingpurposestosearchforindividualmembersinsuch adimension,itmaynotbetherightproblemtosolvewithacube. Whenyoubuildcubes,askyourself:isthisacubeproblem?Forexample,thinkofthistypicaltelco modelofcalldetailrecords.

Figure 88: Call detail records (CDRs)

Inthisparticularexample,thereare300millioncustomersinthedatamodel.Thereisnogoodwayto groupthesecustomersandallowadhocaccesstothecubeatreasonablespeeds.Evenifyoumanageto optimizethespaceusedtofitinthe4GBstringstore,howwouldusersbrowseacustomerdimension likethis? Ifyoufindyourselfinasituationwhereadimensionbecomestoolargeandunwieldy,considerbuilding thecubeontopofanaggregate.Forthetelcoexample,imagineatransformationlikethefollowing.

17


Figure 99: Cube built on aggregate

Usinganaggregatedfacttable,thisturnsa300millionrowdimensionprobleminto100,000row dimensionproblem.Youcanconsideraggregatingthefactstosavestoragetooalternatively,youcan addademographicskeydirectlytotheoriginalfacttable,processontopofthisdatasource,andrelyon MOLAPcompressiontoreducedatasizes.

2.1.2 Partitioning a Cube

Partitionsseparatemeasuregroupdataintophysicalstorageunits.Effectiveuseofpartitionscan enhancequeryperformance,improveprocessingperformance,andfacilitatedatamanagement.This sectionspecificallyaddresseshowyoucanusepartitionstoimprovequeryperformance.Youmustoften makeatradeoffbetweenqueryandprocessingperformanceinyourpartitioningstrategy. Youcanusemultiplepartitionstobreakupyourmeasuregroupintoseparatephysicalcomponents.The advantagesofpartitioningforimprovingqueryperformancearepartitioneliminationandaggregation design. PartitioneliminationPartitionsthatdonotcontaindatainthesubcubearenotqueriedatall,thus avoidingthecostofreadingtheindex(orscanningatableiftheserverisinROLAPmode).Whilereading apartitionindexandfindingnoavailablerowsisacheapoperation,asthenumberofconcurrentusers grows,thesereadsbegintoputastraininthethreadpool.Also,forqueriesthatdonothaveindexesto supportthem,AnalysisServiceswillhavetoscanallpotentiallymatchingpartitionsfordata. AggregationdesignEachpartitioncanhaveitsownorsharedaggregationdesign.Therefore,partitions queriedmoreoftenordifferentlycanhavetheirowndesigns.

18

Figure 1010: Intelligent querying by partitions

Figure10displaystheprofilertraceofqueryrequestingResellerSalesAmountbyBusinessTypefrom AdventureWorks.TheResellerSalesmeasuregroupoftheAdventureWorkscubecontainsfour partitions:oneforeachyear.Becausethequerysliceson2003,thestorageenginecangodirectlytothe 2003ResellerSalespartitionandignoreotherpartitions.

2.1.2.1 Partition Slicing

Partitionsareboundtoasourcetable,view,orsourcequery.Whentheformulaenginerequestsa subcube,thestorageenginelooksatthemetadataofpartitionfortherelevantmeasuregroup.Each partitionmaycontainaslicedefinition,ahighleveldescriptionoftheminimumandmaximumattribute DataIDsthatexistinthatdimension.Ifitcanbedeterminedfromtheslicedefinitionthattherequested subcubedataisnotpresentinthepartition,thatpartitionisignored.Iftheslicedefinitionismissingorif theinformationinthesliceindicatesthatrequireddataispresent,thepartitionisaccessedbyfirst lookingattheindexes(ifany)andthenscanningthepartitionsegments. Thesliceofapartitioncanbesetintwoways: AutoslicewhenAnalysisServicesreadsthedataduringprocessing,itkeepstrackofthe minimumandmaximumattributeDataIDreads.Thesevaluesareusedtosettheslicewhenthe indexesarebuiltonthepartition. ManualslicerTherearecaseswhereautoslicewillnotworkthesearedescribedinthenext section.Forthosesituations,youcanmanuallysettheslice.Manualslicesaretheonlyavailable sliceoptionforROLAPpartitionsandproactivecachingpartitions.

2.1.2.1.1 Auto Slice DuringprocessingofMOLAPpartitions,AnalysisServicesinternallyidentifiestherangeofdatathatis containedineachpartitionbyusingtheMinandMaxDataIDsofeachattributetocalculatetherangeof datathatiscontainedinthepartition.Thedatarangeforeachattributeisthencombinedtocreatethe slicedefinitionforthepartition.

19

TheMinandMaxDataIDscanspecifyaeitherasinglememberorarangeofmembers.Forexample, partitioningbyyearresultsinthesameMinandMaxDataIDslicefortheyearattribute,andqueriestoa specificmomentintimeonlyresultinpartitionqueriestothatyearspartition. ItisimportanttorememberthatthepartitionsliceismaintainedasarangeofDataIDsthatyouhaveno explicitcontrolover.DataIDsareassignedduringdimensionprocessingasnewmembersare encountered.BecauseAnalysisServicesjustlooksattheminimumandmaximumvalueoftheDataID, youcanendupreadingpartitionsthatdontcontainrelevantdata. Forexample:ifyouhaveapartition,P2003_4,thatcontainsboth2003and2004data,youarenot guaranteedthattheminimumandmaximumDataIDintheslidecontainvaluesnexttoeachother(even thoughtheyearsareadjacent).Inourexample,letussaytheDataIDfor2003is42andtheDataIDfor 2004is45.BecauseyoucannotcontrolwhichDataIDgetsassignedtowhichmembers,youcouldbeina situationwheretheDataIDfor2005is44.Whenauserrequestsdatafor2005,AnalysisServiceslooksat thesliceforP2003_4,seesthatitcontainsdataintheinterval42to45andthereforeconcludesthatthis partitionhastobescannedtomakesureitdoesnotcontainthevaluesforDataID44(because44is between42and45). Becauseofthisbehavior,autoslicetypicallyworksbestifthedatacontainedinthepartitionmapstoa singleattributevalue.Whenthatisthecase,themaximumandminimumDataIDcontainedintheslice willbeequalandtheslicewillworkefficiently. Notethattheautosliceisnotdefinedandindexesarenotbuiltforpartitionswithfewerrowsthan IndexBuildThreshold(whichhasadefaultvalueof4096). 2.1.2.1.2 Manually Setting Slices NometadataisavailabletoAnalysisServicesaboutthecontentofROLAPandproactivecaching partitions.Becauseofthis,youmustmanuallyidentifythesliceinthepropertiesofthepartition.Itisa bestpracticetomanuallysetslicesinROLAPandproactivecachingpartitions. However,asshownintheprevioussection,therearecaseswhereautoslicewillnotgiveyouthe desiredpartitioneliminationbehavior.Inthesecasesyoucanbenefitfromdefiningthesliceyourselffor MOLAPpartitions.Forexample,ifyoupartitionbyyearwithsomepartitionscontainingarangeofyears, definingthesliceexplicitlyavoidstheproblemofoverlappingDataIDs.Thiscanonlybedonewith knowledgeofthedatawhichiswhereyoucanaddsomeoptimizationasaBIdeveloper. Itisgenerallynotabestpracticetocreatepartitionsbeforeyouarereadytofillthemwithdata.Butfor realtimecubes,itissometimesagoodideatocreatepartitionsinadvancetoavoidlockingissues. Whenyoutakethisapproach,itisalsoagoodideatosetamanualsliceonMOLAPpartitionstomake surethestorageenginedoesnotspendtimescanningemptypartitions.

20

2.1.2.2 Partition Sizing

Fornondistinctcountmeasuregroups,testswithpartitionsizesintherangeof200MBtoupto3GB indicatethatpartitionsizealonedoesnothaveasubstantialimpactonqueryspeeds.Infact,wehave successfullydeployedgoodqueryperformanceonpartitionslargerthan3GB. Thefollowinggraphshowsfourdifferentqueryrunswithdifferentpartitionsizes(theverticalaxisis totalruntimeinhours).Performanceiscomparablebetweenpartitionsizesandisonlyaffectedbythe designofthesecurityfeaturesinthisparticularcustomercube.

Figure 1111: Throughput by partition size (higher is better)

Thepartitioningstrategyshouldbebasedonthesefactors: Increasingprocessingspeedandflexibility Increasingmanageabilityofbringinginnewdata Increasingqueryperformancefrompartitioneliminationasdescribedearlier Supportfordifferentaggregationdesigns

Asyouaddmorepartitions,themetadataoverheadofmanagingthecubegrowsexponentially.This affectsProcessUpdateandProcessAddoperationsondimensions,whichhavetotraversethemetadata dependenciestoupdatethecubewhendimensionschange.Asaruleofthumb,youshouldtherefore seektokeepthenumberofpartitionsinthecubeinthelowthousandswhileatthesametime balancingtherequirementsdiscussedhere. Forlargecubes,preferlargerpartitionsovercreatingtoomanypartitions.Thisalsomeansthatyoucan safelyignoretheAnalysisManagementObjects(AMO)warninginMicrosoftVisualStudiothatpartition sizesshouldnotexceed20millionrows.

2.1.2.3 Partition Strategy

Fromguidanceonpartitionsizing,wecandevelopsomecommondesignpatternsforpartition strategies.

21

2.1.2.3.1 Partition by Date Mostcubesarebuiltonatleastonecolumncontainingadate.Becausedataoftenarrivesinmonthly, weekly,daily,orevenhourlyslices,itmakessensetopartitionthecubeondate.Partitioningondate allowsyoutoreplaceafulldayincaseyouloadfaultydata.Itallowsyoutoselectivelyarchiveolddata bymovingthepartitiontocheapstorage.Andfinally,itallowsyoutoeasilygetridofdata,byremoving anentirepartition.Typically,adatepartitioningschemelookssomewhatlikethis.

Figure 1212: Partitioning by Date

Notethatinordertomovethepartitiontocheaperstorage,youwillhavetochangethedatalocation andreprocessesthepartition.Thisdesignworksverywellforsmalltomediumsizedcubes.Itis reasonablysimpletoimplementandthenumberofpartitionsiskeptlow.However,itdoessufferfroma fewdrawbacks: 1. Ifthegranularityofthepartitioningissmallenough(forexample,hourly),thenumberof partitionscanquicklybecomeunmanageable. 2. Assumingdataisaddedonlytothelatestpartition,partitionprocessingislimitedtooneTCP/IP connectionreadingfromthedatasource.Ifyouhavealotofdata,thiscanbeascalabilitylimit. Ad1)Ifyouhavealotofdatebasedpartitions,itisoftenagoodideatomergetheolderonesintolarge partitions.YoucandothiseitherbyusingtheAnalysisServicesmergefunctionalityorbydroppingthe oldpartitions,creatinganew,largerpartition,andthenreprocessingit.Reprocessingwilltypicallytake

22

longerthanmerging,butwehavefoundthatcompressionofthepartitioncanoftenincreaseifyou reprocess.Amodified,datepartitioningschememaylooklikethis.

Figure 1313: Modified Date Partitioning

Thisdesignaddressesthemetadataoverheadofhavingtoomanypartitions.Butitisstillbottlenecked bythemaximumspeedoftheProcessAddorProcessFullforthelatestpartition.Ifyourdatasourceis SQLServer,thespeedofasingledatabaseconnectioncanbehundredsofthousandsofrowsevery secondwhichworkswellformostscenarios.Butifthecuberequiresevenfasterprocessingspeeds, considermatrixpartitioning. 2.1.2.3.2 Matrix Partitioning Forlargecubes,itisoftenagoodideatoimplementamatrixpartitioningscheme:partitiononboth dateandsomeotherkey.Thedatepartitioningisusedtoselectivelydeleteormergeoldpartitionsas describedearlier.Theotherkeycanbeusedtoachieveparallelismduringpartitionprocessingandto restrictcertainuserstoasubsetofthepartitions.Forexample,consideraretailerthatoperatesinUS, Europe,andAsia.Youmightdecidetopartitionlikethis.

23


Figure 1414: Example of matrix partitioning

Iftheretailergrows,theymaychoosetosplittheregionpartitionsintosmallerpartitionstoincrease parallelismofloadfurtherandtolimittheworstcasescansthatausercanperform.Forcubesthatare expectedtogrowdramatically,itisagoodideatochooseapartitionkeythatgrowswiththebusiness andgivesyouoptionsforextendingthematrixpartitioningstrategyappropriately.Thefollowingtable containsexamplesofsuchpartitioningkeys. Industry Webretail Storeretail Datahosting 24 Examplepartitionkey Customerkey Storekey HostIDorracklocation Sourceofdataproliferation Addingcustomersandtransactions Addingnewstores Addinganewserver

Telecommunications Computerized manufacturing Investmentbanking Retailbanking Onlinegaming

SwitchID,countrycode,orarea Expandingintonewgeographical code regionsoraddingnewservices ProductionlineIDormachineID Addingproductionlinesor(for machines)sensors Stockexchangeorfinancial Addingnewfinancialinstruments, instrument products,ormarkets Creditcardnumberorcustomer Increasingcustomertransactions key Gamekeyorplayerkey Addingnewgamesorplayers

Ifyouimplementamatrixpartitioningscheme,youshouldpayspecialattentiontouserqueries.Queries touchingseveralpartitionsforeverysubcuberequest,suchasaquerythatasksforahighlevel aggregateofthepartitionbusinesskey,resultinahighthreadusageinthestorageengine.Becauseof this,werecommendthatyoupartitionthebusinesskeysothatsinglequeriestouchnomorethanthe numberofcoresavailableonthetargetserver.Forexample,ifyoupartitionbyStoreKeyandyouhave 1,000stores,queriestouchingtheaggregationofallstoreswillhavetotouch1,000partitions.Insucha design,itisagoodideatogroupthestoresintoanumberofbuckets(thatis,groupthestoresoneach partition,ratherthanhavingindividualpartitionsforeachstore).Forexample,ifyourunona16core server,youcangroupthestoreintobucketsofaround62storesforeachpartition(1,000storesdivided into16buckets). 2.1.2.3.3 Hash Partitioning Sometimesitisnotpossibletocomeupwithagooddistributionofbusinesskeysforpartitioningthe cube.Perhapsyoujustdonthaveagoodkeycandidatethatfitsthedescriptionintheprevioussection, orperhapsthedistributionofthekeyisunknownatdesigntime.Insuchcases,abruteforceapproach canbeused:Partitiononthehashvalueofakeythathasahighenoughcardinalityandwherethereis littleskew.Ifyouexpecteveryquerytotouchmanypartitions,itisimportantthatyoupayspecial attentiontotheCoordinatorQueryBalancingFactorandtheCoordinatorQueryMaxThreadsettings, whicharedescribedinPart2.

2.1.3 Relational Data Source Design

Cubesaretypicallybuiltontopofrelationaldatasourcestoserveasdatamarts.Throughthedesign surface,AnalysisServicesallowsyoutocreatepowerfulabstractionsontopoftherelationalsource. Computedcolumnsandnamedqueriesareexamplesofthis.Thisallowsfastprototypingandalso enabledyoutocorrectpoorrelationaldesignwhenyouarenotincontroloftheunderlyingdatasource. ButtheAnalysisServicesdesignsurfaceisnopanaceaawelldesignedrelationaldatasourcecanmake queriesandprocessingofacubefaster.Inthissection,weexploresomeoftheoptionsthatyoushould considerwhendesigningarelationaldatasource.Afulltreatmentofrelationaldatawarehousingisout ofscopeforthisdocument,butwewillprovidereferenceswhereappropriate.

25

2.1.3.1 Use a Star Schema for Best Performance

Itiswidelydebatedwhatthemostefficientadreportmodelingtechniqueis:starschema,snowflake schema,orevenathirdtofifthnormalformordatavaultmodels(inorderoftheincreased normalization).Allareconsideredbywarehousedesignersascandidatesforreporting. NotethattheAnalysisServicesUnifiedDimensionalModel(UDM)isadimensionalmodel,withsome additionalfeatures(referencedimensions)thatsupportsnowflakesandmanytomanydimensions.No matterwhichmodelyouchooseastheenduserreportingmodel,performanceoftherelationalmodel boilsdowntoonesimplefact:joinsareexpensive!ThisisalsopartiallytruefortheAnalysisServices engineitself.Forexample:Ifasnowflakeisimplementedasanonmaterializedreferencedimension, userswillwaitlongerforqueries,becausethejoinisdoneatruntimeinsidetheAnalysisServices engine. Thelargestimpactofsnowflakesoccursduringprocessingofthepartitiondata.Forexample:Ifyou implementafacttableasajoinoftwobigtables(forexample,separatingorderlinesandorderheaders insteadofstoringthemasprejoinedvalues),processingoffactswilltakelonger,becausetherelational enginehastocomputethejoin. ItispossibletobuildanAnalysisServicescubeontopofahighlynormalizedmodel,butbepreparedto paythepriceofjoinswhenaccessingtherelationalmodel.Inmostcases,thatpriceispaidatprocessing time.InMOLAPdatamodels,materializedreferencedimensionshelpyoustoretheresultofthejoined tablesondiskandgiveyouhighspeedqueriesevenonnormalizeddata.However,ifyouarerunning ROLAPpartitions,querieswillpaythepriceofthejoinatquerytime,andyouruserresponsetimesor yourhardwarebudgetwillsufferifyouareunabletoresistnormalization.

2.1.3.2 Consider Moving Calculations to the Relational Engine

SometimescalculationscanbemovedtotheRelationalEngineandbeprocessedassimpleaggregates withmuchbetterperformance.Thereisnosinglesolutionhere;butifyoureencounteringperformance issues,considerwhetherthecalculationcanberesolvedinthesourcedatabaseordatasourceview (DSV)andprepopulated,ratherthanevaluatedatquerytime. Forexample,insteadofwritingexpressionslikeSum(Customer.City.Members, cint(Customer.City.Currentmember.properties(Population))),considerdefiningaseparatemeasure groupontheCitytable,withasummeasureonthePopulationcolumn. Asasecondexample,youcancomputetheproductofrevenue*ProductsSoldattheleavesinthecube andaggregatewithcalculations.Butcomputingthisresultinthesourcedatabaseinsteadcanprovide superiorperformance.

2.1.3.3 Use Views

ItisgenerallyagoodideatobuildyourUDMontopofdatabaseviews.Amajoradvantageofviewsis thattheyprovideanabstractionlayerontopofthephysical,relationalmodel.Ifthecubeisbuiltontop ofviews,therelationaldatabasecan,tosomedegree,beremodeledwithoutbreakingthecube. 26

Considerarelationalsourcethathaschosentonormalizetwotablesyouneedtojointoobtainafact tableforexample,adatamodelthatsplitsasalesfactintoorderlinesandorders.Ifyouimplement thefacttableusingquerybinding,yourUDMwillcontainthefollowing.

Figure 1515: Using named queries in UDM

Inthismodel,theUDMnowhasadependencyonthestructureoftheLineItemsandOrderstables alongwiththejoinbetweenthem.IfyouinsteadimplementaSalesviewinthedatabase,youcanmodel likethis.

Figure 1616: Implementing UDM on top of views

27

ThismodelgivestherelationaldatabasethefreedomtooptimizethejoinedresultsofLineItemsand Order(forexamplebystoringitdenormalized),withoutanyimpactonthecube.Itwouldbetransparent forthecubedeveloperiftheDBAoftherelationaldatabaseimplementedthischange.

Figure 1717: Implementing UDM on top of pre-joined tables

Viewsprovideencapsulation,anditisgoodpracticetousethem.Iftherelationaldatamodelersinsist onnormalization,givethemachancetochangetheirmindsanddenormalizewithoutbreakingthecube model. Viewsalsoprovideeasyofdebugging.YoucanissueSQLqueriesdirectlyonviewstocomparethe relationaldatawiththecube.Hence,viewsaregoodwaytoimplementbusinesslogicthatcouldyou couldmimicwithquerybindingintheUDM.WhiletheUDMsyntaxissimilartotheSQLviewsyntax,you cannotissueSQLstatementsagainsttheUDM. 2.1.3.3.1 Query Binding Dimensions QuerybindingfordimensionsdoesnotexistinSQLServer2008AnalysisServices,butyoucan implementitbyusingaview(insteadoftables)foryourunderlyingdimensiondatasource.Thatway, youcanusehints,indexedviews,orotherrelationaldatabasetuningtechniquestooptimizetheSQL statementthataccessesthedimensiontablesthroughyourview.Thisalsoallowsyoutoturna snowflakedesignintherelationalsourceintoaUDMthatisapurestarschema. 2.1.3.3.2 Processing Through Views Dependingontherelationalsource,viewscanoftenprovidemeanstooptimizethebehaviorofthe relationaldatabase.Forexample,inSQLServeryoucanusetheNOLOCKhintintheviewdefinitionto removetheoverheadoflockingrowsastheviewisscanned,balancingthiswiththepossibilityofgetting dirtyreads.ViewscanalsobeusedtopreaggregatelargefacttablesusingaGROUPBYstatement;the relationaldatabasemodelercanevenchoosetomaterializeviewsthatusealotofhardwareresources. 28

2.1.4 Calculation Scripts

Thecalculationscriptinthecubeallowsyoutoexpresscomplexfunctionalityofthecube,conferringthe abilitytodirectlymanipulatethemultidimensionalspace.Inafewlinesofcode,youcanelegantlybuild highlyvaluablebusinesslogic.Butconversely,ittakesonlyafewlinesofpoorlywrittencalculationcode tocreateabigperformanceimpactonusers.Ifyouplantodesignacubewithalargecalculationscript, wehighlyrecommendthatyoulearnthebasicsofwritinggoodMDXcodethelanguageusedfor calculations.Thereferencessectioncontainsresourcesthatwillgetyouofftoagoodstart. Thequerytuningsectioninthisbookprovideshighlevelguidanceontuningindividualqueries.Buteven atdesigntime,therearesomebestpracticesyoushouldapplytothecubethatavoidcommon performancemistakes.Thissectionprovidesyouwithsomebasicrules;thesearethebareminimum youshouldapplywhenbuildingthecubescript. References: MDXhasarichcommunityofcontributorsontheweb.Herearesomelinkstogetyoustarted: Pearson,Bill:StairwaytoMDX o http://www.sqlservercentral.com/stairway/72404/ Piasevoli,Tomislav:MDXwithMicrosoftSQLServer2008R2AnalysisServicesCookbook o http://www.packtpub.com/mdxwithmicrosoftsqlserver2008r2analysis services/book Russo,Marco:MDXBlog: o http://sqlblog.com/blogs/marco_russo/archive/tags/MDX/default.aspx Pasumansky,Mosha:Blog o http://sqlblog.com/blogs/mosha/ Piasevoli,Tomislav:Blog o http://tomislav.piasevoli.com Webb,Christopher:Blog o http://cwebbbi.wordpress.com/category/mdx/ Spofford,George,SivakumarHarinath,ChristopherWebb,DylanHaiHuang,andFrancesco Civardi,:MDXSolutions:WithMicrosoftSQLServerAnalysisServices2005andHyperionEssbase, ISBN:9780471748083

2.1.4.1 Use Attributes Instead of Sets

Whenyouneedtorefertoafixedsubsetofdimensionmembersinacalculation,useanattribute insteadofaset.Attributesenableyoutotargetaggregationstothesubset.Attributesarealsoevaluated fasterthansetsbytheformulaengine.Usinganattributeforthispurposealsoallowsyoutochangethe setbyupdatingthedimensioninsteadofdeployinganewcalculationscripts. Example:Insteadofthis:

29

CREATESET[CurrentDay]ASTAIL([Date].[Calendar].members,1) CREATESET[PreviousDay]ASHEAD(TAIL(Date].[Calendar].members),2),1)

Dothis(assumingtodayis20110616): CalendarKeyAttribute 20110613 20110614 20110615 20110616 DayTypeAttribute (Flexiblerelationshiptokey) OldDates OldDates PreviousDay CurrentDay

ProcessUpdatethedimensionwhenthedaychanges.Userscannowrefertothecurrentdayby addressingtheDayTypeattributeinsteadoftheset.

2.1.4.2 Use SCOPE Instead of IIF When Addressing Cube Space

Sometimes,youwantacalculationtoonlyapplyforaspecificsubsetofcubespace.SCOPEisabetter choicethanIIFinthiscase.Hereisanexampleofwhatnottodo.
CREATEMEMBERCurrentCube.[Measures].[SixMonthRollingAverage]AS IIF([Date].[Calendar].CurrentMember.Level Is[Date].[Calendar].[Month] ,Sum([Date].[Calendar].CurrentMember.Lag(5) :[Date].[Calendar].CurrentMember ,[Measures].[InternetSalesAmount])/6 ,NULL)

Instead,usetheAnalysisServicesSCOPEfunctionforthis.
CREATEMEMBERCurrentCube.[Measures].[SixMonthRollingAverage] ASNULL,FORMAT_STRING="Currency",VISIBLE=1; SCOPE([Measures].[SixMonthRollingAverage],[Date].[Calendar].[Month].Members); THIS=Sum([Date].[Calendar].CurrentMember.Lag(5) :[Date].[Calendar].CurrentMember ,[Measures].[InternetSalesAmount])/6;

30

ENDSCOPE;

2.1.4.3 Avoid Mimicking Engine Features with Expressions

SeveralnativefeaturescanbemimickedwithMDX: Unaryoperators Calculatedcolumnsinthedatasourceview(DSV) Measureexpressions Semiadditivemeasures

YoucanreproduceeachthesefeaturesinMDXscript(infact,sometimesyoumust,becausesomeare onlysupportedintheEnterpriseSKU),butdoingsooftenhurtsperformance. Forexample,usingdistributiveunaryoperators(thatis,thosewhosememberorderdoesnotmatter, suchas+,,and~)isgenerallytwiceasfastastryingtomimictheircapabilitieswithassignments. Therearerareexceptions.Forexample,youmightbeabletoimproveperformanceofnondistributive unaryoperators(thoseinvolving*,/,ornumericvalues)withMDX.Furthermore,youmayknowsome specialcharacteristicofyourdatathatallowsyoutotakeashortcutthatimprovesperformance.Such optimizationsrequireexpertleveltuningandingeneral,youcanrelyontheAnalysisServicesengine featurestodothebestjob. Measureexpressionsalsoprovideauniquechallenge,becausetheydisabletheuseofaggregates(data hastoberolledupfromtheleaflevel).Onewaytoworkaroundthisistouseahiddenmeasurethat containspreaggregatedvaluesintherelationalsource.Youcanthentargetthehiddenmeasuretothe aggregatevalueswithaSCOPEstatementinthecalculationscript.

2.1.4.4 Comparing Objects and Values

Whendeterminingwhetherthecurrentmemberortupleisaspecificobject,useIS.Forexample,the followingqueryisnotonlynonperformant,butincorrect.Itforcesunnecessarycellevaluationand comparesvaluesinsteadofmembers.
[Customer].[CustomerGeography].[Country].&[Australia]=[Customer].[Customer Geography].currentmember

Furthermore,dontperformextrastepswhendeducingwhetherCurrentMemberisaparticular memberbyinvolvingIntersectandCounting.

31

 intersect({[Customer].[CustomerGeography].[Country].&[Australia]}, [Customer].[CustomerGeography].currentmember).count>0

UseISinstead.
[Customer].[CustomerGeography].[Country].&[Australia]is[Customer].[Customer Geography].currentmember

2.1.4.5 Evaluating Set Membership

DeterminingwhetheramemberortupleisinasetisbestaccomplishedwithIntersect.TheRank functiondoestheadditionaloperationofdeterminingwhereinthesetthatobjectlies.Ifyoudontneed it,dontuseit.Forexample,thefollowingstatementmaydomoreworkthanyouneedittodo.
rank([Customer].[CustomerGeography].[Country].&[Australia], <setexpression>)>0

ThisstatementusesIntersecttodeterminewhetherthespecifiedinformationisintheset.
intersect({[Customer].[CustomerGeography].[Country].&[Australia]},<set>).count>0

2.2 Testing Analysis Services Cubes

Asyouprepareforuseracceptanceandpreproductiontestingofacube,youshouldfirstconsiderwhat acubeisandwhatthatmeansforuserqueries.Dependingonyourbackgroundandroleinthe developmentanddeploymentcycle,therearedifferentwaystolookatthis. Asadatabaseadministrator,youcanthinkofacubeasadatabasethatcanacceptanyqueryfrom users,andwheretheresponsetimefromanysuchqueryisexpectedtobereasonableatermthatis oftenvaguelydefined.Inmanycases,youcanoptimizeresponsetimeforspecificqueriesusing aggregates(whichforrelationalDBAsissimilartoindextuning),andtestingshouldgiveyouanearly ideaofgoodaggregationcandidates.Butevenwithaggregates,youmustalsoconsidertheworstcase: Youshouldexpecttoseeleaflevelscanqueries.Suchqueries,whichcanbeeasilyexpressedbytools likeExcel,mayenduptouchingeverypieceofdatathecube.Dependingonyourdesign,thiscanbea 32

significantamountofdata.Youshouldconsiderwhatyouwanttodowithsuchqueries.Thereare multipleoptions:Forexample,youmaychoosetoscaleyourhardwaretohandletheminadecent responsetime,oryoumaysimplychoosetocancelthem.Ineithercase,asyoupreparefortesting, makesuresuchqueriesarepartthetestsuiteandthatyouobservewhathappenstotheAnalysis Servicesinstancewhentheyrun.Youshouldalsounderstandwhatareasonableresponsetimeisfor yourenduserandmakethatpartofthetestsuite. AsaBIdeveloper,youcanlookatthecubeasyourdescriptionofthemultidimensionalspaceinwhich queriescanbeexpressed.Apartofthisspacewillbeinstantiatedwithdatastructuresondisk supportingit:dimensions,measuregroups,andtheirpartitions.However,someofthis multidimensionalspacewillbeservedbycalculationsoradhocmemorystructures,forexample:MDX calculations,manytomanydimensions,andcustomrollups.Wheneverqueriesarenotserveddirectly byinstantiateddata,thereisapotential,querytimecalculationpricetobepaid,thismayshowupas badresponsetime.Asthecubedeveloper,youshouldmakesurethatthetestingcoversthesecases. Hence,astheBIdeveloper,youshouldmakesurethetestqueriesalsostressnoninstantiateddata.This isavaluableexercise,becauseyoucanuseittomeasuretheimpactonusersofcomplexcalculations andthenadjustthedatamodelaccordingly. Yourapproachtotestingwilldependonwhichsituationyoufindyourselfin.Ifyouaredevelopinganew system,youcanworkdirectlytowardsthetestgoalsdrivenbybusinessrequirements.However,ifyou aretryingtoimproveanexistingsystem,itisanadvantagetoacquireatestbaselinefirst.

2.2.1 Testing Goals

Beforeyoudesignatestharness,youshoulddecidewhatyourtestinggoalswillbe.Testingnotonly allowsyoutofindfunctionalbugsinthesystem,italsohelpsquantifythescalabilityandpotential bottlenecksthatmaybehardtodiagnoseandfixinabusyproductionenvironment.Ifyoudonotknow whatyourscalebarrierisorwheretheysystemmightbreak,itbecomeshardtoactwithconfidence whenyoutunethefinalproductionsystem. ConsiderwhatcharacteristicsarereasonabletoexpectfromtheBIsystemsanddocumentthese expectationsaspartofyourtestplan.Thedefinitionofreasonabledependstoalargeextentonyour familiaritywithsimilarsystems,theskillsofyourcubedesigners,andthehardwareyourunon.Itis oftenagoodideatogetasecondopiniononwhattestvaluesarereachableforexamplefroma neutralthirdpartythathasexperiencewithsimilarsystems.Thishelpsyousetexpectationsproperly withbothdevelopersandbusinessusers. BIsystemsvaryinthecharacteristicsorganizationsrequireofthemnoteveryoneneedsscalabilityto thousandsofusers,tensofterabytes,nearzerodowntime,andguaranteedsubsecondresponsetime. Whileallthesegoalscanbeachievedformostcases,itisnotalwayscheaptoacquiretheskillsrequired todesignasystemtosupportthem.Considerwhatyoursystemneedstodoforyourorganization,and avoidoverdesigningintheareaswhereyoudontneedthehighestrequirements.Forexample:youmay decidethatyouneedveryfastresponsetimes,butthatyoualsowantaverylowcostserverthatcan runinasharedstorageenvironment.Forsuchascenario,youmaywanttoreducethedatainthecube 33

toasizethatwillfitinmemory,eliminatingtheneedforthemajorityofI/Ooperationsandproviding fastscantimesevenforpoorlyfilteredqueries. Hereisatableofpotentialtestgoalsyoushouldconsider.Iftheyarerelevantforyourorganizations requirements,youshouldtailorthemtoreflectthoserequirements. TestGoal Scalability Examplegoal Mustsupport10,000 concurrentlyconnectedusers,of which1,000runqueries simultaneously. Simplequeriesreturninga Howfastshouldqueriesreturntotheclient? singleproductgroupforagiven Thismayrequireyoutoclassifyqueriesinto yearshouldreturninlessthan1 differentcomplexities. secondevenatfulluser Notallqueriescanbeansweredquicklyandit concurrency. willoftenbewisetoconsultanexpertcube designertoliaisewithuserstounderstandwhat Queriesthattouchnomore querypatternscanbeexpectedandwhatthe than20%ofthefactrowsshould runinlessthan30seconds. complexityofansweringthesequerieswillbe. Mostotherqueriestouchinga smallpartofthecubeshould Anotherwaytolookatthistestgoalisto returninaround10seconds. measurethethroughputinqueriesanswered Withourworkload,weexpect persecondinamixedworkload. throughputtobearound50 queriesreturnedpersecond. Userqueriesrequestingthe endofmonthcurrencyrate conversionshouldreturninno morethan20seconds.Queries thatdonotrequirecurrency conversionshouldreturninless than5seconds. Whatisthegranularityofeachdimension Thelargestcustomerdimension attribute?Howmuchdatawilleachmeasure willcontain30millionrowsand groupcontain? have10attributesandtwouser hierarchies.Thelargestnonkey Notethatthecubedesignerswilloftenhave attributewillhave1million beenconsideringthisandmayalreadyknowthe members. answer. Thelargestmeasuregroupis sales,with1billionrows.The secondlargestispurchases,with 100millionrows.Allother measuregroupsaretrivialin size. Description Howmanyconcurrentusersshouldbe supportedbythesystem?

Performance/ throughput

DataSizes

34

TargetServer Platforms

Whichservermodeldoyouwanttorunon?Itis oftenagoodideatotestonboththatserver andanevenbiggerserverclass.Thisenables youtoquantifythebenefitsofupgrading.

TargetI/O system

WhichI/Osystemdoyouwanttouse?What characteristicswillthatsystemhave?

Targetnetwork infrastructure

Whichnetworkconnectivitywillbeavailable betweenusersandAnalysisServices,and betweenAnalysisServicesandthedatasources? Notethatyoumayhavetosimulatethese networkconditionsinalab.

Processing Speeds

Howfastshouldrowsbebroughtintothecube andhowoften?

Mustrunon2socket6core Nehalemmachinewith32GBof RAM. Mustbeabletoscaleto4 socketNehalem8coremachine with256GBofRAM. MustrunoncorporateSANand usenomorethan1,000random IOPSat32,000blocksizesat6ms latency. WillrunondedicatedNAND devicesthatsupport80,000IOPS at100slatency. Intheworstcasescenario, userswillconnectovera100ms latencyWANlinkwitha maximumbandwidthof 10Mbit/sec. Therewillbea10Gbit dedicatednetworkavailable betweenthedatasourceandthe cube. Dimensionsshouldbefully processedeverynightwithin30 minutes. Twotimesduringtheday, 100,000,000rowsshouldbe addedtothesalesmeasure group.Thisshouldtakeno longerthan15minutes.

2.2.2 Test Scenarios

Basedontheconsiderationsfromtheprevioussectionyoushouldbeabletocreateauserworkload thatrepresentstypicaluserbehaviorandthatenablesyoutomeasurewhetheryouaremeetingyour testinggoals. Typicaluserbehaviorandwellwrittenqueriesareunfortunatelynottheonlyqueriesyouwillreceivein mostsystems.Aspartofthetestphase,youshouldalsotrytoflushoutpotentialproductionissues beforetheyarise.Werecommendthatyoumakesureyourtestworkloadcontainsthefollowingtypes ofqueriesandteststhemthoroughly:

35

Queriesthattouchseveraldimensionsatthesametime EnoughqueriestotesteveryMDXexpressioninthecalculationscript Queriesthatexercisemanytomanydimensions Queriesthatexercisecustomrollups Queriesthatexerciseparent/childdimensions Queriesthatexercisedistinctcountmeasuregroups Queriesthatcrossjoinattributesfromdimensionsthatcontainmorethan100,000members Queriesthattoucheverysinglepartitioninthedatabase Queriesthattouchalargesubsetofpartitionsinthedatabase(forexample,currentyear) Queriesthatreturnalotofdatatotheclient(forexample,morethan100,000rows) Queriesthatusecubesecurityversusqueriesthatdonotuseit Queriesexecutingconcurrentlywithprocessingoperationsifthisispartofyourdesign

Youshouldtestonthefulldatasetfortheproductioncube.Ifyoudont,theresultswillnotbe representativeofreallifeoperations.Thisisespeciallytrueforcubesthatarelargerthanthememory onthemachinetheywilleventuallyrunon. Ofcourse,youshouldstillmakesurethatyouhaveplentyofqueriesinthetestscenariosthatrepresent typicaluserbehaviorsrunningonaworkloadthatonlyshowcasestheslowestperformingpartsofthe cubewillnotrepresentarealproductionenvironment(unlessofcourse,theentirecubeispoorly designed). Asyourunthetests,youwilldiscoverthatcertainqueriesaremoredisruptivethanothers.Onegoalof testingistodiscoverwhatsuchquerieslooklike,sothatyoucaneitherscalethesystemtodealwith themorprovideguidanceforuserssothattheycanavoidexercisingthecubeinthiswayifpossible. Partofyourtestscenariosshouldalsoaimtoobservethecubesbehaviorasuserconcurrencygrows. YoushouldworkwithBIdevelopersandbusinessuserstounderstandwhattheworstcasescenariofor userconcurrencyis.Testingatthatconcurrencywillshakeoutpoorlyscalabledesignsandhelpyou configurethecubeandhardwareforbestperformanceandstability.

2.2.3 Load Generation

Itishardtocreatealoadthatactuallylooksliketheexpectedproductionloaditrequiressignificant experienceandcommunicationwithenduserstocomeupwithafullyrepresentativesetofqueries.But afteryouhaveasetofqueriesthatmatchuserbehavior,youcanfeedthemintoatestharness.Analysis Servicesdoesnotshipwithatestharnessoutofthebox,butthereareseveralsolutionsavailablethat helpyougetstarted: ascmdYoucanusethiscommandlinetooltorunasetofqueriesagainstAnalysisServices.Itships withtheAnalysisServicessamplesandismaintainedonCodePlex. VisualStudioYoucanconfigureMicrosoftVisualStudiotogenerateloadagainstAnalysisServices,and youcanalsouseVisualStudiotovisuallyanalyzethatload.

36

ThirdpartytoolsYoucanusetoolssuchasHPLoadRunnertogeneratehighconcurrencyload.Note thatAnalysisServicesalsosupportsanHTTPbasedinterface,whichmeansitmaybepossibletouse webstresstoolstogenerateload. Rollyourown:Wehaveseencustomerswritetheirowntestharnessesusing.NETandtheADOMD.NET interfacetoAnalysisServices.Usingthe.NETthreadinglibraries,itispossibletogeneratealotofuser loadfromasingleloadclient. Nomatterwhichloadtoolyouuse,youshouldmakesureyoucollecttheruntimeofallqueriesandthe PerformanceMonitorcountersforallruns.Thisdataenablesyoutomeasuretheeffectofanychanges youmakeduringyourtestruns.Whenyougenerateuserworkloadtherearealsosomeotherfactorsto consider. Firstofall,youshouldtestbothasequentialrunandaparallelrunofqueries.Thesequentialrungives youthebestpossibleruntimeofthequerywhilenootherusersareonthesystem.Theparallelrun enablesyoutoshakeoutissueswiththecubethataretheresultofmanyusersrunningconcurrently. Second,youshouldmakesurethetestscenarioscontainasufficientnumberofqueriessothatyouwill beabletorunthetestscenarioforsometime.Tostresstheserverproperly,andtoavoidqueryingthe samehotspotvaluesoverandoveragain,queriesshouldtouchavarietyofdatainthecube.Ifallyour queriestouchasmallsetofdimensionvalues,itwillhardlyrepresentarealproductionrun.Onewayto spreadqueriesoverawidersetofcellsinthecubeistousequerytemplates.Eachtemplatecanbeused togenerateasetofqueriesthatareallvariantsofthesamegeneraluserbehavior. Third,yourtestharnessshouldbeabletocreatereproducibletests.Ifyouareusingcodethatgenerates manyqueriesfromasmallsetoftemplates,makesurethatitgeneratesthesamequeriesonevery testrun.Ifnot,youintroduceanelementofrandomnessinthetestthatmakesithardtocompare differentruns. References: Ascmd.exeonMSDNhttp://msdn.microsoft.com/en us/library/ms365187%28v=sql.100%29.aspx AnalysisServicesCommunitysampleshttp://sqlsrvanalysissrvcs.codeplex.com/ o Describeshowtouseascmdforloadgeneration o ContainsVisualStudiosamplecodethatachievesasimilareffect HPLoadRunner https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=111 12617^8_4000_100

2.2.4 Clearing Caches

Tomaketestrunsreproducible,itisimportantthateachrunstartwiththeserverinthesamestateas thepreviousrun.Todothis,youmustclearoutanycachescreatedbyearlierruns.

37

TherearethreecachesinAnalysisServicesthatyoushouldbeawareof: Theformulaenginecache Thestorageenginecache Thefilesystemcache

Clearingformulaengineandstorageenginecaches:ThefirsttwocachescanbeclearedwiththeXMLA ClearCachecommand.Thiscommandcanbeexecutedusingtheascmdcommandlineutility:
<ClearCache xmlns="http://schemas.microsoft.com/analysisservices/2003/engine"> <Object> <DatabaseID><database name></DatabaseID> </Object> </ClearCache>

Clearingfilesystemcaches:Thefilesystemcacheisabithardertogetridofbecauseitresidesinside Windowsitself. IfyouhavecreatedaseparateWindowsvolumeforthecubedatabase,youcandismountthevolume itselfusingthefollowingcommand: fsutil.exevolumedismount<DriveLetter|MountPoint> Thisclearsthefilesystemcacheforthisdriveletterormountpoint.Ifthecubedatabaseresidesonlyon thislocation,runningthiscommandresultsinacleanfilesystemcache. Alternatively,youcanusetheutilityRAMMapfromsysinternals.Thisutilitynotonlyallowsyoutoread thefilesystemcachecontent,italsoallowsyoutopurgeit.Ontheemptymenu,clickEmptySystem WorkingSet,andthenclickEmptyStandbyList.Thisclearsthefilesystemcachefortheentiresystem. NotethatwhenRAMMapstartsup,ittemporarilyfreezesthesystemwhileitreadsthememorycontent thiscantakesometimeonalargemachine.Hence,RAMMapshouldbeusedwithcare. ThereiscurrentlyaCodePlexprojectcalledASStoredProceduresfoundat: http://asstoredprocedures.codeplex.com/wikipage?title=FileSystemCache.Thisprojectcontainscode forautilitythatenablesyoutoclearthefilesystemcacheusingastoredprocedurethatyoucanrun directlyonAnalysisServices. NotethatneitherFSUTILnorRAMMapshouldbeusedinproductioncubesbothcausedisruptionto usersconnectedtothecube.AlsonotethatneitherRAMMaporASStoredProceduresissupportedby Microsoft.

2.2.5 Actions During Testing for Operations Management

Whileyourtestteamiscreatingandrunningthetestharness,youroperationsteamcanalsotakesteps topreparefordeployment.

38

Testyourdatacollectionsetup:Testrunsgiveyouauniquechancetotryoutyourdatacollection proceduresbeforeyougointoproduction.Thedatacollectionyouperformcanalsobeusedtodrive earlyfeedbacktothedevelopmentteam. Understandserverutilization:Whileyoutest,youcangetanearlyinsightintoserverutilization.You willbeabletomeasurethememoryusageofthecubeandthewaythenumberofusersmapstoI/O loadandCPUutilization.Ifthecubeislargerthanmemory,youcanalsomeasuretheeffectof concurrencyandleaflevelscanontheI/Osubsystem.Remembertomeasuretheworstcaseexamples describedearliertounderstandwhattheimpactonthesystemis. Earlythreadtuning:Duringtesting,youcandiscoverthreadingbottlenecks,asdescribedinPart1.This enablesyoutogointoproductionwithpretunedsettingsthatimproveuserexperience,scalability,and hardwareutilizationofthesolution.

2.2.6 Actions After Testing

Whentestingiscomplete,youhavereportsthatdescribetheruntimeofeachquery,andyoualsohave agreaterunderstandingoftheserverutilization.Thisisagoodtimetoreviewthedesignofthecube withtheBIdevelopersusingthenumbersyoucollectedduringtesting.Often,someeasywinscanbe harvestedatthispoint. Whenyouputacubeintoproduction,itisimportanttounderstandthelongtermeffectsofusers buildingspreadsheetsandreportsreferencingit.Considerthedependenciesthataregeneratedasthe cubeissuccessfullydeployedinadhocdatastructuresacrosstheorganization.Thedatamodelcreated andexposedbythecubewillbelinkedintospreadsheetsandreports,anditbecomeshardtomakedata modelchangeswithoutdisturbingusers.Fromanoperationalperspective,preproductiontestingis typicallyyourlastchancetorequestcheapdatamodelchangesfromyourBIdevelopersbeforebusiness usersinevitablylockthemselvesintothedatastructuresthatunlocktheirdata.Noticethatthisis differentfromtypical,staticreportinganddevelopmentcycles.Withstaticreports,theBIdevelopers areincontrolofthedependencies;ifyougiveusersExcelorotheradhocaccesstocubes,thatcontrolis lost.Explorativedatapowercomesataprice.

2.3 Tuning Query Performance

Toimprovequeryperformance,youshouldunderstandthecurrentsituation,diagnosethebottleneck, andthenapplyoneofseveraltechniquesincludingoptimizingdimensiondesign,designingandbuilding aggregations,partitioning,andapplyingbestpractices.Theseshouldbethefirststopsforoptimization, beforediggingintoqueriesingeneral. Muchtimecanbeexpendedpursuingdeadendsitisimportanttofirstunderstandthenatureofthe problembeforeapplyingspecifictechniques.Togainthisunderstanding,itisoftenusefultohavea mentalmodelofhowthequeryengineworks.Wewillthereforestartwithabriefintroductiontothe AnalysisServicesqueryprocessor.

39

2.3.1 Query Processor Architecture

Tomakethequeryingexperienceasfastaspossibleforendusers,theAnalysisServicesquerying architectureprovidesseveralcomponentsthatworktogethertoefficientlyretrieveandevaluatedata. Thefollowingfigureidentifiesthethreemajoroperationsthatoccurduringqueryingsession management,MDXqueryexecution,anddataretrievalaswellastheservercomponentsthat participateineachoperation.

ClientApp(MDX)

SessionManagement

XMLA Listener Session Manager SecurityManager

QueryProcessing

QueryProcessor

QueryProcCache

DataRetrieval

Storage Engine

SECache

DimensionData AttributeStore HierarchyStore

MeasureGroupData FactData Aggregations

Figure 1818: Analysis Services query processor architecture

2.3.1.1 Session Management

ClientapplicationscommunicatewithAnalysisServicesusingXMLforAnalysis(XMLA)overTCP/IPor HTTP.AnalysisServicesprovidesanXMLAlistenercomponentthathandlesallXMLAcommunications betweenAnalysisServicesanditsclients.TheAnalysisServicesSessionManagercontrolshowclients connecttoanAnalysisServicesinstance.UsersauthenticatedbytheWindowsoperatingsystemand whohaveaccesstoatleastonedatabasecanconnecttoAnalysisServices.Afterauserconnectsto 40

AnalysisServices,theSecurityManagerdeterminesuserpermissionsbasedonthecombinationof AnalysisServicesrolesthatapplytotheuser.Dependingontheclientapplicationarchitectureandthe securityprivilegesoftheconnection,theclientcreatesasessionwhentheapplicationstarts,andthenit reusesthesessionforalloftheusersrequests.Thesessionprovidesthecontextunderwhichclient queriesareexecutedbythequeryprocessor.Asessionexistsuntilitisclosedbytheclientapplicationor theserver.

2.3.1.2 Query Processing

ThequeryprocessorexecutesMDXqueriesandgeneratesacellsetorrowsetinreturn.Thissection providesanoverviewofhowthequeryprocessorexecutesqueries.Formoreinformationabout optimizingMDX,seeOptimizingMDX. Toretrievethedatarequestedbyaquery,thequeryprocessorbuildsanexecutionplantogeneratethe requestedresultsfromthecubedataandcalculations.Therearetwomajordifferenttypesofquery executionplans:cellbycell(nave)evaluationorblockmode(subspace)computation.Whichoneis chosenbytheenginecanhaveasignificantimpactonperformance.Formoreinformation,seeSubspace Computation. Tocommunicatewiththestorageengine,thequeryprocessorusestheexecutionplantotranslatethe datarequestintooneormoresubcuberequeststhatthestorageenginecanunderstand.Asubcubeisa logicalunitofquerying,caching,anddataretrievalitisasubsetofcubedatadefinedbythecrossjoin ofoneormoremembersfromasinglelevelofeachattributehierarchy.AnMDXquerycanberesolved intomultiplesubcuberequests,dependingtheattributegranularitiesinvolvedandcalculation complexity;forexample,aqueryinvolvingeverymemberoftheCountryattributehierarchy(assuming itsnotaparentchildhierarchy)wouldbesplitintotwosubcuberequests:onefortheAllmemberand anotherforthecountries. Asthequeryprocessorevaluatescells,itusesthequeryprocessorcachetostorecalculationresults.The primarybenefitsofthecachearetooptimizetheevaluationofcalculationsandtosupportthereuseof calculationresultsacrossusers(withthesamesecurityroles).Tooptimizecachereuse,thequery processormanagesthreecachelayersthatdeterminethelevelofcachereusability:global,session,and query. 2.3.1.2.1 Query Processor Cache DuringtheexecutionofanMDXquery,thequeryprocessorstorescalculationresultsinthequery processorcache.Theprimarybenefitsofthecachearetooptimizetheevaluationofcalculationsandto supportreuseofcalculationresultsacrossusers.Tounderstandhowthequeryprocessorusescaching duringqueryexecution,considerthefollowingexample:YouhaveacalculatedmembercalledProfit Margin.WhenanMDXqueryrequestsProfitMarginbySalesTerritory,thequeryprocessorcachesthe nonnullProfitMarginvaluesforeachSalesTerritory.Tomanagethereuseofthecachedresultsacross users,thequeryprocessordistinguishesdifferentcontextsinthecache:

41

QueryContextcontainstheresultofcalculationscreatedbyusingtheWITHkeywordwithina query.Thequerycontextiscreatedondemandandterminateswhenthequeryisover. Therefore,thecacheofthequerycontextisnotsharedacrossqueriesinasession. SessionContextcontainstheresultofcalculationscreatedbyusingtheCREATEstatement withinagivensession.Thecacheofthesessioncontextisreusedfromrequesttorequestinthe samesession,butitisnotsharedacrosssessions. GlobalContextcontainstheresultofcalculationsthataresharedamongusers.Thecacheof theglobalcontextcanbesharedacrosssessionsifthesessionssharethesamesecurityroles.

Thecontextsaretieredintermsoftheirlevelofreuse.Atthetop,thequerycontextiscanbereused onlywithinthequery.Atthebottom,theglobalcontexthasthegreatestpotentialforreuseacross multiplesessionsandusersbecausethesessioncontextwillderivefromtheglobalcontextandthe querycontextwillderiveitselffromthesessioncontext.

Figure 1919: Cache context layers

Duringexecution,everyMDXquerymustreferenceallthreecontextstoidentifyallofthepotential calculationsandsecurityconditionsthatcanimpacttheevaluationofthequery.Forexample,toresolve aquerythatcontainsaquerycalculatedmember,thequeryprocessorcreatesaquerycontextto resolvethequerycalculatedmember,createsasessioncontexttoevaluatesessioncalculations,and createsaglobalcontexttoevaluatetheMDXscriptandretrievethesecuritypermissionsoftheuser whosubmittedthequery.Notethatthesecontextsarecreatedonlyiftheyarentalreadybuilt.After theyarebuilt,theyarereusedwherepossible. Eventhoughaqueryreferencesallthreecontexts,itwilltypicallyusethecacheofasinglecontext.This meansthatonaperquerybasis,thequeryprocessormustselectwhichcachetouse.Thequery processoralwaysattemptstousethebroadlyapplicablecachedependingonwhetherornotitdetects thepresenceofcalculationsatanarrowercontext. Ifthequeryprocessorencounterscalculationscreatedatquerytime,italwaysusesthequerycontext, evenifaqueryalsoreferencescalculationsfromtheglobalcontext(thereisanexceptiontothis 42

querieswithquerycalculatedmembersoftheformAggregate(<set>)dosharethesessioncache).If therearenoquerycalculations,buttherearesessioncalculations,thequeryprocessorusesthesession cache.Thequeryprocessorselectsthecachebasedonthepresenceofanycalculationinthescope.This behaviorisespeciallyrelevanttouserswithMDXgeneratingfrontendtools.Ifthefrontendtool createsanysessioncalculationsorquerycalculations,theglobalcacheisnotused,evenifyoudonot specificallyusethesessionorquerycalculations. Thereareothercalculationscenariosthatimpacthowthequeryprocessorcachescalculations.When youcallastoredprocedurefromanMDXcalculation,theenginealwaysusesthequerycache.Thisis becausestoredproceduresarenondeterministic(meaningthatthereisnoguaranteewhatthestored procedurewillreturn).Asaresult,afteranondeterministiccalculationisencounteredduringthequery, nothingiscachedgloballyorinthesessioncache.Instead,theremainingcalculationsarestoredinthe querycache.Inaddition,thefollowingscenariosdeterminehowthequeryprocessorcachescalculation results: TheuseofMDXfunctionsthatarelocaledependent(suchasCaptionor.Properties)prevents theuseoftheglobalcache,becausedifferentsessionsmaybeconnectedwithdifferentlocales andcachedresultsforonelocalemaynotbecorrectforanotherlocale. Theuseofcellsecurity;functionssuchasUserName,StrToSet,StrToMember,andStrToTuple; orLookupCubefunctionsintheMDXscriptorinthedimensionorcellsecuritydefinitiondisable theglobalcache.Thatis,justoneexpressionthatusesanyofthesefunctionsorfeatures disablesglobalcachingfortheentirecube. IfvisualtotalsareenabledforthesessionbysettingthedefaultMDXVisualModepropertyin theAnalysisServicesconnectionstringto1,thequeryprocessorusesthequerycacheforall queriesissuedinthatsession. IfyouenablevisualtotalsforaquerybyusingtheMDXVisualTotalsfunction,thequery processorusesthequerycache. Queriesthatusethesubselectsyntax(SELECTFROMSELECT)orarebasedonasessionsubcube (CREATESUBCUBE)resultinthequeryor,respectively,sessioncachetobeused. Arbitraryshapescanonlyusethequerycacheiftheyareusedinasubselect,intheWHERE clause,orinacalculatedmember.Anarbitraryshapeisanysetthatcannotbeexpressedasa crossjoinofmembersfromthesamelevelofanattributehierarchy.Forexample,{(Food,USA), (Drink,Canada)}isanarbitraryset,asis{customer.geography.USA,customer.geography.[British Columbia]}.Notethatanarbitraryshapeonthequeryaxisdoesnotlimittheuseofanycache.

Basedonthisbehavior,whenyourqueryingworkloadcanbenefitfromreusingdataacrossusers,itisa goodpracticetodefinecalculationsintheglobalscope.Anexampleofthisscenarioisastructured reportingworkloadwhereyouhavefewsecurityroles.Bycontrast,ifyouhaveaworkloadthatrequires individualdatasetsforeachuser,suchasinanHRcubewhereyouhavemanysecurityrolesoryouare 43

usingdynamicsecurity,theopportunitytoreusecalculationresultsacrossusersislessenedor eliminated.Asaresult,theperformancebenefitsassociatedwithreusingthequeryprocessorcacheare notashigh.

2.3.1.3 Data Retrieval

Whenyouqueryacube,thequeryprocessorbreaksthequeryintosubcuberequestsforthestorage engine.Foreachsubcuberequest,thestorageenginefirstattemptstoretrievedatafromthestorage enginecache.Ifnodataisavailableinthecache,itattemptstoretrievedatafromanaggregation.Ifno aggregationispresent,itmustretrievethedatafromthefactdatafromameasuregroupspartition data. RetrievingdatafromapartitionrequiresI/Oactivity.ThisI/Ocaneitherbeservedfromthefilesystem cacheorfromdisk.AdditionaldetailsoftheI/OsubsystemofAnalysisServicescanbefoundinPart2.

Figure 2020: High-level overview of the data retrieval process

2.3.1.3.1 Storage Engine Cache Thestorageenginecacheisalsoknownasthedatacacheregistrybecauseitiscomposedofthe dimensionandmeasuregroupcachesthatarethesamestructurally.Whenarequestismadefromthe 44

AnalysisServicesformulaenginetothestorageengine,itsendsarequestintheformofasubcube describingthestructureofthedatarequestandadatacachestructurethatwillcontaintheresultsof therequest.Usingthedatacacheregistryindexes,itattemptstofindacorrespondingsubcube: Ifthereisamatchingsubcube,thecorrespondingdatacacheisreturned. Ifasubcubesupersetisfound,anewdatacacheisgeneratedandtheresultsarefilteredtofit thesubcuberequest. Iflowergraindataexists,thedatacacheregistrycanaggregatethisdataandmakeitavailable aswellandthenewsubcubeanddatacachearealsoregisteredinthecacheregistry. Ifdatadoesnotexist,therequestgoestothestorageengineandtheresultsarecachedinthe cacheregistryforfuturequeries.

AnalysisServicesallocatesmemoryviamemoryholdersthatcontainstatisticalinformationaboutthe amountofmemorybeingused.Memoryholdersareintheformofnonshrinkableandshrinkable memory;eachcombinationofasubcubeanddatacacheformsasingleshrinkablememoryholder. WhenAnalysisServicesisunderheavymemorypressure,cleanerthreadsremoveshrinkablememory. Therefore,ensureyoursystemhasenoughmemory;ifitdoesnot,yourdatacacheregistrywillbe clearedout(resultinginslowerqueryperformance)whenitisplacedundermemorypressure. 2.3.1.3.2 Aggressive Data Scanning Sometimes,intheevaluationofanexpression,moredataisrequestedthanrequiredtodeterminethe result. Ifyoususpectmoredataisbeingretrievedthanisrequired,youcanuseSQLServerProfilertodiagnose howaqueryintosubcubequeryeventsandpartitionscans.Forsubcubescans,checktheverbose subcubeeventandwhethermoremembersthanrequiredareretrievedfromthestorageengine.For smallcubes,thislikelyisntaproblem.Forlargercubeswithmultiplepartitions,itcangreatlyreduce queryperformance.Thefollowingfiguredemonstrateshowasinglequerysubcubeeventresultsin partitionscans.

Figure2121:Aggressivepartitionscanning Therearetwopotentialsolutionstothis.Ifacalculationexpressioncontainsanarbitraryshape(thisis definedinthesectiononthequeryprocessorcache),thequeryprocessormaynotbeabletodetermine thatthedataislimitedtoasinglepartitionandrequestdatafromallpartitions.Trytoeliminatethe arbitraryshape.

45

Othertimes,thequeryprocessorissimplyoverlyaggressiveinaskingfordata.Forsmallcubes,this doesntmatter,butforverylargecubes,itdoes.Ifyouobservethisbehavior,potentialsolutionsinclude thefollowing: ContactMicrosoftCustomerServiceandSupportforfurtheradvice. DisablePrefetch=1(thisisdoneintheconnectionstring):SometimesAnalysisServicesrequests additionaldatafromthesourcetoprepopulatethecache;itmayhelptoturnitoffsothat AnalysisServicesdoesnotrequesttoomuchdata.

2.3.2 Query Processor Internals

ThereareseveralchangestoqueryprocessorinternalsinSQLServer2008AnalysisServicesthatare applicabletoday(comparedtoSQLServer2005AnalysisServices).Inthissection,thesechangesare discussedbeforespecificoptimizationtechniquesareintroduced.

2.3.2.1 Subspace Computation

Thekeyideabehindsubspacecomputationisbestintroducedbycontrastingitwithacellbycell evaluationofacalculation.(Thisisalsoknownasanavecalculation.)Consideratrivialcalculation RollingSumthatsumsthesalesforthepreviousyearandthecurrentyear,andaquerythatrequeststhe RollingSumfor2005forallProducts. RollingSum=(Year.PrevMember,Sales)+Sales SELECT2005oncolumns,Product.MembersonrowsWHERERollingSum

Acellbycellevaluationofthiscalculationproceedsasrepresentedinthefollowingfigure.

46


Figure 2222: Cell-by-cell evaluation

The10cellsfor[2005,AllProducts]areeachevaluatedinturn.Foreach,thepreviousyearislocated, andthenthesalesvalueisobtainedandthenaddedtothesalesforthecurrentyear.Therearetwo significantperformanceissueswiththisapproach. Firstly,ifthedataissparse(thatis,thinlypopulated),cellsarecalculatedeventhoughtheyareboundto returnanullvalue.Inthepreviousexample,calculatingthecellsforanythingbutProduct3andProduct 6isawasteofeffort.Theimpactofthiscanbeextremeinasparselypopulatedcube,thedifference canbeseveralordersofmagnitudeinthenumbersofcellsevaluated. Secondly,evenifthedataistotallydense,meaningthateverycellhasavalueandthereisno wastedeffortvisitingemptycells,thereismuchrepeatedeffort.Thesamework(forexample,getting thepreviousYearmember,settingupthenewcontextforthepreviousYearcell,checkingforrecursion) isredoneforeachProduct.Itwouldbemuchmoreefficienttomovethisworkoutoftheinnerloopof evaluatingeachcell. Nowconsiderthesameexampleperformedusingsubspacecomputation.Insubspacecomputation,the engineworksitswaydownanexecutiontreedeterminingwhatspacesneedtobefilled.Giventhe query,thefollowingspaceneedstobecomputed,where*meanseverymemberoftheattribute hierarchy. [Product.*,2005,RollingSum]

Giventhecalculation,thismeansthatthefollowingspaceneedstobecomputedfirst. 47 [Product.*,2004,Sales]

Next,thefollowingspacemustbecomputed. [Product.*,2005,Sales]

Finally,the+operatorneedstobeaddedtothosetwospaces. IfSaleswereitselfcoveredbycalculations,thespacesnecessarytocalculateSaleswouldbedetermined andthetreewouldbeexpanded.InthiscaseSalesisabasemeasure,sothestorageenginedataisused tofillthetwospacesattheleaves,andthen,workingupthetree,theoperatorisappliedtofillthe spaceattheroot.Hencetheonerow(Product3,2004,3)andthetworows{(Product3,2005,20), (Product6,2005,5)}areretrieved,andthe+operatorappliedtothemtoyieldsthefollowingresult.

Figure 2323: Execution plan

The+operatoroperatesonspaces,notsimplyscalarvalues.Itisresponsibleforcombiningthetwogiven spacestoproduceaspacethatcontainseachproductthatappearsineitherspacewiththesummed value.Thisisthequeryexecutionplan.Notethatitoperatesonlyondatathatcouldcontributetothe result.Thereisnonotionofthetheoreticalspaceoverwhichthecalculationmustbeperformed. Aqueryexecutionplanisnotoneortheotherbutcancontainbothsubspaceandcellbycellnodes. Somefunctionsarenotsupportedinsubspacemode,causingtheenginetofallbacktocellbycell mode.Butevenwhenevaluatinganexpressionincellbycellmode,theenginecanreturntosubspace mode.

2.3.2.2 Expensive vs. Inexpensive Query Plans

Itcanbecostlytobuildaqueryplan.Infact,thecostofbuildinganexecutionplancanexceedthecost ofqueryexecution.TheAnalysisServicesenginehasacoarseclassificationschemeexpensiveversus inexpensive.Aplanisdeemedexpensiveifcellbycellmodeisusedorifcubedatamustbereadtobuild theplan.Otherwisetheexecutionplanisdeemedinexpensive. 48

Cubedataisusedinqueryplansinseveralscenarios.Somequeryplansresultinthemappingofone membertoanotherbecauseofMDXfunctionssuchasPrevMemberandParent.Themappingsarebuilt fromcubedataandmaterializedduringtheconstructionofthequeryplans.TheIIf,CASE,andIF functionscangenerateexpensivequeryplansaswell,shoulditbenecessarytoreadcubedatainorder topartitioncubespaceforevaluationofoneofthebranches.Formoreinformation,seeIIfFunctionin SQLServer2008AnalysisServices.

2.3.2.3 Expression Sparsity

Anexpressionssparsityreferstothenumberofcellswithnonnullvaluescomparedtothetotalnumber ofcellsintheresultoftheevaluationoftheexpression.Iftherearerelativelyfewnonnullvalues,the expressionistermedsparse.Iftherearemany,theexpressionisdense.Asweshallseelater,whether anexpressionissparseordensecaninfluencethequeryplan. Buthowcanyoutellwhetheranexpressionisdenseorsparse?Considerasimplenoncalculated measureisitdenseorsparse?InOLAP,basefactmeasuresareconsideredsparsebytheAnalysis Servicesengine.Thismeansthatthetypicalmeasuredoesnothavevaluesforeveryattributemember. Forexample,acustomerdoesnotpurchasemostproductsonmostdaysfrommoststores.Infactits thequitetheopposite.Atypicalcustomerpurchasesasmallpercentageofallproductsfromasmall numberofstoresonafewdays.Thefollowingtablelistssomeothersimplerulesforpopular expressions. Expression Regularmeasure ConstantValue Scalarexpression;forexample,count, .properties <exp1>+<exp2> <exp1><exp2> <exp1>*<exp2> <exp1>/<exp2> Sum(<set>,<exp>) Aggregate(<set>,<exp>) IIf(<cond>,<exp1>,<exp2>) Sparse/dense Sparse Dense(excludingconstantnullvalues, true/falsevalues) Dense Sparseifbothexp1andexp2aresparse; otherwisedense Sparseifeitherexp1orexp2issparse; otherwisedense Sparseif<exp1>issparse;otherwisedense Inheritedfrom<exp> Determinedbysparsityofdefaultbranch (refertoIIffunction)

Formoreinformationaboutsparsityanddensity,seeGrossmargindensevs.sparseblockevaluation modeinMDX(http://sqlblog.com/blogs/mosha/archive/2008/11/01/grossmargindensevssparse blockevaluationmodeinmdx.aspx).

49

2.3.2.4 Default Values

Everyexpressionhasadefaultvaluethevaluetheexpressionassumesmostofthetime.Thequery processorcalculatesanexpressionsdefaultvalueandreusesacrossmostofitsspace.Mostofthetime thisisnullbecauseoftentimes(butnotalways)theresultofanexpressionwithnullinputvaluesisnull. Theenginecanthencomputethenullresultonce,andthenitneedstocomputeonlyvaluesforthe muchreducednonnullspace. AnotherimportantuseofthedefaultvaluesisintheconditionintheIIffunction.Knowingwhichbranch isevaluatedmoreoftendrivestheexecutionplan.Thedefaultvaluesofsomepopularexpressionsare listedinthefollowingtable. Expression Regularmeasure IsEmpty(<regularmeasure>) Defaultvalue Null True Comment None. Themajorityoftheoreticalspaceis occupiedbynullvalues.Therefore, IsEmptywillreturnTruemostoften. Valuesforbothmeasuresareprincipally null,sothisevaluatestoTruemostofthe time. Thisisdifferentthancomparingvalues theengineassumesthatdifferent membersarecomparedmostofthetime.

<regularmeasureA>=<regular measureB> <memberA>IS<memberB>

True

False

2.3.2.5 Varying Attributes

Cellvaluesmostlydependonattributecoordinates.Butsomecalculationsdonotdependonevery attribute.Forexample,thefollowingexpressiondependsonlyontheCustomerattributeinthecustomer dimension. [Customer].[CustomerGeography].properties("PostalCode") Whenthisexpressionisevaluatedoverasubspaceinvolvingotherattributes,anyattributesthe expressiondoesntdependoncanbeeliminated,andthentheexpressioncanberesolvedandprojected backovertheoriginalsubspace.Theattributesanexpressiondependsonaretermeditsvarying attributes.Forexample,considerthefollowingquery.
withmembermeasures.Zipas [Customer].[CustomerGeography].currentmember.properties("PostalCode") selectmeasures.zipon0, [Product].[Category].memberson1 from[AdventureWorks]

50

where[Customer].[CustomerGeography].[Customer].&[25818]

Theexpressiondependsonthecustomerattributeandnotthecategoryattribute;therefore,customer isavaryingattributeandcategoryisnot.Inthiscasetheexpressionisevaluatedonlyonceforthe customerandnotasmanytimesasthereareproductcategories.

2.3.3 Optimizing MDX

Debuggingcalculationperformanceissuesacrossacubecanbedifficultiftherearemanycalculations. Thefirststepistotrytonarrowdownwheretheproblemexpressionisandthenapplybestpracticesto theMDX.Inordertonarrowdownaproblem,youwillfirstneedabaseline.

2.3.3.1 Baselining Query Speeds

Beforebeginningoptimization,youneedreproduciblecoldcachebaselinemeasurements. Todothis,youshouldbeawareofthefollowingthreeAnalysisServicescaches: Theformulaenginecache Thestorageenginecache Thefilesystemcache

BoththeAnalysisServicesandtheoperatingsystemcachesneedtobeclearedbeforeyoustarttaking measurements. 2.3.3.1.1 Clearing the Analysis Services Caches TheAnalysisServicesformulaengineandstorageenginecachescanbeclearedwiththeXMLA ClearCachecommand.YoucanuseSQLServerManagementStudiotorunClearCache.
<ClearCache xmlns="http://schemas.microsoft.com/analysisservices/2003/engine"> <Object> <DatabaseID><database name></DatabaseID> </Object> </ClearCache>

2.3.3.1.2 Clearing the Operating System Caches ThefilesystemcacheisabithardertogetridofbecauseitresidesinsideWindowsitself.Youcanuse anyofthefollowingtoolstoperformthistask: Fsutil.exe:WindowsFileSystemUtility IfyouhavecreatedaseparateWindowsvolumeforthecubedatabase,youcandismountthe volumeitselfusingthefollowingcommand: fsutil.exevolumedismount<DriveLetter|MountPoint> Thisclearsthefilesystemcacheforthisdriveletterormountpoint.Ifthecubedatabaseresides onlyonthislocation,runningthiscommandresultsinacleanfilesystemcache.

51

RAMMap:Sysinternalstool Alternatively,youcanuseRAMMapfromSysinternals(asofthiswriting,RAMMapv1.11is availableat:http://technet.microsoft.com/enus/sysinternals/ff700229.aspx).RAMMapcan helpyouunderstandhowWindowsmanagesmemory.Thistoolnotonlyallowsyoutoreadthe filesystemcachecontent,italsoallowsyoutopurgeit.Ontheemptymenu,clickEmpty SystemWorkingSet,andthenclickEmptyStandbyList.Thisclearsthefilesystemcacheforthe entiresystem.NotethatwhenRAMMapstartsup,ittemporarilyfreezesthesystemwhileit readsthememorycontentthiscantakesometimeonalargemachine.Hence,RAMMap shouldbeusedwithcare. AnalysisServicesStoredProcedureProject(CodePlex):FileSystemCacheclass ThereiscurrentlyaCodePlexprojectcalledtheAnalysisServicesStoredProcedureProjectfound at:http://asstoredprocedures.codeplex.com/wikipage?title=FileSystemCache.Thisproject containscodeforautilitythatenablesyoutoclearthefilesystemcacheusingastored procedurethatyoucanrundirectlyonAnalysisServices.

NotethatneitherFSUTILnorRAMMapshouldbeusedinproductioncubesbothcausedisruptionto service.AlsonotethatneitherRAMMapnortheAnalysisServicesStoredProceduresProjectis supportedbyMicrosoft. 2.3.3.1.3 Measure Query Speeds Whenallcachesareclear,youshouldinitializethecalculationscriptbyexecutingaquerythatreturns andcachesnothing.Hereisanexample.

select{}on0from[AdventureWorks]

ExecutethequeryyouwanttooptimizeandthenuseSQLServerProfilerwiththeStandard(default) traceandtheseadditionaleventsenabled: QueryProcessing\QuerySubcubeVerbose QueryProcessing\GetDataFromAggregation

Savetheprofilertrace,becauseitcontainsimportantinformationthatyoucanusetodiagnoseslow querytimes.

52


Figure 2424: Sample trace

Thetextforthequerysubcubeverboseeventdeservessomeexplanation.Itcontainsinformationfor eachattributeineverydimension: 0:Indicatesthattheattributeisnotincludedinquery(theAllmemberishit). *:Indicatesthateverymemberoftheattributewasrequested. +:Indicatesthattwoormoremembersoftheattributewererequested. :Indicatesthataslicebelowgranularityisrequested. <integervalue>:Indicatesthatasinglememberoftheattributewashit.Theinteger representsthemembersdataID(aninternalidentifiergeneratedbytheengine).

Formoreinformationaboutthequerysubcubeverboseeventtextdata,seethefollowing: IdentifyingandResolvingMDXQueryPerformanceBottlenecksinSQLServer2005Analysis Services(http://sqlcat.com/sqlcat/b/whitepapers/archive/2007/12/16/identifyingand resolvingmdxqueryperformancebottlenecksinsqlserver2005analysisservices.aspx) ConfiguringtheAnalysisServicesQueryLog(http://msdn.microsoft.com/en us/library/cc917676.aspx):RefertotheTheDatasetColumnintheQueryLogTablesection

SQLServerManagementStudiodisplaysthetotalquerytime.Butbecareful:Thistimeistheamountof timetakentoretrieveanddisplaythecellset.Forlargeresults,thetimetorenderthecellsetonthe clientcanrivalthetimeittooktheservertogenerateit.InsteadofusingSQLServerManagement Studio,usetheSQLServerProfilerQueryEndeventtomeasurehowlongthequerytakesfromthe serversperspectiveandgettheAnalysisServicesengineduration.

2.3.3.2 Isolating the Problem

Diagnosingtheproblemmaybestraightforwardifasimplequerycallsoutaspecificcalculation(in whichcaseyoushouldcontinuetothenextsection),butiftherearechainsofexpressionsoracomplex 53

query,itcanbetimeconsumingtolocatetheproblem.Trytoreducethequerytothesimplest expressionpossiblethatcontinuestoreproducetheperformanceissue.Ifpossible,removeexpressions suchasMDXscripts,unaryoperators,measureexpressions,custommemberformulas,semiadditive measures,andcustomrollupproperties.Withsomeclientapplications,thequerygeneratedbythe clientitself,notthecube,canbetheproblem.Forexample,problemscanarisewhenclientapplications generatequeriesthatdemandlargedatavolumes,pushdowntounnecessarilylowgranularities, unnecessarilybypassaggregations,orcontainquerycalculationsthatbypasstheglobalandsession queryprocessorcaches.Ifyoucanconfirmthattheissueisinthecubeitself,commentoutcalculated membersinthecubeorqueryuntilyouhavenarroweddowntheoffendingcalculation.Usingabinary chopmethodisusefultoquicklyreducethequerytothesimplestformthatreproducestheissue. Experiencedtunerswillbeabletoquicklynarrowinontypicalcalculationissues. Whenyouhaveremovedcalculationsuntiltheperformanceissuereproduces,thefirststepisto determinewhethertheproblemliesinthequeryprocessor(theformulaengine)orthestorageengine. Todeterminetheamountoftimetheenginespendsscanningdata,usetheSQLServerProfilertrace createdearlier.Limittheeventstononcachedstorageengineretrievalsbyselectingonlythequery subcubeverboseeventandfilteringonevent subclass = 22.Theresultwillbesimilartothe following.

Figure 2525: Trace of query subcube events

Ifthemajorityoftimeisspentinthestorageenginewithlongrunningquerysubcubeevents,the problemislikelywiththestorageengine.Inthiscase,consideroptimizingdimensiondesign,designing aggregations,orusingpartitionstoimprovequeryperformance.Inaddition,youmaywanttoconsider optimizingthedisksubsystem. Ifthemajorityoftimeisnotspentinthestorageenginebutinthequeryprocessor,focusonoptimizing theMDXscriptorthequeryitself.Note,theproblemcaninvolveboththeformulaandstorageengines. AfragmentedqueryspacecanbediagnosedwithSQLServerProfilerifyouseemanyquerysubcube eventsgeneratedbyasinglequery.Eachrequestmaynottakelong,butthesumofthemmay.Ifthisis thecase,considerwarmingthecachetomakesuresubcubesandcalculationsarealreadycached.Also, considerrewritingthequerytoremovearbitraryshapes,becausearbitrarysubcubescannotbecached. Formoreinformation,seeCacheWarminginalatersection.

54

IfthecubeandMDXqueryarealreadyfullyoptimized,youmayconsiderdoingthread,memory,and configurationtuningofthecube.Youmayevenwanttolookatlargerhardware.Serverleveltuning techniquesaredescribedinPart2. References: TheSQLServer2008R2AnalysisServicesOperations Guide(http://sqlcat.com/sqlcat/b/whitepapers/archive/2011/06/01/sqlserver2008r2analysis servicesoperationsguide.aspx) PredeploymentI/OBestPractices (http://sqlcat.com/sqlcat/b/whitepapers/archive/2007/11/21/predeploymentiobest practices.aspx):TheconceptsinthisdocumentprovideanoverviewofdiskI/Oanditsimpact queryperformance;focusontherandomI/Ocontext. ScalableSharedDatabasesPart5 (http://sqlcat.com/sqlcat/b/whitepapers/archive/2011/06/01/sqlserver2008r2analysis servicesoperationsguide.aspx):Reviewtobetterunderstandonqueryperformanceincontext ofrandomI/Ovs.sequentialI/O.

2.3.3.3 Cell-by-Cell Mode vs. Subspace Mode

Almostalways,performanceobtainedbyusingsubspace(orblockcomputation)modeissuperiorto thatobtainedbyusingcellbycell(nornave)mode.Formoreinformation,includingthelistoffunctions supportedinsubspacemode,seePerformanceImprovementsforMDXinSQLServer2008Analysis Services(http://msdn.microsoft.com/enus/library/bb934106(v=SQL.105).aspx)inSQLServerBooks Online. Thefollowingtableliststhemostcommonreasonsforleavingsubspacemode. Featureorfunction Setaliases Comment Replacewithasetexpressionratherthananalias.Forexample,thisquery operatesinsubspacemode.
with member measures.SubspaceMode as sum( [Product].[Category].[Category].members, [Measures].[Internet Sales Amount] ) select {measures.SubspaceMode,[Measures].[Internet Sales Amount]} on 0 , [Customer].[Customer Geography].[Country].memberson 1 from [Adventure Works] cellpropertiesvalue

However,almostthesamequery,wherethesetisreplacedwithanalias, operatesincellbycellmode: 55

with set y as [Product].[Category].[Category].members member measures.Naive as sum( y, [Measures].[Internet Sales Amount] ) select {measures.Naive,[Measures].[Internet Sales Amount]} on 0 , [Customer].[Customer Geography].[Country].memberson 1 from [Adventure Works]

cellpropertiesvalue Note:ThisfunctionalityhasbeenfixedwiththelatestservicepackofSQL Server2008R2AnalysisServices. Latebindinginfunctions: Latebindingfunctionsarefunctionsthatdependonquerycontextand cannotbestaticallyevaluated.Forexample,thefollowingcodeisstatically bound. LinkMember,StrToSet, StrToMember, withmember measures.x as StrToValue

(strtomember("[Customer].[Customer Geography].[Country].&[Australia]"),[Measures].[Internet Sales Amount]) select measures.x on 0, [Customer].[Customer Geography].[Country].memberson 1 from [Adventure Works] cell properties value

Aqueryislateboundifanargumentcanbeevaluatedonlyincontext.
withmember measures.x as (strtomember([Customer].[Customer Geography].currentmember.uniquename),[Measures].[Internet Sales Amount]) select measures.x on 0, [Customer].[Customer Geography].[Country].memberson 1 from [Adventure Works] cell properties value

Userdefinedstored procedures

LookupCube Applicationofcelllevel security

Userdefinedstoredproceduresareevaluatedincellbycellmode.Some popularMicrosoftVisualBasicforApplications(VBA)functionsarenatively supportedinMDX,buttheyarestillnotoptimizedtoworkinsubspace mode. Linkedmeasuregroupsareoftenaviablealternative. Bydefinition,celllevelsecurityrequirescellbycellevaluationtoensurethe correctsecuritycontextisapplied;thereforeperformanceimprovementsof blockcomputationcannotbeapplied.

56

2.3.3.4 Avoid Assigning Nonnull Values to Otherwise Empty Cells

TheAnalysisServicesengineisveryefficientatusingsparsityofthedatatoimproveperformance. AddingcalculationswithnonemptyvaluesreplacingemptyvaluesdoesnotallowAnalysisServicesto eliminatetheserows.Forexample,thefollowingqueryreplacesemptyvalueswiththedash;therefore thenonemptykeyworddoesnoteliminatethem.
withmembermeasures.xas iif(notisempty([Measures].[InternetSalesAmount]),[Measures].[InternetSales Amount],"") selectdescendants([Date].[Calendar].[CalendarYear].&[2004])on0, nonempty[Customer].[CustomerGeography].[Customer].memberson1 from[AdventureWorks] wheremeasures.x

Note,nonemptyoperatesoncellvaluesbutnotonformattedvalues.Inrarecasesyoucaninsteaduse theformatstringtoreplacenullvalueswiththesamecharacterwhilestilleliminatingemptyrowsand columnsinroughlyhalftheexecutiontime.

withmembermeasures.xas [Measures].[InternetSalesAmount],FORMAT_STRING="#.00;(#.00);#.00;" selectdescendants([Date].[Calendar].[CalendarYear].&[2004])on0, nonempty[Customer].[CustomerGeography].[Customer].memberson1 from[AdventureWorks] wheremeasures.x

Thereasonthiscanonlybeusedinrarecasesisthatthequeryisnotequivalentthesecondquery eliminatescompletelyemptyrows.Moreimportantly,neitherExcelnorSQLServerReportingServices supportsthefourthargumentintheformat_string. References: Formoreinformationaboutusingtheformat_stringcalculationproperty,seeFORMAT_STRING Contents(MDX)(http://msdn.microsoft.com/enus/library/ms146084.aspx)inSQLServerBooks Online. FormoreinformationabouthowExcelusestheformat_stringproperty,seeCreateordeletea customnumberformat(http://office.microsoft.com/enus/excelhelp/createordeletea customnumberformatHP010342372.aspx).

57

2.3.3.5 Sparse/Dense Considerations with expr1 * expr2 Expressions

Whenyouwriteexpressionsasproductsoftwootherexpressions,placethesparseroneontheleft handside.Recall,anexpressionissparseiftherearefewnonnullvaluescomparedtothetotalnumber ofcells;formoreinformation,seeExpressionSparsityearlierinthissection. Considerthefollowingtwoqueries,whichhavethesignatureofacurrencyconversioncalculationof applyingtheexchangerateatleavesofthedatedimensioninAdventureWorks.Theonlydifferenceis thattheorderoftheexpressionsintheproductofthecellcalculationchanges.Theresultsarethesame, butusingthesparserinternetsalesamountfirstresultsinabouta10%savings.(Thatsnotmuchinthis case,butitcouldbesubstantiallymoreinothers.Savingsdependsonrelativesparsitybetweenthetwo expressions,andperformancebenefitsmayvary). SparseFirst
withcellCALCULATIONxfor'({[Measures].[InternetSalesAmount]},leaves([Date]))' as[Measures].[InternetSalesAmount]* ([Measures].[AverageRate],[DestinationCurrency].[DestinationCurrency].&[EURO]) select nonempty[Date].[Calendar].memberson0, nonempty[Product].[ProductCategories].memberson1 from[AdventureWorks] where([Measures].[InternetSalesAmount],[Customer].[CustomerGeography].[State Province].&[BC]&[CA])

DenseFirst
withcellCALCULATIONxfor'({[Measures].[InternetSalesAmount]},leaves([Date]))' as ([Measures].[AverageRate],[DestinationCurrency].[DestinationCurrency].&[EURO])* [Measures].[InternetSalesAmount] select nonempty[Date].[Calendar].memberson0, nonempty[Product].[ProductCategories].memberson1 from[AdventureWorks] where([Measures].[InternetSalesAmount],[Customer].[CustomerGeography].[State Province].&[BC]&[CA])

58

2.3.3.6 IIf Function in SQL Server 2008 Analysis Services

TheIIfMDXfunctionisacommonlyusedexpressionthatcanbecostlytoevaluate.Theengine optimizesperformancebasedonafewsimplecriteria.TheIIffunctiontakesthreearguments:
iif(<condition>, <then branch>, <else branch>)

Wheretheconditionevaluatestotrue,thevaluefromthethenbranchisused;otherwisetheelse branchexpressionisused.Notethetermusedoneorbothbranchesmaybeevaluatedevenifthe valueisnotused.Itmaybecheaperfortheenginetoevaluatetheexpressionovertheentirespaceand useitwhenneededtermedaneagerplanthanitwouldbetochopupthespaceintoapotentially enormousnumberoffragmentsandevaluateonlywhereneededastrictplan. Note:OneofthemostcommonerrorsinMDXscriptingisusingIIfwhentheconditiondepends oncellcoordinatesinsteadofvalues.Iftheconditiondependsoncellcoordinates,usescopes andassignmentsasdescribedinsection2.Whenthisisdone,theconditionisnotevaluated overthespaceandtheenginedoesnotevaluateoneorbothbranchesovertheentirespace. Admittedly,insomecases,usingassignmentsforcessomeunwieldyscopingandrepetitionof assignments,butitisalwaysworthwhilecomparingthetwoapproaches. IIfconsiderations: 1) Thefirstconsiderationiswhetherthequeryplanisexpensiveorinexpensive. MostIIfconditionqueryplansareinexpensive,butcomplexnestedconditionswithmoreIIf functionscangotocellbycell. 2) Thenextconsiderationtheenginemakesiswhatvaluetheconditiontakesmost.Thisisdriven bytheconditionsdefaultvalue.Iftheconditionsdefaultvalueistrue,thethenbranchisthe defaultbranchthebranchthatisevaluatedovermostofthesubspace. Knowingafewsimplerulesonhowtheconditionisevaluatedhelpstodeterminethedefaultbranch: Insparseexpressions,mostcellsareempty.ThedefaultvalueoftheIsEmptyfunctionona sparseexpressionistrue. Comparisontozeroofasparseexpressionistrue. ThedefaultvalueoftheISoperatorisfalse. Iftheconditioncannotbeevaluatedinsubspacemode,thereisnodefaultbranch.

Forexample,oneofthemostcommonusesoftheIIffunctionistocheckwhetherthedenominatoris nonzero: 59

iif([Measures].[Internet Sales Amount]=0 , null , [Measures].[Internet Order Quantity]/[Measures].[Internet Sales Amount])

ThereisnocalculationonInternetSalesAmount;thereforeitisaregularmeasureexpressionanditis sparse.Thereforethedefaultvalueoftheconditionistrue.Thusthedefaultbranchisthethenbranch withthenullexpression. ThefollowingtableshowshoweachbranchofanIIffunctionisevaluated. Branchqueryplan Expensive Inexpensive Inexpensive Inexpensive Branchisdefault branch Notapplicable True False False Branchexpression sparsity Notapplicable Notapplicable Dense Sparse Evaluation Strict Eager Strict Eager

InSQLServer2008AnalysisServices,youcanoverrulethedefaultbehaviorwithqueryhints.
iif( [<condition> , <then branch> [hint [Eager | Strict]] , <else branch> [hint [Eager | Strict]] )

Herearethemostcommonscenarioswhereyoumightwanttochangethedefaultbehavior: Theenginedeterminesthequeryplanfortheconditionisexpensiveandevaluateseachbranch instrictmode. Theconditionisevaluatedincellbycellmode,andeachbranchisevaluatedineagermode. Thebranchexpressionisdensebuteasilyevaluated. Forexample,considerthefollowingsimpleexpression,whichtakestheinverseofameasure.

withmember measures.xas iif( [Measures].[InternetSalesAmount]=0 ,null ,(1/[Measures].[InternetSalesAmount])) select{[Measures].x}on0,

60

[Customer].[CustomerGeography].[Country].members* [Product].[ProductCategories].[Category].memberson1 from[AdventureWorks] cellpropertiesvalue

Thequeryplanisnotexpensive,theelsebranchisnotthedefaultbranch,andtheexpressionisdense, soitisevaluatedinstrictmode.Thisforcestheenginetomaterializethespaceoverwhichitis evaluated.ThiscanbeseeninSQLServerProfilerwithquerysubcubeverboseeventsselectedas displayedinFigure26.

Figure 2626: Default IIf query trace

NotethesubcubedefinitionfortheProductandCustomerdimensions(dimensions7and8respectively) withthe+indicatorontheCountryandCategoryattributes.Thismeansthatmorethanonebutnotall membersareincludedthequeryprocessorhasdeterminedwhichtuplesmeettheconditionand partitionedthespace,anditisevaluatingthefractionoverthatspace. Topreventthequeryplanfrompartitioningthespace,thequerycanbemodifiedasfollows(inbold).

withmember measures.xas iif(

61

[Measures].[InternetSalesAmount]=0 ,null ,(1/[Measures].[InternetSalesAmount])hinteager) select{[Measures].x}on0, [Customer].[CustomerGeography].[Country].members* [Product].[ProductCategories].[Category].memberson1 from[AdventureWorks] cellpropertiesvalue

Figure 2727: IIf trace with MDX query hints

Nowthesameattributesaremarkedwitha*indicator,meaningthattheexpressionisevaluatedover theentirespaceinsteadofapartitionedspace.

2.3.3.7 Cache Partial Expressions and Cell Properties

Partialexpressions(thosethatarepartofacalculatedmemberorassignment)arenotcached.Soifan expensivesubexpressionisusedmorethanonce,considercreatingaseparatecalculatedmemberto allowthequeryprocessortocacheandreuse.Forexample,considerthefollowing.
this=iif(<expensiveexpression>=0,1/<expensiveexpression>,null);

Therepeatedpartialexpressionscanbeextractedandreplacedwithahiddencalculatedmemberas follows. 62

 createmembercurrentcube.measures.MyPartialExpressionas<expensiveexpression>, visible=0; this=iif(measures.MyPartialExpression>=0,1/measures.MyPartialExpression,null);

Onlythevaluecellpropertyiscached.Ifyouhavecomplexcellpropertiestosupportsuchthingsas bubbleupexceptioncoloring,considercreatingaseparatecalculatedmeasure.Forexample,this expressionincludescolorinthedefinition,whichcreatesextraworkeverytimetheexpressionisused.

createmembercurrentcube.measures.[Value]as<exp>,backgroundColor=<complex expression>;

Thefollowingismoreefficientbecauseitcreatesacalculatedmeasuretohandlethecoloreffect.
createmembercurrentcube.measures.MyCellPropertyas<complexexpression>, visible=0; createmembercurrentcube.measures.[Value]as<exp>, backgroundColor=<MyCellProperty>;

2.3.3.8 Eliminate Varying Attributes in Set Expressions

Setexpressionsdonotsupportvaryingattributes.ThisimpactsallsetfunctionsincludingFilter, Aggregate,Avg,andothers.Youcanworkaroundthisproblembyexplicitlyoverwritinginvariant attributestoasinglemember. Forexample,inthiscalculation,theaverageofsalesonlyincludingthoseexceeding$100iscomputed.
withmembermeasures.AvgSalesas avg( filter( descendants([Customer].[CustomerGeography].[AllCustomers],,leaves) ,[Measures].[InternetSalesAmount]>100 ) ,[Measures].[InternetSalesAmount] ) selectmeasures.AvgSaleson0, [Customer].[CustomerGeography].[City].memberson1 from[AdventureWorks]

63

Onadesktopbox,thiscalculationtakesapproximately2:29.However,theaverageofsalesforall customerseverywheredoesnotdependonthecurrentcity(thisisjustanotherwayofsayingthatcityis notavaryingattribute).Youcanexplicitlyeliminatecityasavaryingattributebyoverwritingittotheall memberasfollows.

withmembermeasures.AvgSalesas avg( filter( descendants([Customer].[CustomerGeography].[AllCustomers],,leaves) ,[Measures].[InternetSalesAmount]>100 ) ,[Measures].[InternetSalesAmount] ) membermeasures.AvgSalesWithOverWriteas(measures.AvgSales,[AllCustomers]) selectmeasures.AvgSalesWithOverWriteon0, [Customer].[CustomerGeography].[City].memberson1 from[AdventureWorks]

Withthemodification,thisquerytakeslessthantwosecondstocomplete.Thefollowingisapartial viewaggregatingtheSQLServerProfilertracesofthetwoqueriesintheexamplebyEventClassand EventSubClass. EventClass>EventSubClass AvgSalesWithOverwrite AvgSales Events Duration Events Duration QueryCubeEnd 1 515 1 161526 SerialResultsEnd 1 499 1 161526 QueryDimension 586 GetDataFromCache>GetData 586 fromFlatCache QuerySubcube>NonCacheData 5 64 5 218 TheQuerySubcube>NonCacheDatadurationsarerelativelysmall,denotingthatmostofthequery calculationisdonebytheAnalysisServicesformulaengine.ThisisapparentwiththeAvgSales calculationbecausemostofthequerydurationscorrespondtotheSerialResultsevent,whichreports thestatusofserializingaxesandcells.Theuseof[All Customers]ensuresthattheexpressionis evaluatedonlyonceforeachCustomer,improvingperformance.

2.3.3.9 Eliminate Cost of Computing Formatted Values

Insomecircumstances,thecostofdeterminingtheformatstringforanexpressionoutweighsthecost ofthevalueitself.Todeterminewhetherthisappliestoaslowrunningquery,compareexecutiontimes withandwithouttheformattedvaluecellproperty,asinthefollowingquery.

64

select[Measures].[InternetAverageSalesAmount]on0from[AdventureWorks]cell propertiesvalue

Iftheresultisnoticeablefasterwithouttheformatting,applytheformattingdirectlyinthescriptas follows.
scope([Measures].[InternetAverageSalesAmount]); FORMAT_STRING(this)="currency"; endscope;

Executethequery(withformattingapplied)todeterminetheextentofanyperformancebenefit.

2.3.3.10

NON_EMPTY_BEHAVIOR

Insomesituations,itisexpensivetocomputetheresultofanexpression,evenifyouknowitwillbenull beforehandbasedonthevalueofsomeindicatortuple.InearlierversionsofSQLServerAnalysis Services,theNON_EMPTY_BEHAVIORpropertyissometimeshelpfulforthesekindsofcalculations. Whenthispropertyevaluatestonull,theexpressionisguaranteedtobenulland(mostofthetime)vice versa. Thispropertyoftentimesresultedinsubstantialperformanceimprovementsinpastreleases.However, startingwithSQLServer2008,thepropertyisoftentimesignored(becausetheengineautomatically dealswithnonemptycellsinmanycases)andcansometimesresultindegradedperformance.Eliminate itfromtheMDXscriptandadditbackafterperformancetestingdemonstratesimprovement. Forassignments,thepropertyisusedasfollows. this = <e1>; Non_Empty_Behavior(this) = <e2>; ForcalculatedmembersintheMDXscript,thepropertyisusedthisway.
createmembercurrentcube.measures.xas<e1>,non_empty_behavior=<e2>

InSQLServer2005AnalysisServices,therewerecomplexrulesonhowthepropertycouldbedefined, whentheengineuseditorignoredit,andhowtheenginewoulduseit.InSQLServer2008Analysis Services,thebehaviorofthispropertyhaschanged: 65

ItremainsaguaranteethatwhenNON_EMPTY_BEHAVIORisnullthattheexpressionmustalso benull.(Ifthisisnottrue,incorrectqueryresultscanstillbereturned.) However,thereverseisnotnecessarilytrue;thatis,theNON_EMPTY_BEHAVIORexpressioncan returnnonnullwhentheoriginalexpressionisnull. Theenginemoreoftenthannotignoresthispropertyanddeducesthenonemptybehaviorof theexpressiononitsown.

Ifthepropertyisdefinedandisappliedbytheengine,itissemanticallyequivalent(notperformance equivalent,however)tothefollowingexpression. this = <e1> * iif(isempty(<e2>), null, 1) TheNON_EMPTY_BEHAVIORpropertyisusedif<e2>issparseand<e1>isdenseor<e1>isevaluatedin thenavecellbycellmode.Iftheseconditionsarenotmetandboth<e1>and<e2>aresparse(thatis,if <e2>ismuchsparserthan<e1>),youmaybeabletoachieveimprovedperformancebyforcingthe behaviorasfollows. this = iif(isempty(<e2>), null, <e1>); TheNON_EMPTY_BEHAVIORpropertycanbeexpressedasasimpletupleexpressionincludingsimple membernavigationfunctionssuchas.prevmemberor.parentoranenumeratedset.Anenumeratedset isequivalenttoNON_EMPTY_BEHAVIORoftheresultantsum. References BelowarelinkstosomehandyMDXoptimizationarticles,books,andblogposts: Querycalculatedmembersinvalidateformulaenginecache (http://cwebbbi.wordpress.com/2009/01/30/formulacachingandqueryscope/)byChrisWebb Subselectpreventingcaching(http://cwebbbi.wordpress.com/2008/10/28/reportingservices generatedmdxsubselectsandformulacaching/)byChrisWebb Measuredatatypes (http://bidshelper.codeplex.com/wikipage?title=Measure%20Group%20Health%20Check&Proje ctName=bidshelper) Currencydatatype(http://sqlcat.com/sqlcat/b/technicalnotes/archive/2008/09/25/themany benefitsofmoneydatatype.aspx)

2.3.4 Aggregations
AnaggregationisadatastructurethatstoresprecalculateddatathatAnalysisServicesusestoenhance queryperformance.Youcandefinetheaggregationdesignforeachpartitionindependently.Each partitioncanbethoughtofasbeinganaggregationatthelowestgranularityofthemeasuregroup. Aggregationsthataredefinedforapartitionareprocessedoutoftheleaflevelpartitiondataby aggregatingittoahighergranularity.

66

Whenaqueryrequestsdataathigherlevels,theaggregationstructurecandeliverthedatamorequickly becausethedataisalreadyaggregatedinfewerrows.Asyoudesignaggregations,youmustconsider thequeryingbenefitsthataggregationsprovidecomparedwiththetimeittakestocreateandrefresh theaggregations.Infact,addingunnecessaryaggregationscanworsenqueryperformancebecausethe rarehitsmovetheaggregationintothefilecacheatthecostofmovingsomethingelseout. Whileaggregationsarephysicallydesignedpermeasuregrouppartition,theoptimizationtechniquesfor maximizingaggregationdesignapplywhetheryouhaveoneormanypartitions.Inthissection,unless otherwisestated,aggregationsarediscussedinthefundamentalcontextofacubewithasingle measuregroupandsinglepartition.Formoreinformationabouthowyoucanimprovequery performanceusingmultiplepartitions,seePartitionStrategy.

2.3.4.1 Detecting Aggregation Hits

UseSQLServerProfilertoviewhowandwhenaggregationsareusedtosatisfyqueries.Within SQLServerProfiler,thereareseveraleventsthatdescribehowaqueryisfulfilled.Theeventthat specificallypertainstoaggregationhitsistheGetDataFromAggregationevent.

Figure 2828: Scenario 1: SQL Server Profiler trace for cube with an aggregation hit

ThisfiguredisplaysaSQLServerProfilertraceofthequerysresolutionagainstacubewithaggregations. IntheSQLServerProfilertrace,theoperationsthatthestorageengineperformstoproducetheresult setarerevealed. ThestorageenginegetsdatafromAggregationC0000,0001,0000asindicatedbytheGetDataFrom Aggregationevent.Inadditiontotheaggregationname,AggregationC,Figure29displaysavector,000, 0001,0000,thatdescribesthecontentoftheaggregation.Moreinformationonwhatthisvector actuallymeansisdescribedinthenextsection,HowtoInterpretAggregations.Theaggregationdatais loadedintothestorageenginemeasuregroupcachefromwherethequeryprocessorretrievesitand returnstheresultsettotheclient. Whennoaggregationscansatisfythequeryrequest,noticethemissingGetDataFromAggregation eventfromthesamecubewithnoaggregationsasnotedinthefollowingfigure.

67

Figure 2929: Scenario 2: SQL Server Profiler trace for cube with no aggregation hit

Afterthequeryissubmitted,ratherthanretrievingdatafromanaggregation,thestorageenginegoesto thedetaildatainthepartition.Fromthispoint,theprocessisthesame.Thedataisloadedintothe storageenginemeasuregroupcache.

2.3.4.2 How to Interpret Aggregations

WhenAnalysisServicescreatesanaggregation,eachdimensionisnamedbyavector,indicating whethertheattributepointstotheattributeortotheAlllevel.TheAttributelevelisrepresentedby1 andtheAlllevelisrepresentedby0.Forexample,considerthefollowingexamplesofaggregation vectorsfortheproductdimension: AggregationByProductKeyAttribute=[ProductKey]:1[Color]:0[Subcategory]:0[Category]:0 or1000 AggregationByCategoryAttribute=[ProductKey]:0[Color]:0[Subcategory]:0[Category]:1or 0001 AggregationByProductKey.AllandColor.AllandSubcategory.AllandCategory.All=[Product Key]:0[Color]:0[Subcategory]:0[Category]:0or0000

Toidentifyeachaggregation,AnalysisServicescombinesthedimensionvectorsintoonelongvector path,alsocalledasubcube,witheachdimensionvectorseparatedbycommas. Theorderofthedimensionsinthevectorisdeterminedbytheorderofthedimensionsinthemeasure group.Tofindtheorderofdimensionsinthemeasuregroup,useoneofthefollowingtwotechniques: 1. WiththecubeopenedinSQLServerBusinessIntelligenceDevelopmentStudio,reviewtheorder ofdimensionsinameasuregroupontheCubeStructuretab.Theorderofdimensionsinthe cubeisdisplayedintheDimensionspane. 2. Asanalternative,reviewtheorderofdimensionslistedinthecubesXMLAdefinition. Theorderofattributesinthevectorforeachdimensionisdeterminedbytheorderofattributesinthe dimension.YoucanidentifytheorderofattributesineachdimensionbyreviewingthedimensionXML file. Forexample,thesubcubedefinition(0000,0001,0001)describesanaggregationforthefollowing:

68

Product All, All, All, All Customer All, All, All, State/Province Order Date All, All, All, Year

UnderstandinghowtoreadthesevectorsishelpfulwhenyoureviewaggregationhitsinSQLServer Profiler.InSQLServerProfiler,youcanviewhowthevectormapstospecificdimensionattributesby enablingtheQuerySubcubeVerboseevent.Insomecases(suchaswhenattributesaredisabled),it maybeeasiertoviewtheAggregationDesigntabandusetheAdvancedViewoftheaggregations.

2.3.4.3 Aggregation Tradeoffs

Aggregationscanimprovequeryresponsetimebuttheycanincreaseprocessingtimeanddiskstorage space,useupmemorythatcouldbeallocatedtocache,andpotentiallyslowthespeedofotherqueries. Thelattermayoccurbecausethereisadirectcorrelationbetweenthenumberofaggregationsandthe durationfortheAnalysisServicesstorageenginetoparsethem.Aswell,aggregationsmaycause thrashingduetotheirpotentialimpacttothefilesystemcache.Ageneralruleofthumbisthat aggregationsshouldbelessthan1/3thesizeofthefacttable.

2.3.4.4 BuildingAggregations
IndividualaggregationsareorganizedintocollectionsofaggregationscalledAggregationDesigns.You canapplyanAggregationDesigntomanypartitions.Aswell,onemeasuregroupcanhavemultiple AggregationDesignssothatyoucanchoosedifferentsetsofaggregationsfordifferentpartitions.To helpAnalysisServicessuccessfullyapplytheAggregationDesignalgorithm,youcanperformthe followingoptimizationtechniquestoinfluenceandenhancetheAggregationDesign.Inthissectionwe willdiscussthefollowing: Theimportanceofattributehierarchies Aggregationdesignandpartitions Specifyingstatisticsaboutcubedata Suggestingaggregationcandidates Usagebasedoptimization Largecubeaggregations Distinctcountpartitionaggregationconsiderations

2.3.4.4.1 Importance of Attribute Hierarchies Aggregationsworkbetterwhenthecubeisbasedonamultidimensionaldatamodelthatincludes naturalhierarchies.Whileitiscommoninrelationaldatabasestohaveattributesindependentofeach other,multidimensionalstarschemashaveattributesrelatedtoeachothertocreatenatural hierarchies.Thisisimportantbecauseitallowsaggregationsbuiltatalowerlevelofanaturalhierarchy tobeusedwhenqueryingatahigherlevel. Notethatattributesthatareexposedonlyinattributehierarchiesarenotautomaticallyconsideredfor aggregationbytheAggregationDesignWizard.Therefore,queriesinvolvingtheseattributesare satisfiedbysummarizingdatafromtheprimarykey.Withoutthebenefitofaggregations,query 69

performanceagainsttheseattributeshierarchiescanbeslow.Toenhanceperformance,itispossibleto flaganattributeasanaggregationcandidatebyusingtheAggregationUsageproperty.Formore informationaboutthistechnique,seeSuggestingAggregationCandidates.However,beforeyoumodify theAggregationUsageproperty,youshouldconsiderwhetheryoucantakeadvantageofuser hierarchies. 2.3.4.4.2 Aggregation Design and Partitions Whenyoudefineyourpartitions,theydonotnecessarilyhavetocontainuniformdatasetsor aggregationdesigns.Forexample,foragivenmeasuregroup,youmayhave3yearlypartitions, 11monthlypartitions,3weeklypartitions,and17dailypartitions.Heterogeneouspartitionswith differentlevelsofdetailallowsyoutomoreeasilymanagetheloadingofnewdatawithoutdisturbing existing,larger,andstalepartitions(moreonthisintheprocessingsection)andyoucandesign aggregationsforgroupsofpartitionsthatsharethesameaccesspattern.Foreachpartition,youcanuse adifferentaggregationdesign.Bytakingadvantageofthisflexibility,youcanidentifythosedatasets thatrequirehigheraggregationdesign. Considerthefollowingexample.Inacubewithmultiplemonthlypartitions,newdatamayflowintothe singlepartitioncorrespondingtothelatestmonth.Generallythatisalsothepartitionmostfrequently queried.Acommonaggregationstrategyinthiscaseistoperformusagebasedoptimizationtothemost recentpartition,leavingolder,lessfrequentlyqueriedpartitionsastheyare. Ifyouautomatepartitioncreation,itiseasytosimplysettheAggregationDesignIDforthenewpartition atcreationtimeandspecifythesliceforthepartition;nowitisreadytobeprocessed.Atalaterstage, youmaychoosetoupdatetheaggregationdesignforapartitionwhenitsusagepatternchanges again,youcanjustupdatetheAggregationDesignID,butyouwillalsoneedtoinvokeProcessIndexesso thatthenewaggregationdesigntakeseffectfortheprocessedpartition. 2.3.4.4.3 Specifying Statistics About Cube Data Tomakeintelligentassessmentsofaggregationcosts,thedesignalgorithmanalyzesstatisticsaboutthe cubeforeachaggregationcandidate.Examplesofthismetadataincludemembercountsandfacttable counts.Ensuringthatyourmetadataisuptodatecanimprovetheeffectivenessofyouraggregation design. Wheneveryouusemultiplepartitionsforagivenmeasuregroup,ensurethatyouupdatethedata statisticsforeachpartition.Morespecifically,itisimportanttoensurethatthepartitiondataand membercounts(suchasEstimatedRowsandEstimatedCountproperties)accuratelyreflectthespecific datainthepartitionandnotthedataacrosstheentiremeasuregroup. 2.3.4.4.4 Suggesting Aggregation Candidates WhenAnalysisServicesdesignsaggregations,theaggregationdesignalgorithmdoesnotautomatically considereveryattributeforaggregation.Consequently,inyourcubedesign,verifytheattributesthat areconsideredforaggregationanddeterminewhetheryouneedtosuggestadditionalaggregation candidates.Tostreamlinethisprocess,AnalysisServicesusestheAggregationUsagepropertyto determinewhichattributesitshouldconsider.Foreverymeasuregroup,verifytheattributesthatare 70

automaticallyconsideredforaggregationandthendeterminewhetheryouneedtosuggestadditional aggregationcandidates. AggregationUsageRules AnaggregationcandidateisanattributethatAnalysisServicesconsidersforpotentialaggregation.To determinewhetherornotaspecificattributeisanaggregationcandidate,thestorageenginerelieson thevalueoftheAggregationUsageproperty.TheAggregationUsagepropertyisassignedapercube attribute,soitgloballyappliesacrossallmeasuregroupsandpartitionsinthecube.Foreachattributein acube,theAggregationUsagepropertycanhaveoneoffourpotentialvalues:Full,None,Unrestricted, andDefault. FullEveryaggregationforthecubemustincludethisattributeorarelatedattributethatis lowerintheattributechain.Forexample,youhaveaproductdimensionwiththefollowing chainofrelatedattributes:Product,ProductSubcategory,andProductCategory.Ifyouspecify theAggregationUsageforProductCategorytobeFull,AnalysisServicesmaycreatean aggregationthatincludesProductSubcategoryasopposedtoProductCategory,giventhat ProductSubcategoryisrelatedtoCategoryandcanbeusedtoderiveCategorytotals. NoneNoaggregationforthecubecanincludethisattribute. UnrestrictedNorestrictionsareplacedontheaggregationdesigner;however,theattribute muststillbeevaluatedtodeterminewhetheritisavaluableaggregationcandidate. DefaultThedesignerappliesadefaultrulebasedonthetypeofattributeanddimension.This isthedefaultvalueoftheAggregationUsageproperty.

Thedefaultruleishighlyconservativeaboutwhichattributesareconsideredforaggregation.The defaultruleisbrokendownintofourconstraints. DefaultConstraint1UnrestrictedForadimensionsmeasuregroupgranularityattribute, defaultmeansUnrestricted.Thegranularityattributeisthesameasthedimensionskey attributeaslongasthemeasuregroupjoinstoadimensionusingtheprimarykeyattribute. DefaultConstraint2NoneforSpecialDimensionTypesForallattributes(exceptAll)in manytomany,nonmaterializedreferencedimensions,anddataminingdimensions,default meansNone.Thismeansyoucansometimesbenefitfromcreatingleaflevelprojectionsfor manytomanydimensions.Note,thesedefaultsdonotapplyforparentchilddimensions;for moreinformation,seetheSpecialConsiderations>ParentChilddimensionssection. DefaultConstraint3UnrestrictedforNaturalHierarchiesAnaturalhierarchyisauser hierarchywhereallattributesparticipatinginthehierarchycontainattributerelationshipstothe attributesourcingthenextlevel.Forsuchattributes,defaultmeansUnrestricted,exceptfor nonaggregatableattributes,whicharesettoFull(eveniftheyarenotinauserhierarchy). DefaultConstraint4NoneForEverythingElse.Forallotherdimensionattributes,default meansNone.

71

AggregationUsageGuidelines InlightofthebehavioroftheAggregationUsageproperty,usethefollowingguidelines: AttributesexposedsolelyasattributehierarchiesIfagivenattributeisonlyexposedasan attributehierarchysuchasColor,youmaywanttochangeitsAggregationUsagepropertyas follows. o First,changethevalueoftheAggregationUsagepropertyfromDefaulttoUnrestricted iftheattributeisacommonlyusedattributeoriftherearespecialconsiderationsfor improvingtheperformanceinaparticularpivotordrilldown.Forexample,ifyouhave highlysummarizedscorecardstylereports,youwanttoensurethattheusers experiencegoodinitialqueryresponsetimebeforedrillingaroundintomoredetail. o WhilesettingtheAggregationUsagepropertyofaparticularattributehierarchyto Unrestrictedisappropriateissomescenarios,donotsetallattributehierarchiesto Unrestricted.Increasingthenumberofattributestobeconsideredincreasesthe problemspacetheaggregationalgorithmmustconsider.Thewizardcantakeatleastan hourtocompletethedesignandconsiderablymuchmoretimetoprocess.Setthe propertytoUnrestrictedonlyforthecommonlyqueriedattributehierarchies.The generalruleisfivetotenUnrestrictedattributesperdimension. o Next,changethevalueoftheAggregationUsagepropertyfromDefaulttoFullinthe unusualcasethatitisusedinvirtuallyeveryqueryyouwanttooptimize.Thisisarare case,andthischangeshouldbemadeonlyforattributesthathavearelativelysmall numberofmembers. InfrequentlyusedattributesForattributesparticipatinginnaturalhierarchies,youmaywant tochangetheAggregationUsagepropertyfromDefaulttoNoneifuserswouldonly infrequentlyuseit.Usingthisapproachcanhelpyoureducetheaggregationspaceandgetto thefivetotenUnrestrictedattributesperdimension.Forexample,youmayhavecertain attributesthatareonlyusedbyafewadvanceduserswhoarewillingtoacceptslightlyslower performance.Inthisscenario,youareessentiallyforcingtheaggregationdesignalgorithmto spendtimebuildingonlytheaggregationsthatprovidethemostbenefittothemajorityofusers.

2.3.4.4.5 Usage-Based Optimization TheUsageBasedOptimizationWizardreviewsthequeriesinthequerylog(whichyoumustsetup beforehand)anddesignsaggregationsthatcoveruptothetop100slowestqueries.UsetheUsage BasedOptimizationWizardwitha100%performancegainthiswilldesignaggregationstoavoidhitting thepartitiondirectly. Aftertheaggregationsaredesigned,youcanaddthemtotheexistingdesignorcompletelyreplacethe design.Becarefuladdingthemtotheexistingdesignthetwodesignsmaycontainaggregationsthat servealmostidenticalpurposesthatwhencombinedareredundantwithoneanother.Aswell, aggregationdesignshaveacostlymetadataimpactdontoverdesignbuttrytokeepthenumberof aggregationdesignspermeasuregrouptoaminimum.Inspectthenewaggregationscomparedtothe

72

oldandensuretherearenonearduplicates.Theaggregationdesigncanbecopiedtootherpartitionsin SQLServerManagementStudioorBusinessIntelligenceDesignStudio. References: ReintroducingUsageBasedOptimizationinSQLServer2008AnalysisServices (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2008/11/18/reintroducingusagebased optimizationinsqlserver2008analysisservices.aspx) AnalysisServices2005AggregationDesignStrategy (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2007/09/11/analysisservices2005 aggregationdesignstrategy.aspx) MicrosoftSQLServerCommunitySamples:AnalysisServices (http://sqlsrvanalysissrvcs.codeplex.com/):ThisCodePlexprojectcontainsmanyusefulAnalysis ServicesCodePlexsamples,includingtheAggregationManager

2.3.4.4.6 Large Cube Aggregation Considerations Itisimportanttonotethatsmallcubesmaynotneedaggregations,becauseaggregationsarenoteven builtforpartitionswithfewerrecordsthantheIndexBuildThreshold(whichhasadefaultvalueof4096). EvenifthecubepartitionsexceedtheIndexBuildThreshold,aggregationsthatarecorrectlydesignedfor smallercubesmaynotbethecorrectonesforlargecubes. However,ascubesbecomelarger,itbecomesmoreimportanttodesignaggregationsandtodoso correctly.Asageneralruleofthumb,MOLAPperformanceisapproximatelybetween10and40million rowspersecondpercore,plustheI/Oforaggregatingdata. Itisimportanttonotethatlargercubeshavemoreconstraintssuchassmallprocessingwindowsand/or notenoughdiskspace.Thereforeitmaybedifficulttocreateallofyourdesiredaggregations.Theresult isatradeoffindesigningaggregationstobeconsideredmorecarefully.

2.3.5 Cache Warming

Cachewarmingcanbealastditcheffortforimprovingtheperformanceofaquery.Thefollowing sectionsdescribeguidelinesandimplementationstrategiesforcachewarming.

2.3.5.1 Cache Warming Guidelines

Duringquerying,memoryisprimarilyusedtostorecachedresultsinthestorageengineandquery processorcaches.Tooptimizethebenefitsofcaching,youcanoftenincreasequeryresponsivenessby preloadingdataintooneorbothofthesecaches.Thiscanbedonebyeitherpreexecutingoneormore queriesorusingtheCREATECACHEstatement(whichreturnsnocellsetsandhastheadvantageof executingfasterbecauseitbypassesthequeryprocessor).Thisprocessiscalledcachewarming. Whenpossible,AnalysisServicesreturnsresultsfromtheAnalysisServicesdatacachewithoutusing aggregations(becauseitisthefastestwaytogetdata).Withsmallercubestheremaybeenough memorytokeepalargeportionofthedatainthecache.Inthiscase,aggregationsarenotneededand 73

existingaggregationsmayneverbeused.Inthisscenario,cachewarmingcanbeusedsothatuserswill alwayshaveexcellentperformance. Butwithlargercubes,theremaybeinsufficientmemorytokeepenoughofthedataincache.Forthat matter,cachedresultscanbepushedoutbyotherqueryresults.Hence,cachewarmingwillonlyhelpa portionofthequeriesitisimportanttocreatewelldesignedaggregationstoprovidesolidquery performance.Butbecauseofthememorybottlenecks,itisimportanttonotethattoomany aggregationsmaythrashthecacheasdifferentdataresultsetsandaggregationsarerequestedand swappedfromthecache.

2.3.5.2 Implementing a Cache Warming Strategy

Whilecachewarmingcanimprovetheperformanceofaquery,youshouldnotethatthereisa significantdifferencebetweentheperformanceofthequeryonacoldcacheandawarmcache.Aswell, itisimportanttoensurethereisenoughmemoryavailablesothatthecacheisnotbeingthrashed. Towarmthecache,itisimportanttorememberthattheAnalysisServicesformulaenginecanonlybe warmedbyMDXqueries.Towarmthestorageenginecaches,youcanusetheWITHCACHEorCREATE CACHEstatements: Todiscoverwhatneedstobecached(whichcanbedifficultattimes),useSQLServerProfilerto tracethequeryexecutionandexaminethesubcubeevents. Findingmanysubcuberequeststothesamegrainmayindicatethatthequeryprocessoris makingmanyrequestsforslightlydifferentdata,resultinginthestorageenginemakingmany smallbuttimeconsumingI/Orequestswhereitcouldmoreefficientlyretrievethedataen masseandthenreturnresultsfromcache. Topreexecutequeries,createanapplication(orusesomethinglikeascmd)thatexecutesaset ofgeneralizedqueriestosimulatetypicaluseractivityinordertoexpeditetheprocessof populatingthecache.ExecutethesequeriespostAnalysisServicesstartuporpostprocessingto preloadthecachepriortouserqueries. Todeterminehowtogeneralizeyourqueries,youcanpotentiallyrefertotheAnalysisServices querylogtodeterminethedimensionattributestypicallyqueried.Becarefulwhenyou generalizebecauseyoumayincludeattributesorsubcubesthatarenotbeneficialand unnecessarilytakeupcache. Whentestingtheeffectivenessofdifferentcachewarmingqueries,youshouldemptythequery resultscachebetweeneachtesttoensurethevalidityofyourtesting. Becausecachedresultscanbepushedoutbyotherqueryresults,itmaybenecessaryto schedulerefreshesofthecacheresults.Also,limitcachewarmingtowhatcanfitinmemory, leavingenoughforotherqueriestobecached.

References:

74

HowtowarmuptheAnalysisServicesdatacacheusingCreateCachestatement? (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2007/09/11/howtowarmuptheanalysis servicesdatacacheusingcreatecachestatement.aspx)

2.3.6 Scale-Out
IfyouhavemanyconcurrentusersqueryingyourAnalysisServicescubes,apotentialqueryperformance solutionistoscaleoutyourAnalysisServicesqueryservers.Therearedifferentformsofscaleout, whicharediscussedinPart2,butthebasicprincipleisthatyouhavemultiplequeryserversaimedat thesamedatabase(orthedatabaseisreplicated)sotherearemultipleserverstoaddressuserqueries. Thiscanbebeneficialinthecaseslikethefollowing: Incaseswhereyourserverisundermemorypressureduetoconcurrency,scalingoutallowsyou todistributethequeryloadtomultipleservers,thusalleviatingmemorybottlenecksonasingle server.Memorypressurecanbecausedbymanyissues,including(butnotlimitedto): o Usersexecutingmanydifferentuniquequeriesthusfillingupandthrashingavailable cache. o Complexorlargequeriesrequiringlargesubcubesthusrequiringalargememoryspace. o Toomanyconcurrentusersaccessingthesameserver. YouhavemanylongrunningqueriesagainstyourAnalysisServicescube,whichwill: o Blockotherqueries. o Blockprocessingcommits. Inthiscase,scalingoutthelongrunningqueriestoseparateserverscanhelpalleviate contentionproblems. References: SQLServer2008R2AnalysisServicesOperationsGuide (http://sqlcat.com/sqlcat/b/whitepapers/archive/2011/06/01/sqlserver2008r2analysis servicesoperationsguide.aspx) ScaleOutQueryingforAnalysisServiceswithReadOnlyDatabases (http://sqlcat.com/sqlcat/b/whitepapers/archive/2010/06/08/scaleoutqueryingforanalysis serviceswithreadonlydatabases.aspx) ScaleOutQueryingwithAnalysisServices (http://sqlcat.com/sqlcat/b/whitepapers/archive/2007/12/16/scaleoutqueryingwithanalysis services.aspx) ScaleOutQueryingwithAnalysisServicesUsingSANSnapshots (http://sqlcat.com/sqlcat/b/whitepapers/archive/2007/11/19/scaleoutqueryingwithanalysis servicesusingsansnapshots.aspx)

75

2.4 Tuning Processing Performance

Inthefollowingsectionswewillprovideguidanceontuningcubeprocessing.Processingistheoperation thatloadsdatafromoneormoredatasourcesintooneormoreAnalysisServicesobjects.Although OLAPsystemsarenotgenerallyjudgedbyhowfasttheyprocessdata,processingperformanceimpacts howquicklynewdataisavailableforquerying.Everyapplicationhasdifferentdatarefresh requirements,rangingfrommonthlyupdatestonearrealtimedatarefreshes;however,inallcases,the fastertheprocessingperformance,thesooneruserscanqueryrefresheddata. AnalysisServicesprovidesseveralprocessingcommands,allowinggranularcontroloverthedataloading andrefreshfrequencyofcubes. Tomanageprocessingoperations,AnalysisServicesusescentrallycontrolledjobs.Aprocessingjobisa genericunitofworkgeneratedbyaprocessingrequest. Fromanarchitecturalperspective,ajobcanbebrokendownintoparentjobsandchildjobs.Foragiven object,youcanhavemultiplelevelsofnestedjobsdependingonwheretheobjectislocatedintheOLAP databasehierarchy.Thenumberandtypeofparentandchildjobsdependon1)theobjectthatyouare processing,suchasadimension,cube,measuregroup,orpartition,and2)theprocessingoperationthat youarerequesting,suchasProcessFull,ProcessUpdate,orProcessIndexes. Forexample,whenyouissueaProcessFulloperationforameasuregroup,aparentjobiscreatedfor themeasuregroupwithchildjobscreatedforeachpartition.Foreachpartition,aseriesofchildjobsare spawnedtocarryouttheProcessFulloperationofthefactdataandaggregations.Inaddition,Analysis Servicesimplementsdependenciesbetweenjobs.Forexample,cubejobsaredependentondimension jobs. Themostsignificantopportunitiestotuneperformanceinvolvetheprocessingjobsforthecore processingobjects:dimensionsandpartitions.Eachoftheseiscoveredinthesectionsthatfollow. References: AdditionalbackgroundinformationonprocessingcanbefoundinthetechnicalnoteAnalysis Services2005ProcessingArchitecture(http://msdn.microsoft.com/en us/library/ms345142(SQL.90).aspx).

2.4.1 Baselining Processing

Toquantifytheeffectsofyourtuninganddiagnoseproblems,youshouldfirstcreateabaseline.The baselineallowsyoutoanalyzerootcausesandtotargetoptimizationeffort. Thissectiondescribeshowtosetupthebaseline.

2.4.1.1 Performance Monitor Trace

WindowsperformancecountersarethebreadandbutterofperformancetuningAnalysisServices.Use thetoolperfmontosetupatracewiththesecounters: 76

MSOLAP:Processing o Rowsread/sec MSOLAP:ProcAggregations o TempFileBytesWrites/sec o Rowscreated/Sec o CurrentPartitions MSOLAP:Threads o Processingpoolidlethreads o Processingpooljobqueuelength o Processingpoolbusythreads MSSQL:MemoryManager o TotalServerMemory o TargetServerMemory Process o VirtualBytesmsmdsrv.exe o WorkingSetmsmdsrv.exe o PrivateBytesmsmdsrv.exe o %ProcessorTimemsmdsrv.exeandsqlservr.exe MSOLAP:Memory o QuoteBlocked LogicalDisk: o Avg.Disksec/TransferAllInstances Processor: o %ProcessorTimeTotal System: o ContextSwitches/sec

Configurethetracetosavedatatoafile.Measuringevery15secondswillbesufficientfortuning processing. Asyoutuneprocessing,youshouldmeasurethesecountersagainaftereachchangetoseewhetheryou aregettingclosertoyourperformancegoal.Alsonotethetotaltimeusedbyprocessing.Thefollowing sectionsexplainhowtouseandinterprettheindividualcounters.

2.4.1.2 Profiler Trace

TooptimizetheSQLqueriesthatformpartofprocessing,youshouldtracetherelationaldatabasetoo.If therelationaldatabaseisSQLServer,youuseSQLServerProfilerforthis.IfyouarenotusingSQL Server,consultyourdatabasevendororDBAforhelpontuningthedatabase.Inthefollowingwewill assumethatyouuseSQLServerastherelationalfoundationforAnalysisServices.Forusersofother databases,theknowledgeherewillmostlikelytransfercleanlytoyourplatform. InyourSQLServerProfilertraceyoushouldalsocapturetheevents: 77

Performance/ShowplanXMLStatisticsProfile TSQL/SQL:BatchCompleted

Includetheseeventcolumns: TextData Reads DatabaseName SPID Duration

YoucanusetheTuningtemplateandjustaddtheReadscolumnandShowplanXMLStatisticsProfiles. Liketheperfmontrace,configurethetracetosavetoafileforlateranalysis. ConfigureyourSQLServerProfilertracetologtoatableinsteadofafile.Thismakesiteasierto correlatethetraceslater. Theperformancedatagatheredbythesetraceswillbeusedinthefollowingsectiontohelpyoutune processing.

2.4.1.3 Determining Where You Spend Processing Time

Toproperlytargetthetuningofprocessing,youshouldfirstdeterminewhereyouarespendingyour time:partitionprocessingordimensionprocessing. Toassistwithtuningandfuturemonitoring,itisusefultosplitthedimensionprocessingandpartition processingintotwodifferentcommandsintheprocessing,totuneeachindividually. Forpartitionprocessing,youshoulddistinguishbetweenProcessDataandProcessIndexthetuning techniquesforeachareverydifferent.Ifyoufollowourrecommendedbestpracticeofdoing ProcessDatafollowedbyProcessIndexinsteadofProcessFull,thetimespentineachshouldbeeasyto read. IfyouuseProcessFullinsteadofsplittingintoProcessDataandProcessIndex,youcangetanideaof wheneachphaseendsbyobservingthefollowingperfmoncounters: DuringProcessDatathecounterMSOLAP:ProcessingRowsread/Secisgreaterthanzero. DuringProcessIndexthecounterMSOLAP:ProcAggregationsRowcreated/Secisgreaterthan zero.

ProcessDatacanbefurthersplitintothetimespentbytheSQLServerprocessandthetimespentbythe AnalysisServicesprocess.YoucanusetheProcesscounterscollectedtoseewheremostoftheCPU timeisspent.Thefollowingdiagramprovidesanoverviewoftheoperationsincludedinafullcube processing.

78


Figure 3030: Full cube processing overview

2.4.2 Tuning Dimension Processing

Theperformancegoalofdimensionprocessingistorefreshdimensiondatainanefficientmannerthat doesnotnegativelyimpactthequeryperformanceofdependentpartitions.Thefollowingtechniques foraccomplishingthisgoalarediscussedinthissection: Reducingattributeoverhead. OptimizingSQLsourcequeries.

Toprovideamentalmodeloftheworkload,wewillfirstintroducethedimensionprocessing architecture.

2.4.2.1 Dimension Processing Architecture

DuringtheprocessingofMOLAPdimensions,jobsareusedtoextract,index,andpersistdatainaseries ofdimensionstores. Tocreatethesedimensionstores,thestorageengineusestheseriesofjobsdisplayedinthefollowing diagram.

79

Figure 3 3131: Dime ension processing job bs

BuildAttr ributeStoresForeachat ttributeinad dimension,ajobisinstantiatedtoextractandpersistthe attributemembersintoanattribute estore.Thea attributestore econsistsoft thekeystore e,namestore, ,and hipstore.The edatastructuresbuildduringdimensionprocessingaresavedtodiskwiththe e relationsh followingextensions: Hierarchystores:*.ostore,*.sstoreand*.lstore Ke eystore:*.ks store,*.khsto oreand*.ksst tore NameStore:*.asstore,*.ah hstoreand*. .hstore Relationshipst tore:*.dataa and*.data.hd dr DecodingStore es:*.dstore es:*.mapand d*.map.hdr Bitmapindexe

therelationsh hipstorescon ntaininformationaboutde ependentattr ributes,anor rderingofthe e Becauset processingjobsisrequ uired.Toprov videthecorre ectworkflow,thestoragee engineanalyz zesthe dependen nciesbetween nattributes,a andthenitcr reatesanexe ecutiontreew withthecorre ectordering.T The execution ntreeisthenusedtodeterminethebestparallelexe ecutionofthe edimensionp processing. Figure32displaysane exampleexecutiontreefor raTimedime ension.Theso olidarrowsre epresentthe nsion.Thedas shedarrowsr representthe eimplicitrelationshipofea ach attributerelationshipsinthedimen attributetotheAllattr ribute. Note:The edimensionh hasbeenconf figuredusingcascadingatt tributerelatio onships,whic chisabest practicefo oralldimensiondesigns.

80


Figure 3 3232: Exec cution tree example

Inthisexa ample,theAllattributeproceedsfirst,giventhatith hasnodepen ndenciestoan notherattribu ute, followedbytheFiscalYearandCale endarYearat ttributes,whichcanbeprocessedinpa arallel.Theot ther sproceedacc cordingtothe edependenciesintheexec cutiontree,w withthekeya attributealwa ays attributes beingprocessedlast,b becauseitalw wayshasatleastoneattrib buterelations ship,exceptw whenitistheonly sion. attributeinthedimens Thetimet takentoproc cessanattributeisgenerallydependen nton1)thenu umberofmembersand2) )the numbero ofattributere elationships.W Whileyoucan nnotcontrolt thenumbero ofmembersf foragiven attribute,youcanimproveprocessi ingperformancebyusingcascadingatt tributerelatio onships.Thisis ycriticalforth hekeyattribu ute,becausei ithasthemo ostmembersa andallotherjobs(hierarchy, especially decoding, ,bitmapindexes)arewaitingforittocomplete.Attr ributerelatio onshipslowerthememory y requireme entduringpr rocessing.Wh henanattribu uteisprocess sed,alldepen ndentattribut tesmustbekept inmemor ry.Ifyouhave enoattribute erelationship ps,allattributesmustkept tinmemoryw whilethekey attributeisprocessed.Thismaycau useoutofme emorycondit tions. BuildDec codingStoresDecodingstoresareuse edextensively ybythestora ageengine.Duringqueryin ng, theyareu usedtoretrievedatafromthedimensio on.Duringprocessing,the eyareusedto obuildthe dimension nsbitmapind dexes. BuildHier rarchyStores sAhierarchy ystoreisape ersistentrepresentationof fthetreestru ucture.Forea ach naturalhierarchyinthedimension,ajobisinstantiatedtocre eatethehiera archystores. BuildBitm mapIndexes Toefficientl lylocateattributedataint therelationsh hipstoreatq queryingtime,the storageen nginecreates sbitmapinde exesatproces ssingtime.Fo orattributesw withaverylargenumbero of members,thebitmapindexescant takesometim metoprocess s.Inmostscenarios,thebi itmapindexes s providesi ignificantque eryingbenefit ts;however,w whenyouhav vehighcardinalityattributes,thequerying

81

benefitthatthebitmapindexprovidesmaynotoutweightheprocessingcostofcreatingthebitmap index.

2.4.2.2 Dimension-Processing Commands

Whenyouneedtoperformaprocessoperationonadimension,youissuedimensionprocessing commands.Eachprocessingcommandcreatesoneormorejobstoperformthenecessaryoperations. Fromaperformanceperspective,thefollowingdimensionprocessingcommandsarethemost important: ProcessData ProcessFull ProcessUpdate ProcessAdd

TheProcessFullandProcessDatacommandsdiscardallstoragecontentsofthedimensionandrebuild them.Behindthescenes,ProcessFullexecutesalldimensionprocessingjobsandperformsanimplicit ProcessClearonalldependentpartitions.ThismeansthatwheneveryouperformaProcessFull operationofadimension,youneedtoperformaProcessFulloperationondependentpartitionstobring thecubebackonline.ProcessFullalsobuildsindexesonthedimensiondataitself(notethatindexeson thepartitionsarebuiltseparately).IfyoudoProcessDataonadimension,youshoulddoProcessIndexes subsequentlysothatdimensionqueriesareabletousetheseindexes. UnlikeProcessFull,ProcessUpdatedoesnotdiscardthedimensionstoragecontents.Instead,itapplies updatesintelligentlyinordertopreservedependentpartitions.Morespecifically,ProcessUpdatesends SQLqueriestoreadtheentiredimensiontableandthenapplieschangestothedimensionstores. ProcessAddoptimizesProcessUpdateinscenarioswhereyouonlyneedtoinsertnewmembers. ProcessAdddoesnotdeleteorupdateexistingmembers.TheperformancebenefitofProcessAddisthat youcanuseadifferentsourcetableordatasourceviewnamedquerythatrestricttherowsofthe sourcedimensiontabletoonlyreturnthenewrows.Thiseliminatestheneedtoreadallofthesource data.Inaddition,ProcessAddalsoretainsallindexesandaggregations(flexibleandrigid). ProcessUpdateandProcessAddhavesomespecialbehaviorsthatyoushouldbeawareof.These behaviorsarediscussedinthefollowingsections. 2.4.2.2.1 ProcessUpdate AProcessUpdatecanhandleinserts,updates,anddeletions,dependingonthetypeofattribute relationships(rigidversusflexible)inthedimension.NotethatProcessUpdatedropsinvalid aggregationsandindexes,requiringyoutotakeactiontorebuildtheaggregationsinordertomaintain queryperformance.However,flexibleaggregationsaredroppedonlyifachangeisdetected. WhenProcessUpdateruns,itmustwalkthroughthepartitionsthatdependonthedimension.Foreach partition,allindexesandaggregationmustbecheckedtoseewhethertheyrequireupdating.Onacube 82

withmanypartitions,indexes,andaggregates,thiscantakeaverylongtime.Becausethisdependency walkisexpensive,ProcessUpdateisoftenthemostexpensiveofallprocessingoperationsonawell tunedsystem,dwarfingevenlargepartitionprocessingcommands. 2.4.2.2.2 ProcessAdd NotethatProcessAddisonlyavailableasanXMLAcommandandnotfromSQLServerManagement Studio.ProcessAddisthepreferredwayofmanagingType2changingdimensions.BecauseAnalysis Servicesknowsthatexistingindexesdonotneedtobecheckedforinvalidation,ProcessAddtypically runsmuchfasterthanProcessUpdate. InthedefaultconfigurationofAnalysisServices,ProcessAddtypicallytriggersaprocessingerrorwhen run,reportingduplicatekeyvalues.Thisiscausedbytheadditionofnonkeypropertiesthatalready existinthedimension.Forexample,considertheadditionofanewcustomertoadimension.Ifthe customerlivesinacountrythatisalreadypresentinthedimension,thiscountrycannotbeadded(itis alreadythere)andAnalysisServicesthrowsanerror.Thesolutioninthiscaseistosetthe <KeyDuplicate>toIgnoreErroronthedimensionprocessingcommand. NotethatyoucannotrunaProcessAddonanemptydimension.Thedimensionmustfirstbefully processed. References: FordetailedinformationaboutautomatingProcessAdd,seeGregGallowaysblogentry: http://www.artisconsulting.com/blogs/greggalloway/Lists/Posts/Post.aspx?ID=4 ForinformationabouthowtoavoidsettheKeyDuplicate,seethisforumthread: http://social.msdn.microsoft.com/Forums/enUS/sqlanalysisservices/thread/8e7f130456a1 467e9cc668428bd92aa6?prof=required

2.4.3 Tuning Cube Dimension Processing

Insection2,wedescribedhowtocreateagoodandhighperformancedimensiondesign.InSQLServer 2008andSQLServer2008R2AnalysisServices,theAnalysisManagementObjects(AMO)warningsare providedbyBusinessIntelligenceDevelopmentStudiotoassistyouwithfollowingthesebestpractices. Whenitcomestodimensionprocessing,youmustpayapriceforhavingmanyattributes.Ifthe processingtimeforthedimensionisrestrictive,youmostlikelyhavetochangetheattributetodesignin ordertoimproveperformance.

2.4.3.1 Reduce Attribute Overhead

Everyattributethatyouincludeinadimensionimpactsthecubesize,thedimensionsize,the aggregationdesign,andprocessingperformance.Wheneveryouidentifyanattributethatwillnotbe usedbyendusers,deletetheattributeentirelyfromyourdimension.Afteryouhaveremoved extraneousattributes,youcanapplyaseriesoftechniquestooptimizetheprocessingofremaining attributes. 83

2.4.3.1.1 Remove Bitmap Indexes Duringprocessingoftheprimarykeyattribute,bitmapindexesarecreatedforeveryrelatedattribute. Buildingthebitmapindexesfortheprimarykeycantaketimeifithasoneormorerelatedattributes withhighcardinality.Atquerytime,thebitmapindexesfortheseattributesarenotusefulinspeeding upretrieval,becausethestorageenginestillmustsiftthroughalargenumberofdistinctvalues.This mayhaveanegativeimpactonqueryresponsetimes. Forexample,theprimarykeyofthecustomerdimensionuniquelyidentifieseachcustomerbyaccount number;however,usersalsowanttosliceanddicedatabythecustomerssocialsecuritynumber.Each customeraccountnumberhasaonetoonerelationshipwithacustomersocialsecuritynumber.You canconsiderremovingthecreationofbitmapsforthesocialsecuritynumber. Youcanalsoconsiderremovingbitmapindexesfromattributesthatarealwaysqueriedtogetherwith otherattributesthatalreadyhavebitmapindexesthatarehighlyselective.Iftheotherattributeshave sufficientselectivity,addinganotherbitmapindextofilterthesegmentswillnotyieldagreatbenefit. Forexample,youarecreatingasalesfactandusersalwaysquerybothdateandstoredimensions. Sometimesafilterisalsoappliedbythestoreclerkdimension,butbecauseyouhavealreadyfiltered downtostores,addingabitmaponthestoreclerkmayonlyyieldatrivialbenefit.Inthiscase,youcan considerdisablingbitmapindexesonstoreclerkattributes. Youcandisablethecreationofbitmapindexesforanattributebysettingthe AttributeHierarchyOptimizedStatepropertytoNotOptimized. 2.4.3.1.2 Optimize Attribute Processing Across Multiple Data Sources Whenadimensioncomesfrommultipledatasources,usingcascadingattributerelationshipsallowsthe systemtosegmentattributesduringprocessingaccordingtodatasource.Ifanattributeskey,name, andattributerelationshipscomefromthesamedatabase,thesystemcanoptimizetheSQLqueryfor thatattributebyqueryingonlyonedatabase.Withoutcascadingattributerelationships,theSQLServer OPENROWSETfunction,whichprovidesamethodforaccessingdatafrommultiplesources,isusedto mergethedatastreams.Inthissituation,theprocessingfortheattributeisextremelyslow,becauseit mustaccessmultipleOPENROWSETderivedtables. Ifyouhavetheoption,considerperformingETLtobringalldataneededforthedimensionintothesame SQLServerdatabase.ThisallowsyoutoutilizetheRelationalEnginetotunethequery.

2.4.3.2 Tuning the Relational Dimension Processing Queries

Unlikefactpartitions,whichonlysendonequerytotheserverperpartition,dimensionprocess operationssendmultiplequeries.Dimensionstendtobesmall,complextableswithveryfewchanges, comparedtofactsthataretypicallysimplertables,butwithmanychanges.Tablesthathavethe characteristicsofdimensionscanoftenbeheavilyindexedwithlittleinsert/updateperformance overheadtothesystem.Youcanusethistoyouradvantageduringprocessingandbewastefulwiththe relationalindexes.

84

Toquickly ytunetherelationalqueriesusedfordimensionpro ocessingyouc canusetheD DatabaseEngine TuningAd dvisoronapr rofilertraceo ofthedimensionprocessin ng.Forthesm malldimension ntables,chan nces arethaty youcangetaw waywithaddingeverysug ggestedindex x.Forthelarg gertables,targettheindex xes towardst thelongestru unningquerie es.Fordetaile edtuningadv viceonlarged dimensiontab bles,seePart t2. 2.4.3.2.1 Using ByT Table Proce essing gtheProcessi ingGrouppro opertyofthedimensionto obeByTabley youwillchan ngehowAnaly ysis Bysetting Servicesb behavesdurin ngdimensionprocessing.Insteadofsen ndingmultipleSELECTDIST TINCTqueries,the processingtaskinstead drequeststheentiretable ewithonequ uery.Ifyouha aveenoughm memorytoho oldall thenewd dimensiondat tawhileproc cessingishappening,thiso optioncanprovideafastw waytooptimize processing.However,y youshouldbe ecarefulaboutthissetting gifAnalysis sServicesrun nsoutofmem mory ocessing,this swillhaveala argeimpacto onbothquery yandprocess singperforma ance.Experim ment duringpro withthiss settingcarefu ullybeforepu uttingitintop production. NotealsothatByTable eprocessingw willcausedup plicatekey(K KeyDuplicate)errorsbecau useSELECT tedforeacha attribute,and dthesameme emberswillb beencountere edrepeatedly y DISTINCTisnotexecut duringpro ocessing.The erefore,youw willneedtosp pecifyacusto omerrorconf figurationand ddisablethe KeyDuplic cateerrors.

2.4.4 T Tuning Partition Pro ocessing

Theperfo ormancegoalofpartitionp processingist torefreshfac ctdataandag ggregationsin nanefficient mannerth hatsatisfiesy youroveralld datarefreshre equirements. . Thefollow wingtechniqu uesforaccom mplishingthisgoalarediscussedinthissection:optim mizingSQLso ource queriesan ndusingapartitioningstra ategy(bothin nthecubean ndtherelationaldatabase) )tooptimize processing.Fordetaile edguidanceo onservertuning,hardware eoptimization nandrelation nalindexing,see Part2.

2.4.4.1 Partition Processin Archite ng ecture

Duringpa artitionprocessing,sourcedataisextractedandstor redondiskus singtheseriesofjobsdisp played InFigure3 33.

Figure 3 3333: Parti ition proce essing jobs s

ProcessFactDataFac ctdataisprocessedusingthreeconcur rrentthreadsthatperform mthefollowin ng tasks: 85

SendSQLstatementstoextractdatafromdatasources. Lookupdimensionkeysindimensionstoresandpopulatetheprocessingbuffer. Whentheprocessingbufferisfull,writeoutthebuffertodisk.

BuildAggregationsandBitmapIndexesAggregationsarebuiltinmemoryduringprocessing.Although toofewaggregationsmayhavelittleimpactonqueryperformance,excessiveaggregationscanincrease processingtimewithoutmuchaddedvalueonqueryperformance. Ifaggregationsdonotfitinmemory,chunksarewrittentotempfilesandmergedattheendofthe process.Bitmapindexesarealsobuiltduringthisphaseandwrittentodiskonasegmentbysegment basis.

2.4.4.2 Partition-Processing Commands

Whenyouneedtoperformaprocessoperationonapartition,youissuepartitionprocessing commands.Eachprocessingcommandcreatesoneormorejobstoperformthenecessaryoperations. Thefollowingpartitionprocessingcommandsareavailable: ProcessFull ProcessData ProcessIndexes ProcessAdd ProcessClear ProcessClearIndexes

ProcessFulldiscardsthestoragecontentsofthepartitionandthenrebuildsthem.Behindthescenes, ProcessFullexecutesProcessDataandProcessIndexesjobs. ProcessDatadiscardsthestoragecontentsoftheobjectandrebuildsonlythefactdata. ProcessIndexesrequiresapartitiontohavebuiltitsdataalready.ProcessIndexespreservesthedata createdduringProcessDataandcreatesnewaggregationsandbitmapindexesbasedonit. ProcessAddinternallycreatesatemporarypartition,processesitwiththetargetfactdata,andthen mergesitwiththeexistingpartition.NotethatProcessAddisthenameoftheXMLAcommand,in BusinessIntelligenceDevelopmentStudioandSQLServerManagementStudiothisisexposedas ProcessIncremental. ProcessClearremovesalldatafromthepartition.NotetheProcessClearisthenameoftheXMLA command.InBusinessIntelligenceDevelopmentStudioandSQLServerManagementStudio,itis exposedasUnProcess. ProcessClearIndexesremovesallindexesandaggregatesfromthepartition.Thisbringsthepartitionsin thesamestateasifProcessClearfollowedbyProcessDatahadjustbeenrun.Notethat

86

ProcessClearIndexesisthenameoftheXMLAcommand.ThiscommandisnotavailableinBusiness IntelligenceDevelopmentStudioandSQLServerManagementStudio.

2.4.4.3 Partition Processing Performance Best Practices

Whendesigningyourfacttables,usetheguidanceinthefollowingtechnicalnotes: Top10BestPracticesforBuildingaLargeScaleRelationalDataWarehouse (http://sqlcat.com/sqlcat/b/top10lists/archive/2008/02/06/top10bestpracticesforbuilding alargescalerelationaldatawarehouse.aspx) AnalysisServicesProcessingBestPractices (http://sqlcat.com/sqlcat/b/whitepapers/archive/2007/11/15/analysisservicesprocessingbest practices.aspx)

2.4.4.4 Optimizing Data Inserts, Updates, and Deletes

Thissectionprovidesguidanceonhowtoefficientlyrefreshpartitiondatatohandleinserts,updates, anddeletes. 2.4.4.4.1 Inserts Ifyouhaveabrowsable,processedcubeandyouneedtoaddnewdatatoanexistingmeasuregroup partition,youcanapplyoneofthefollowingtechniques: ProcessFullPerformaProcessFulloperationfortheexistingpartition.DuringtheProcessFull operation,thecuberemainsavailableforbrowsingwiththeexistingdatawhileaseparatesetof datafilesarecreatedtocontainthenewdata.Whentheprocessingiscomplete,thenew partitiondataisavailableforbrowsing.NotethatProcessFullistechnicallynotnecessary,given thatyouareonlydoinginserts.Tooptimizeprocessingforinsertoperations,youcanuse ProcessAdd. ProcessAddUsethisoperationtoappenddatatotheexistingpartitionfiles.Ifyoufrequently performProcessAdd,werecommendthatyouperiodicallyperformProcessFullinorderto rebuildandrecompressthepartitiondatafiles.ProcessAddinternallycreatesatemporary partitionandmergesit.Thisresultsindatafragmentationovertimeandtheneedtoperiodically performProcessFull.

Ifyourmeasuregroupcontainsmultiplepartitions,asdescribedintheprevioussection,amoreeffective approachistocreateanewpartitionthatcontainsthenewdataandthenperformProcessFullonthat partition.Thistechniqueallowsyoutoaddnewdatawithoutimpactingtheexistingpartitions.When thenewpartitionhascompletedprocessing,itisavailableforquerying. 2.4.4.4.2 Updates Whenyouneedtoperformdataupdates,youcanperformaProcessFull.Ofcourseitisusefulifyoucan targettheupdatestoaspecificpartitionsoyouonlyhavetoprocessasinglepartition.Ratherthan directlyupdatingfactdata,abetterpracticeistouseajournalingmechanismtoimplementdata changes.Inthisscenario,youturnanupdateintoaninsertionthatcorrectsthatexistingdata.Withthis 87

approach,youcansimplycontinuetoaddnewdatatothepartitionbyusingaProcessAdd.Byusing journaling,youalsohaveanaudittrailofthechangesthathavebeenmadetothefacttable. 2.4.4.4.3 Deletes Fordeletions,multiplepartitionsprovideagreatmechanismforyoutorolloutexpireddata.Consider thefollowingexample.Youcurrentlyhave13monthsofdatainameasuregroup,1monthperpartition. Youwanttorollouttheoldestmonthfromthecube.Todothis,youcansimplydeletethepartition withoutaffectinganyoftheotherpartitions. Ifthereareanyolddimensionmembersthatonlyappearedintheexpiredmonth,youcanremovethese usingaProcessUpdateoperationonthedimension(butonlyifitcontainsflexiblerelationships).In ordertodeletemembersfromthekey/granularityattributeofadimension,youmustsetthe dimensionsUnknownMemberpropertytoHidden.Thisisbecausetheserverdoesnotknowifthereis afactrecordassignedtothedeletedmember.Withthispropertysetappropriately,thememberwillbe hiddenatquerytime.Anotheroptionistoremovethedatafromtheunderlyingtableandperforma ProcessFulloperation.However,thismaytakelongerthanProcessUpdate. Asyourdimensiongrowslarger,youmaywanttoperformaProcessFulloperationonthedimensionto completelyremovedeletedkeys.However,ifyoudothis,allrelatedpartitionsmustalsobe reprocessed.Thismayrequirealargebatchwindowandisnotviableforallscenarios.

2.4.4.5 Picking Efficient Data Types in Fact Tables

Duringprocessing,datahastobemovedoutofSQLServerandintoAnalysisServices.Thewideryour rowsare,themorebandwidthmustbespentmovingtherows. Somedatatypesare,bythenatureoftheirdesign,fastertousethanothers.Forfastestperformance, considerusingonlythesedatatypesinfacttables. Factcolumntype Surrogatekeys Datekey Integermeasures Numericmeasures FastestSQLServerdatatypes tinyint,smallint,int,bigint intintheformatyyyyMMdd tinyint,smallint,int,bigint smallmoney,money,real,float (Notethatdecimalandvardecimalrequiremore CPUpowertoprocessthanmoneyandfloattypes) tinyint,smallint,int,bigint (Ifyourcountcolumnischar,considereither hashingorreplacingwithsurrogatekey)

Distinctcountcolumns

2.4.4.6 Tuning the Relational Partition Processing Query

DuringtheProcessDataphase,rowsarereadfromarelationalsourceandintoAnalysisServices. AnalysisServicescanconsumerowsataveryhighrateduringthisphase.Toachievethesehighspeeds, youneedtotunetherelationaldatabasetoprovideaproperthroughput. 88

Inthefollowingsubsection,weassumethatyourrelationalsourceisSQLServer.Ifyouareusing anotherrelationalsource,someoftheadvicestillappliesconsultyourdatabasespecialistforplatform specificguidance. AnalysisServicesusesthepartitioninformationtogeneratethequery.Unlessyouhavedoneanyquery bindingintheUDM,theSELECTstatementissuestotherelationalsourceisverysimple.Itconsistsof: ASELECTofthecolumnsrequiredtoprocess.Thiswillbethedimensioncolumnsandthe measures. Optionally,aWHEREcriterionifyouusepartitions.YoucancontrolthisWHEREcriterionby changingthequerybindingofthepartition.

2.4.4.6.1 Getting Rid of Joins IfyouareusingadatabasevieworaUDMnamedqueryasthebasisofpartitions,youshouldseekto eliminatejoinsinthequerysendtothedatabase.Youcanachievethisbydenormalizingthejoined columnstothefacttable.Ifyouareusingastarschemadesign,youshouldalreadyhavedonethis. References: Forbackgroundonrelationalstarschemasandhowtodesignanddenormalizeforoptimal performance,referto:RalphKimball,TheDataWarehouseToolkit.

2.4.4.6.2 Getting Relational Partitioning Right Ifyouusepartitioningontherelationalside,youshouldensurethateachcubepartitiontouchesatmost onerelationalpartition.Tocheckthis,usetheXMLShowplaneventfromyourSQLServerProfilertrace. Ifyougotridofalljoins,yourqueryplanshouldlooksomethinglikethefollowingfigure.

Figure 3434: An optimal partition processing query

Clickonthetablescan(itmayalsobearangescanorindexseekinyourcase)andbringupthe propertiespane.

89


Figure 3535: Too many partitions accessed

Bothpartition4andpartition5areaccessed.ThevalueforActualPartitionCountshouldbe1.Ifthisis notthecase(asinthefigure),youshouldconsiderrepartitioningtherelationalsourcedatasothateach cubepartitiontouchesatmostonerelationalpartition.

2.4.4.7 Splitting Processing Index and Process Data

Itisgoodpracticetosplitpartitionprocessingintoitscomponents:ProcessDataandProcessIndex.This hasseveraladvantages. First,itallowsyoutorestartafailedprocessingatthelastvalidstate.Forexample,ifyoufailprocessing duringProcessIndex,youcanrestartthisphaseinsteadofrevertingtorunningProcessDataagain. Second,ProcessDataandProcessIndexhavedifferentperformancecharacteristics.Typically,youwant tohavemoreparallelcommandsexecutingduringProcessDatathanyouwantduringProcessIndex.By splittingthemintotwodifferentcommands,youcanoverrideparallelismontheindividualcommands. Ofcourse,ifyoudontwanttomicromanagepartitionprocessing,youmayjustoptforrunninga ProcessFullonthemeasuregroup.Forsmallcubeswhereperformanceisnotaconcern,thiswillwork well.

2.4.4.8 Increasing Concurrency by Adding More Partitions

IfyourtuningisboundonlybytheamountofCPUpoweryouhave(asopposedtoI/O,forexample),you shouldoptimizetomakethebestuseoftheCPUcoresavailabletoyou.Itistimetohavealookatthe Processor:Totalcounterfromthebaselinetrace.Ifthiscounterisnot100%,youarenottakingfull advantageofyourCPUpower.Asyoucontinuethetuning,keepcomparingthebaselinestomeasure improvement,andwatchoutforbottleneckstoappearagainasyoupushmoredatathroughthe system. Usingmultiplepartitionscanenhanceprocessingperformance.Partitionsallowyoutoworkonmany, smallerpartsofthefacttableinparallel.BecauseasingleconnectiontoSQLServercanonlytransfera limitedamountofrowspersecond,addingmorepartitions,andhence,moreconnections,canincrease throughput.HowmanypartitionsyoucanprocessinparalleldependsonyourCPUandmachine architecture.Asaruleofthumb,keepincreasingparallelismuntilyounolongerseeanincreasein MSOLAP:ProcessingRowsread/Sec.Youcanmeasuretheamountofconcurrentpartitionsyouare processingbylookingattheperfmoncounterMSOLAP:ProcAggregationsCurrentPartitions. 90

Beingabletoprocessmultiplepartitionsinparallelisusefulinavarietyofscenarios;however,thereare afewguidelinesthatyoumustfollow.Keepinmindthatwheneveryouprocessameasuregroupthat hasnoprocessedpartitions,AnalysisServicesmustinitializethecubestructureforthatmeasuregroup. Todothis,ittakesanexclusivelockthatpreventsparallelprocessingofpartitions.Youshouldeliminate thislockbeforeyoustartthefullparallelprocessonthesystem.Toremovetheinitializationlock,ensure thatyouhaveatleastoneprocessedpartitionpermeasuregroupbeforeyoubegintheparallel operation.Ifyoudonothaveaprocessedpartition,youcanperformaProcessStructureonthecubeto builditsinitialstructureandthenproceedtoprocessmeasuregrouppartitionsinparallel.Youwillnot encounterthislimitationifyouprocesspartitionsinthesameclientsessionandusetheMaxParallel XMLAelementtocontrolthelevelofparallelism.

2.4.4.9 Adjusting Maximum Number of Connections

Whenyouincreaseparallelismoftheprocessingabove10concurrentpartitions,youwillneedtoadjust themaximumnumberofconnectionsthatAnalysisServiceskeepsopenonthedatabase.Thisnumber canbechangedinthepropertiesofthedatasource(theMaximumnumberofconnectionsbox).

Figure 3636: Adding more database connections

Setthisnumbertoatleastthenumberofpartitionsyouwanttoprocessinparallel.

2.4.4.10

Tuning the Process Index Phase

DuringtheProcessIndexphasetheaggregationsinthecubearebuilt.Atthispoint,nomoreactivity happensintheRelationalEngine,andifAnalysisServicesandtheRelationalEnginearesharingthesame box,youcandedicateallyourCPUcorestoAnalysisServices.

91

ThekeyfigureyouoptimizeduringProcessIndexistheperformancecounterMSOLAP:Proc AggregationsRowcreated/Sec.Asthecounterincreases,theProcessIndextimedecreases.Youcan usethiscountertocheckifyourtuningeffortsimprovethespeed. Anadditionalimportantcountertolookatisthetemporaryfilescounterwhenanaggregationdoesnt fitinmemory,theaggregationdataisspilledtotemporarydiskfiles.Buildingdiskbasedaggregationsis muchmoreexpensive,andifyounoticethisyoumaybeabletofindawaytoeitherallowmorememory tobeavailablefortheindexbuildingphase,ordropsomeofthelargeraggregationstoavoidthisissue. 2.4.4.10.1 Add Partitions to Increase Parallelism AswasthecasewithProcessData,processingmorepartitionsinparallelcanspeedupProcessIndex. Thesametuningstrategyapplies:Keepincreasingpartitioncountuntilyounolongerseeanincreasein processingspeed.

2.4.4.11

Partitioning the Relational Source

Thebestpartitionstrategytoimplementintherelationalsourcevariesbydatabaseproductcapabilities, butsomegeneralguidanceapplies. Itisoftenagoodideatoreflectthecubepartitionstrategyintherelationdesign.Partitionsinthe relationalsourceserveascoarseindexes,andmatchingrelationalpartitionswiththecubeallowsyou togetthebestpossibletablescanspeedsbytouchingonlytherecordsyouneed.Anotherwayto achievethateffectistouseaSQLServerclusteredindex(ortheequivalentinyourpreferreddatabase engine)tosupportfastscanqueriesduringpartitionprocessing.Ifyouhaveusedamatrixpartition schemaasdescribedearlier,youmayevenwanttocombinethepartitionandclusterindexstrategy, usingpartitioningtosupportoneofthepartitioneddimensionandclusterindexestosupporttheother. Thefollowingfiguresillustratesomeexamplesofpartitionstrategiesyoushouldconsider.

Figure 3737: Matching partition strategies

92


Figure 3838: Clustering the relational table

Figure 3939: Supporting matrix partitioning with a combination of relational layouts

2.5 Special Considerations

TherearecertainfeaturesofAnalysisServicesthatprovidealotofbusinessintelligencevalue,butthat requirespecialattentiontosucceed.Thissectiondescribesthesescenariosandthetuningyoucanapply whenyouencounterthem.

2.5.1 Distinct Count

DistinctcountmeasuresarearchitecturallyverydifferentfromotherAnalysisServicesmeasures becausetheyarenotadditiveinnature.Thismeansthatmoredatamustbekeptondiskandingeneral, mostdistinctcountquerieshaveaheavyimpactonthestorageengine.

2.5.1.1 Partition Design

Whendistinctcountpartitionsarequeried,eachpartitionssegmentjobsmustcoordinatewithone anothertoavoidcountingduplicates.Forexample,ifcountingdistinctcustomerswithcustomerIDand thesamecustomerIDisinmultiplepartitions,thepartitionsjobsmustrecognizethematchsothatthey donotcountthesamecustomermorethanonce. Ifeachpartitioncontainsnonoverlappingrangeofvalues,thiscoordinationbetweenjobsisavoidedand queryperformancecanimprovebyordersofmagnitude,dependingonhardware!Aswell,therearea numberofadditionaloptimizationstohelpimprovedistinctcountperformance: 93

Thekeytoimprovingdistinctcountqueryperformanceistohaveapartitioningstrategythat involvesatimeperiodandyourdistinctcountvalue.Startbypartitioningbytimeandxnumber ofdistinctvaluepartitionsofequalsizewithnonoverlappingranges,wherexisthenumberof cores.Refinexbytestingwithdifferentpartitioningschemes. Todistributeyourdistinctvalueacrossyourpartitionswithnonoverlappingranges,considering buildingahashofthedistinctvalue.Amodulofunctionissimpleandstraightforwardbutit requiresextraprocessing(forexample,convertcharacterkeytointegervalues)andstorage(for example,tomaintainanIDENTITYtable).AhashfunctionsuchastheSQLHashBytesfunction willavoidthelatterissuesbutmayintroducehashkeycollisions(thatis,whenthehashvalueis repeatedbasedondifferentsourcevalues). Thedistinctcountmeasuremustbedirectlycontainedinthequery.Ifyoupartitionyourcube bythehashofthedistinctvalue,itisimportantthatyourqueryisagainstthehashofthedistinct value(versusthedistinctvalueitself).Evenifthedistinctvalueandthehashofthedistinct valuehavethesamedistributionofdata,andevenifyoupartitiondatabythelatter,theheader filescontainonlytherangeofvaluesassociatedwiththehashofthedistinctvalue.This ultimatelymeansthattheAnalysisServicesstorageenginemustqueryallofthepartitionsto performthedistinctonthedistinctvalue. Thedistinctcountvaluesneedtobecontinuous.

Formoreinformation,seeAnalysisServicesDistinctCountOptimizationUsingSolidStateDevices (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2010/09/20/analysisservicesdistinctcount optimizationusingsolidstatedevices.aspx).

2.5.1.2 Processing of Distinct Count

Distinctcountmeasuregroupshavespecialrequirementsforpartitioning.Normally,youusetimeand potentiallysomeotherdimensionasthepartitioningcolumn(seethesectiononPartitioningearlierin thisbook).However,ifyoupartitionadistinctcountmeasuregroup,youshouldpartitiononthevalue ofthedistinctcountmeasurecolumninsteadofadimension. Groupthedistinctcountmeasurecolumnintoseparate,nonoverlappingintervals.Eachintervalshould containapproximatelythesameamountofrowsfromthesource.Theseintervalsthenformthesource ofyourAnalysisServicespartitions. BecausetheparallelismoftheProcessDataphaseislimitedbytheamountofpartitionsyouhave,for optimalprocessingperformance,splitthedistinctcountmeasureintoasmanyequalsized nonoverlappingintervalsasyouhaveCPUcoresontheAnalysisServicescomputer. StartingwithSQLServer2005AnalysisServices,itispossibletousenonintegercolumnsfordistinct countmeasuregroups.However,forperformancereasons(andthepotentialtohitthe4GBlimit)you shouldavoidthis.ThewhitepaperAnalysisServicesDistinctCountOptimization (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2010/09/20/analysisservicesdistinctcount optimizationusingsolidstatedevices.aspx)describeshowyoucanusehashfunctionstotransform

94

nonintegercolumnsintointegersfordistinctcount.Italsoprovidesexamplesofthenonoverlapping intervalpartitioningstrategy. YoushouldalsoinvestigatethepossibilityofoptimizingtherelationaldatabasefortheparticularSQL queriesthataregeneratedduringprocessingofdistinctcountpartitions.Theprocessingquerywillsend anORDERBYclauseintheSQL,andtheremaybetechniquesthatyoucanfollowtobuildindexesinthe relationaldatabasethatwillproducebetterperformanceforthisquerypattern.

2.5.1.3 Distinct Count Partition Aggregation Considerations

Aggregationscreatedfordistinctcountpartitionsaredifferentbecausedistinctcountvaluescannotbe naturallyaggregatedatdifferentlevels.AnalysisServicescreatesaggregationsatthedifferent granularitiesbyalsoincludingthevaluethatneedstobecounted.Ifyouthinkofanaggregationasa GROUPBYontheaggregationgranularities,adistinctcountaggregationisaGROUPBYonthe aggregationgranularitiesandthedistinctcountcolumn.Havingthedistinctcountcolumninthe aggregationdataallowstheaggregationtobeusedwhenqueryingahighergranularitybut unfortunatelyitalsomakestheaggregationsmuchlarger. Togetthemostvalueoutofaggregationsfordistinctcountpartitions,designaggregationsata commonlyviewedhigherlevelattributerelatedtothedistinctcountattribute.Forexample,areport aboutcustomersistypicallyviewedattheCustomerGrouplevel;hence,buildaggregationsatthatlevel. Acommonapproachisrunthetypicalqueriesagainstyourdistinctcountpartitionanduseusagebased optimizationtobuildtheappropriateaggregations.

2.5.1.4 Optimize the Disk Subsystem for Random I/O

Asnotedinthebeginningofthissection,distinctcountquerieshaveaheavyimpactontheAnalysis Servicesstorageengine,whichforlargecubesmeansthereisalargeimpactonthedisksubsystem.For eachquery,AnalysisServicesgeneratespotentiallymultipleprocesseseachoneparsingthedisk subsystemtoperformaportionofthedistinctcountcalculation.ThisresultsinheavyrandomI/Oonthe disksubsystem,whichcansignificantlyreducethequeryperformanceofyourdistinctcounts(andallof yourAnalysisServicesqueriesoverall). ThediskoptimizationtechniquesdescribedinPart2areespeciallyimportantfordistinctcountmeasure groups. References: SQLServer2008R2AnalysisServicesOperationsGuide (http://sqlcat.com/sqlcat/b/whitepapers/archive/2011/06/01/sqlserver2008r2analysis servicesoperationsguide.aspx) AnalysisServicesDistinctCountOptimization (http://sqlcat.com/sqlcat/b/whitepapers/archive/2008/04/17/analysisservicesdistinctcount optimization.aspx)

95

AnalysisServicesDistinctCountOptimizationUsingSolidStateDevices (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2010/09/20/analysisservicesdistinct countoptimizationusingsolidstatedevices.aspx) SQLBIManytoManyProject (http://www.sqlbi.com/Projects/Manytomanydimensionalmodeling/tabid/80/Default.aspx)

2.5.2 Large Many-to-Many Dimensions

Manytomanyrelationshipsareusedheavilyinmanybusinessscenariosrangingfromsalesto accountingtohealthcare.Butattimestheremaybequeryperformanceissueswhendealingwithalarge numberofmanytomanyrelationshipsandperceivedaccuracyissues.Onewaytothinkaboutamany tomanydimensionisthatitisageneralizationofthedistinctcountmeasure.Theuseofmanytomany dimensionsenablesyoutoapplydistinctcountlogictootherAnalysisServicesmeasuressuchassum, count,max,min,andsoon.Tocalculatethesedistinctcountoraggregates,theAnalysisServicesstorage enginemustparsethroughthelowestlevelofgranularityofdata.Thisisbecausewhenaqueryincludes amanytomanydimension,thequerycalculationisperformedatquerytimebetweenthemeasure groupandintermediatemeasuregroupattheattributelevel.Theresultisaprocessorandmemory intensiveprocesstoreturntheresult. Performanceandaccuracyissuesconcerningmanytomanydimensionsincludethefollowing: Thejoinbetweenthemeasuregroupandintermediatemeasuregroupisahashjoinstrategy; henceitisverymemoryintensivetoperformthisoperation. Becausequeriesinvolvingmanytomanydimensionsresultinajoinbetweenthemeasure groupandanintermediatemeasuregroup,reducingthesizeofyourintermediatemeasure group(ageneralruleislessthan1millionrows)providesthebestperformance.Foradditional techniques,seetheAnalysisServicesManytoManyDimensions:QueryPerformance OptimizationTechniqueswhitepaper (http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=137). Manytomanyrelationshipscannotbeaggregated(althoughitgenerallyisnotveryeasyto creategeneralpurposeaggregatesfordistinctcountsaswell).Therefore,queriesinvolving manytomanydimensionscannotuseaggregationsoraggregatecachesonlyadirecthitwill work.Therearespecificsituationswheremanytomanyrelationshipscanbeaggregated;you canfindmoreinformationintheAnalysisServicesManytoManyDimensions:Query PerformanceOptimizationTechniqueswhitepaper. o Becausemanytomanycannotbeaggregated,therearevariousMDXcalculationissues withVisualTotals,subselects,andCREATESUBCUBE. Similartodistinctcount,theremaybeperceiveddoublecountingissuesbecauseitisdifficultto identifywhichmembersofthedimensionareinvolvedwiththemanytomanyrelationship.

Tohelpimprovetheperformanceofmanytomanydimensions,onecanmakeuseoftheManyto Manymatrixcompression,whichremovesrepeatedmanytomanyrelationshipsthusreducingthesize ofyourintermediatemeasuregroup.Ascanbeseeninthefollowingfigure,aMatrixKeyiscreated 96

basedoneachsetofcommondimensionmembercombinationssothatrepeatedcombinationsare eliminated.

Figure 4040: Compressing the FactInternetSalesReason intermediate fact table (from Analysis Services Many-to-Many Dimensions: Query Performance Optimization Techniques)

References: ManytoManyMatrixCompression(http://bidshelper.codeplex.com/wikipage?title=Manyto Many%20Matrix%20Compression) SQLBIManytoManyProject (http://www.sqlbi.com/Projects/Manytomanydimensionalmodeling/tabid/80/Default.aspx) AnalysisServices:Shouldyouusemanytomanydimensions? (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2008/02/11/analysisservicesshouldyou usemanytomanydimensions.aspx)

97

2.5.3 Parent-Child Dimensions

Theparentchilddimensionisacompactandpowerfulwaytorepresenthierarchiesinarelational databaseespeciallyraggedandunbalancedhierarchies.YetwithinAnalysisServices,thequery performancetendstobesuboptimal,especiallyforlargeparentchilddimensions,becauseaggregations arecreatedonlyforthekeyattributeandthetopattribute(thatis,theAllattribute)unlessitisdisabled. Therefore,acommonbestpracticeistorefrainfromusingparentchildhierarchiesthatcontainalarge numberofmembers.(Howbigislarge?Thereisntaspecificnumberbecausequeryperformanceat intermediatelevelsoftheparentchildhierarchydegradeslinearlywiththenumberofmembers.) Additionally,limitthenumberofparentchildhierarchiesinyourcube. Ifyouareinadesignscenariowithalargeparentchildhierarchy,consideralteringthesourceschemato reorganizepartorallofthehierarchyintoaregularhierarchywithafixednumberoflevels.For example,sayyouhaveaparentchildhierarchysuchastheoneshownhere.

Figure 4141: Sample parent-child hierarchy

Thedatafromthisparentchildhierarchyisrepresentedinrelationalformatasperthefollowingtable. SK 1 2 3 4 5 Parent_SK NULL 1 2 2 1

Convertingthistabletoaregularlyhierarchyresultsinarelationaltablewiththefollowingformat. SK 1 2 3 4 5 98 Level0_SK 1 1 1 1 1 Level1_SK NULL 2 2 2 5 Level2_SK NULL NULL 3 4 NULL

 Afterthedatahasbeenreorganizedintotheuserhierarchy,youcanusetheHideMemberIfproperty ofeachleveltohidetheredundantormissingmembers.Tohelpconvertyourparentchildhierarchy intoaregularhierarchy,refertotheAnalysisServicesParentChildDimensionNaturalizertoolin CodePlex (http://pcdimnaturalize.codeplex.com/wikipage?title=Home&version=12&ProjectName=pcdimnaturaliz e). References: AnalysisServicesParentChildDimensionNaturalizer (http://pcdimnaturalize.codeplex.com/wikipage?title=Home&version=12&ProjectName=pcdimn aturalize) IncludingChildMembersMultiplePlacesinaParentChildHierarchy (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2008/03/17/includingchildmembers multipleplacesinaparentchildhierarchy.aspx)

2.5.4 Near Real Time and ROLAP

AsyourAnalysisServicesdatabecomesmorevaluabletothebusiness,acommonnextrequirementisto providenearrealtimecapabilitiessouserscanhaveimmediateaccesstotheirbusinessintelligence system.Nearrealtimedatahasspecialrequirements: Typicallythedatamustresideinmemoryforlowlatencyaccess. Often,youdonothavetimetomaintainindexesonthedata. Youwilltypicallyrunintolockingand/orconcurrencyissuesthatmustbedealtwith.

ItisimportanttonotethatduetothelockinglogicinvokedbyAnalysisServices,longrunningqueriesin AnalysisServicescanbothpreventprocessingfromcommittingandblockotherqueries. ToprovidenearrealtimeresultsandavoidtheAnalysisServicesquerylocking,startwithusingROLAP sothatthequeriesgodirectlytotherelationaldatabase.Yetevenrelationaldatabaseshavelocking and/orconcurrencyissuesthatneedtobedealtwith.Tominimizetheimpactofblockingquerieswithin yourrelationaldatabase,placetherealtimeportionofthedataintoitsownseparatetablebutkeep historicaldatawithinyourpartitionedtable.Afteryouhavedonethis,youcanapplyothertechniques.In thissectionwediscussthefollowing: MOLAPswitching ROLAP+MOLAP ROLAPpartitioning

99

2.5.4.1 MOLAP Switching

ThebasicprinciplebehindMOLAPswitchingistocreatesomepartitionsforhistoricaldataandanother setofpartitionsforthelatestdata.Thelatenciesassociatedwithfrequentlyprocessingthecurrent MOLAPpartitionsareinminutes.Thismethodologyiswellsuitedforsomethinglikeatimezone scenarioinwhichyouhaveactivepartitionsthroughouttheday.Forexample,sayyouhaveactive partitionsfordifferentregionssuchasNewYork,London,Mumbai,andTokyo.Inthisscenario,you wouldcreatepartitionsbybothtimeandthespecificregion.Thisprovidesyouwiththefollowing benefits: Youcanfullyprocess(asoftenasneeded)theactiveregion/timepartition(forexample,Tokyo /Day1)withoutinterferingwithotherpartitions(forexample,NewYork/Day1). YoucanrollwiththedaylightandprocessNewYork,London,Mumbai,andTokyowith minimaloverlap.

However,longrunningqueriesforaregioncanblocktheprocessingforthatregion.Aprocessing commitofcurrentNewYorkdatamightbeblockedbyanexistinglongrunningqueryforNewYorkdata. Toalleviatethisproblem,usetwocopiesofthesamecube,alternatingdataprocessingbetweenthem (knownascubeflipping).

Figure 4242: Cube-flipping concept

Whileonecubeprocessesdata,theothercubeisavailableforquerying.Toflipbetweenthecubes,you canusetheAnalysisServicesLoadBalancingToolkit (http://sqlcat.com/sqlcat/b/toolbox/archive/2010/02/08/aslb.aspx)orcreateyourowncustomplugin toyourUI(youcanuseExceltodothis,forexample)thatcandetectwhichcubeitshouldqueryagainst. Itwillbeimportantfortheplugintoholdsessionstatesothatuserqueriesusethequerycache.Session stateshouldautomaticallyrefreshwhentheconnectionstringischanged.

100

2.5.4.2 ROLAP + MOLAP

ThebasicprinciplebehindROLAP+MOLAPistocreatetwosetsofpartitions:aROLAPpartitionfor frequentlyupdatedcurrentdataandMOLAPpartitionsforhistoricaldata.Inthisscenario,youtypically canachievelatenciesintermsofseconds.Ifyouusethistechnique,besuretofollowtheseguidelines: MaintainacoherentROLAPcache.Forexample,ifyouquerytherelationaldata,theresultsare placedintothestorageenginecache.Bydefault,thenextqueryusesthatstorageenginecache entry,butthecacheentrymaynotreflectanynewchangestotheunderlyingrelational database.Itisevenpossibletohaveaggregatevaluesstoredinthedatacachethatwhen aggregatedupdonotaddupcorrectlytotheparent. UseRealTimeOLAP=truewithintheconnectionstring. AssumethattheMOLAPpartitionsarewriteonce/readsometimes.Ifyouneedtomake changestotheMOLAPpartitions,ensurethechangesdonothaveanimpactonusersquerying thesystem. FortheROLAPpartition,ensurethattheunderlyingSQLdatasourcecanhandleconcurrent queriesandload.ApotentialsolutionistouseReadCommittedSnapshotIsolation(RSCI);for moreinformation,seeBulkLoadingDataintoaTablewithConcurrentQueries (http://sqlcat.com/sqlcat/b/technicalnotes/archive/2009/04/06/bulkloadingdataintoatable withconcurrentqueries.aspx).

2.5.4.3 Comparing MOLAP Switching and ROLAP + MOLAP

ThefollowingtablecomparestheMOLAPswitchingandROLAP+MOLAPmethodologies. Component RelationalTuning ASlocking CacheUsage RelationalConcurrency DataStorage AggregationManagement Latency MOLAPSwitching Low Needtohandle Good N/A BestCompression SQLServerProfiler+UBO Minutes ROLAP+MOLAP Mustgetright Minimal Poor RSCI ROLAPsizestypically2xMOLAP Manual Seconds

2.5.4.4 ROLAP
Ingeneral,MOLAPisthepreferredstoragechoiceforAnalysisServices;becauseMOLAPtypically providesfasteraccesstothedata(especiallyifyourdisksubsystemisoptimizedforrandomI/O),itcan handleattributesmoreefficientlyanditiseasiertomanage.However,ROLAPagainstSQLServercanbe asolidchoiceforverylargecubeswithexcellentperformanceandthebenefitofreducingoreven removingtheprocessingtimeoflargecubes.Asnotedearlier,itisoftenarequirementifyouneedto havenearrealtimecubes.Ascanbeseeninthefollowingfigure,thequeryperformanceofaROLAP cubeafterusagebasedoptimizationisappliedcanbecomparabletoMOLAPifthesystemisexpertly tuned. 101


Figure 4343: Showcasing ROLAP vs. MOLAP performance before and after the application of usage-based optimization

2.5.4.4.1 ROLAP Design Recommendations TherecommendationsforhighperformancequeryingofROLAPcubesarelistedhere: SimplifythedatastructureofyourunderlyingSQLdatasourcetominimizepagereads(for example,removeunusedcolumns,trytouseINTcolumns,andsoon). Useastarschemawithoutsnowflaking,becausejoinscanbeexpensive. Avoidscenariossuchasmanytomanydimensions,parentchilddimensions,distinctcount,and ROLAPdimensions.

2.5.4.4.2 ROLAP Aggregation Design Recommendations WhenworkingwithROLAPpartitions,youcancreateaggregationsintwoways: CreatecubebasedaggregationsbyusingtheAnalysisServicesaggregationstools. CreateyourowntransparentaggregationsdirectlyagainsttheSQLServerdatabase.

BothapproachesrelyonthecreationofindexedviewswithinSQLServerbutofferdifferentadvantages anddisadvantages.Oftenthemosteffectivestrategyisacombinationofthesetwoapproachesasnoted inthefollowingtable. Aggregation Type Cubebased Advantages Efficientqueryprocessing:Analysis Servicescanusecubebased aggregationsevenifthequeryand Disadvantages Processingoverhead:AnalysisServices dropsandrecreatesindexedviews associatedwithcubebasedaggregations

102

Transparent

aggregationgranularitiesdonotexactly match.Forexample,aqueryon[Month] canuseanaggregationon[Day],which requiresonlythesummarizationofup to31numbers. Aggregationdesignefficiency:Analysis ServicesincludestheAggregationDesign WizardandtheUsageBased OptimizationWizardtocreate aggregationdesignsbasedonstorage andpercentageconstraintsorqueries submittedbyclientapplications. Reuseofexistingindexesacrosscubes: Whileaggregateviewscanalsobe createdbyqueriesthatdonotknowof theirexistence,theissueisthatAnalysis Servicesmayunexpectedlydropthe indexedviews Lessoverheadduringcubeprocessing: AnalysisServicesisunawareofthe aggregationsanddoesnotdropthe indexedviewsduringpartition processing.Thereisnoneedtodrop indexedviewsbecausetherelational enginemaintainstheindexes continuously,suchasduringINSERT, UPDATE,andDELETEoperationsagainst thebasetables.

duringcubepartitionprocessing. Droppingandrecreatingtheindexes cantakeanexcessiveamountoftimein alargescaledatawarehouse.

Nosophisticatedaggregation algorithms:Indexedviewsmustmatch querygranularity.Thequeryoptimizer doesntconsiderdimensionhierarchies oraggregationgranularitiesinthequery executionplan.Forexample,anSQL querywithGROUPBYon[Month]cant useanindexon[Day]. Maintenanceoverhead:Database administratorsmustmaintain aggregationsbyusingSQLServer ManagementStudioorothertools.Itis difficulttokeeptrackofthe relationshipsbetweenindexedviews andROLAPcubes. Designcomplexity:DatabaseEngine TuningAdvisorcanhelptofacilitate aggregationdesigntasksbyanalyzing SQLServerProfilertraces,butitcant identifyallpossiblecandidates. Moreover,datawarehouse(DW) architectsmustmanuallystudySQL ServerProfilertracestodetermine effectiveaggregations.

Herearesomegeneralrules: Transparentaggregationshavegreatervalueinanenvironmentwheremultiplecubesare referencingthesamefacttable.

103

Transparentaggregationsandcubebasedaggregationscouldbeusedtogethertogetthemost efficientdesign: o Startwithasetoftransparentaggregationsthatwillworkforthemostcommonlyrun queries. o Addcubebasedaggregationsusingusagebasedoptimizationforimportantqueriesthat aretakingalongtimetorun.

2.5.4.4.3 Limitations of ROLAP Aggregations WhileROLAPisverypowerful,therearesomestrictlimitationsthatmustbefirstconsideredbefore usingthisapproach: Youmayhavetodesignusingtablebinding(andnotquerybinding)toanactualtableinsteadof apartition.Thegoalofthisguidanceistoensurepartitionelimination. o ThisadviceisspecifictoSQLServerasadatasource.Forotherdatasources,carefully evaluatethebehaviorofROLAPquerieswhenaccessingapartitionedtable. o Itisnotpossibletocreateanindexedviewonaviewcontainingasubselectstatement. ThiswillpreventAnalysisServicesfromcreatingindexviewaggregations. Relationalpartitioneliminationwillgenerallynotwork: o Normally,DWbestpracticeistousepartitionedfacttables. o IfyouneedtouseROLAPaggregations,youmustuseseparatetablesintherelational databaseforeachcubepartition o Partitionsrequirenamedqueries,andthosegeneratebadSQLplans.Thismayvary dependingontherelationalengineyouuse. Youcannotuse: o AnamedqueryoraviewintheDSV. o AnyfeaturethatwillcauseAnalysisServicestogenerateasubquery.Forexample,you cannotuseaCountofRowsmeasure,becauseasubqueryisalwaysgeneratedwhen thistypeofmeasureisused. Themeasuregroupcannothave: o AnymeasuresthatuseMaxorMinaggregation. o Anymeasuresthatarebasedonnullablefieldsintherelationaldatasource.

References: FormoreinformationabouthowtooptimizeyourROLAPdesign,seethewhitepaperAnalysisServices ROLAPforSQLServerDataWarehouses (http://sqlcat.com/sqlcat/b/whitepapers/archive/2010/08/23/analysisservicesrolapforsqlserver datawarehouses.aspx).

104

3 Part 2: Running a Cube in Production

InPart2,youwillfindinformationonhowtotestandrunMicrosoftSQLServerAnalysisServicesinSQL Server2005,SQLServer2008,andSQLServer2008R2inaproductionenvironment.Thefocusofthis sectionishowyoucantest,monitor,diagnose,andremoveproductionissuesoneventhelargestscaled cubes.Thisbookalsoprovidesguidanceonhowtoconfiguretheserverforbestpossibleperformance. AnalysisServicescubesareaverypowerfultoolinthehandsofthebusinessintelligence(BI)developer. Theyprovideaneasywaytoexposeevenlargedatamodelsdirectlytobusinessusers.Unliketraditional, staticreporting,wherethequeryworkloadisknowninadvance,cubessupportadhocqueries. Typically,suchqueriesaregeneratedbytheMicrosoftExcelspreadsheetsoftwarewithoutthe businessuserbeingawareoftheintricaciesofthequeryengine.Becausecubesallowsuchgreat freedomforusers,thepowertheygivetodeveloperscomeswithresponsibility.Itisintheinteraction betweendevelopmentandoperationsthatthebusinessvalueofthecubesisrealizedandwherethe properstepscanbetakentoensurethatthepowerofthecubesisusedresponsibly. Itisthegoalofthissectiontomakeyouroperationsprocessesaspainlessaspossible,andtohaveyou runwiththebestpossibleperformancewithoutanyadditionaldevelopmentefforttoyourdeployed cubes.InPart2,youwilllearnhowtogetthebestoutofyourexistingdatamodelbymakingchanges transparenttothedatamodelandbymakingconfigurationchangesthatimprovetheuserexperienceof thecube. However,noamountofoperationalreadinesscancureapoorlydesignedcube.Althoughthissection showsyouwhereyoucanmakechangestransparenttoendusers,itisimportanttobeawarethatthere arecaseswheredesignchangeistheonlyviablepathtogoodperformanceandreliability.Cubesdonot takeawaytheubiquitousneedforinformeddatamodeling.Wehopethatyoullmakeuseofthe insightsandbestpracticesinPart1ofthisbookwhenstartingnewsolutionsfromscratch. Cubesdonotexistinisolationtheyrelyonrelationaldatasourcestobuildtheirdatastructures. Althoughafulltreatmentofgoodrelationaldatawarehousemodelingoutofscopeforthisdocument,it stillprovidessomepointersonhowtotunethedatabasesourcesfeedingthecube.Relationalengines varyintheirfunctionality;thisbookfocusesonguidanceforSQLServerdatasources.Muchofthe informationhereshouldapplyequallytootherenginesandyourDBAshouldbeabletotransferthe guidanceheretootherdatabasesystemsyourun.

105

3.1 Configuring the Server

InstallingAnalysisServicesisrelativelystraightforward.However,applyingcertainpostinstallation configurationoptionscanimproveperformanceandscalabilityofthecube.Thissectionintroduces thesesettingsandprovidesguidanceonhowtoconfigurethem.

3.1.1 Operating System

BecauseAnalysisServicesusesthefilesystemcacheAPI,itpartiallyreliesoncodeintheWindows Serveroperatingsystemforperformanceofitscaches.Smallchangesaremadetothefilesystemcache inmostversionsofWindowsServer,andthiscanhaveaneffectonAnalysisServicesperformance.We havefoundthatpatchesrelatedtothefollowingsoftwareshouldbeappliedforbestperformance. WindowsServer2008: KB961719ApplicationsthatperformasynchronouscachedI/Oreadrequestsandthatusea diskarraythathasmultiplespindlesmayencounteralowperformanceissueinWindowsServer 2008,inWindowsEssentialBusinessServer2008,orinWindowsVistaSP1

WindowsServer2008R2: KB979149AcomputerthatisrunningWindows7orWindowsServer2008R2becomes unresponsivewhenyourunalargeapplication KB976700Anapplicationstopsresponding,experienceslowperformance,orexperienceshigh privilegedCPUusageifmanylargeI/OoperationsareperformedinWindows7orWindows Server2008R2 KB982383YouencounteradecreaseinI/OperformanceunderaheavydiskI/Oloadona WindowsServer2008R2basedorWindows7basedcomputer

Kerberosenabledsystems: ForKerberosenabledsystems,youmayneedthefollowingadditionalpatches: http://support.microsoft.com/kb/969083/ http://support.microsoft.com/kb/2285736/

References: ConfigureMonitoringServerforKerberosdelegationhttp://technet.microsoft.com/en us/library/bb838742(office.12).aspx HowtoconfigureSQLServer2008AnalysisServicesandSQLServer2005AnalysisServicesto useKerberosauthenticationhttp://support.microsoft.com/kb/917409 ManageKerberosAuthenticationIssuesinaReportingServicesEnvironment http://msdn.microsoft.com/enus/library/ff679930.aspx

106

3.1.2 msmdsrv.ini
YoucancontrolthebehavioroftheAnalysisServicesenginetoagreatdegreefromthemsmdsrv.inifile. ThisXMLfileisfoundinthe<instancedir>/OLAP/Configfolder.Manyofthesettingsinmsmdsrv.ini shouldnotbechangedwithouttheassistanceofMicrosoftProductSupport,buttherearestillalarge numberofconfigurationoptionsthatyoucancontrolyourself.Itistheaimofthissectiontoprovideyou withtheguidanceyouneedtoproperlyconfigurethesesettings.Thefollowingtableprovidesan overviewofthesettingsavailabletoyouandcanserveasstartingpointforexploringreconfiguration options. Setting <MemoryHeapType> <HeapTypeForObjects> <HardMemoryLimit> Usedfor Increasingthroughputofhigh concurrencysystem PreventingAnalysisServices fromconsumingtoomuch memory <LowMemoryLimt> Reservingmemoryforthe AnalysisServicesprocess <PreAllocate> LockingAnalysisServices memoryallocation,and improvingperformanceon WindowsServer2003and potentiallyWindowsServer 2008 <TotalMemoryLimit> ControllingwhenAnalysis Servicesstartstrimmingworking set <CoordinatorExecutionMode> Settingconcurrencyofqueries <CoordinatorQueryMaxThreads> inthreadpoolduringquery execution <CoordinatorBuildMaxThreads> Increasingprocessingspeeds <AggregationMemoryMin> Increasingparallelismof <AggregationMemoryMax> partitionprocessing <CoordinatorQueryBalancingFactor> Preventingsingleusersfrom <CoordinatorQueryBoostPriorityLevel> monopolizingthesystem <LimitSystemFileCacheSizeMB> Controllingthesizeofthefile systemcache <ThreadPool> Increasingthreadpoolsforhigh (andsubsections) concurrencysystems <BufferMemoryLimit> Increasingcompressionduring <BufferRecordLimit> processing,butconsumingmore memory <DataStorePageSize> Increasingconcurrencyona <DataStoreHashPageSize> largemachine <MinIdleSessionTimeout> Cleaningupidlesessionsfaster <MaxIdleSessionTimeout> onsystemsthathavemany Describedin Section2.3.2.4 Section2.3.2

Section2.3.2 Section2.3.2.1

Section2.3.2

Section2.4.2

Section7.2.1 Section7.2.2 Section2.4.3 Section2.3.2 Section2.4 Section2.3.2.3

Section2.3.2.4 Section2.3.2.5

107

<IdleOrphanSessionTimeout> <IdleConnectionTimeout> <Port> <AllowedBrowsingFolders>

connect/disconnects ChangingtheportthatAnalysis Servicesislisteningon Controllingwhichfoldersare viewablebytheserver administrator Controllingwherediskspillsgo Controllingsegregationof dutiesscenarios Controllingsegregationof dutiesscenarios Killingqueriesthatareblocking aprocessingoperationorkilling theprocessingoperation Section5.1 Section2.5andSection5.5

<TempDir> <BuiltinAdminsAreServerAdmins> <ServiceAccountIsServerAdmin> <ForceCommitTimeout> <CommitTimeout>

Section2.5 Section5 Section5 Section7.4.4.4

3.1.2.1 A Warning About Configuration Errors

OneofthemostcommonmistakesencounteredwhentuningAnalysisServicesismisconfigurationof themsmdsrv.inifile.Becauseofthis,changingyourmsmdsrv.inifilesettingsshouldbedonewith extremecare.IfyouinheritthemanagingresponsibilitiesofanexistingAnalysisServicesserver,oneof thefirstthingsyoushoulddoistoruntheWindif.exeutility,whichcomeswithWindowsServer,against thecurrentmsmdsrv.inifilewiththedefaultmsmdsrv.inifile.Also,ifyouupgradedtoSQLServer2008 orSQLServer2008R2fromSQLServer2005,youwilllikelywanttocomparethecurrentmsmdsrv.ini filesettingswiththedefaultsettingsforSQLServer2008orSQLServer2008R2.Asdescribedinthe ThreadPoolsection,foroptimization,somesettingswerechangedinSQLServer2008andSQLServer 2008R2fromtheirdefaultvaluesinSQLServer2005.

108

Takethefollowingexamplefromacustomerconfiguration.Thecustomerwasexperiencingextremely poorquery/processingperformance.Whenwelookedatthememorycounters,wesawthis.

Figure 44 - Scaling Counters

109


Figure 45 - Memory measurements with scaled counters

Atfirstglance,thisscreenshotseemstoindicatethateverythingisOK;thereisnomemoryissue. However,inthiscase,becausethecustomerusedtheScaleSelectedCountersoptioninWindows PerformanceMonitorincorrectly,itwasnotpossibletocomparethethreecounters.Acloserlook,after thescaleofthecounterswascorrected,showsthis.

110


Figure 46 - Properly scaled counters

Nowthefactthatthereisaseriousmemoryproblemisclear.TakealookattheactualMemoryLimit HighKBvalue.InaPerformanceMonitorlogthisandMemoryLimitLowKBwillalwaysbeconstant valuesthatreflectthevalueofTotalMemoryLimitandLowMemoryLimitrespectivelyinthe msmdsrv.inifile.SoitappearsthatsomeonehasmodifiedtheTotalMemoryLimitfromthedefault80 percenttoanabsolutevalueof12,288bytes,probablythinkingthesettingwasinmegabyteswhenin reality,thesettingisinbytes.Asyoucansee,theresultsofanincorrectlyconfigured.inifilesettingcan bedisastrous. Themoralofthisstoryisthis:Alwaysbeextremelycarefulwhenyoumodifyyourmsmdsrv.inifile settings.OneofthefirstthingstheMicrosoftCustomerServiceandSupportteamdoeswhenworking onanAnalysisServicesissueisgrabthemsmdsrv.inifileandcompareitwiththedefault.inifileforthe customersversionofAnalysisServices.Therearenumerousfilecomparisontoolsyoucanuse,themost obviousbeingWindiff.exe,whichcomeswithWindowsServer. References: 111

HowtoUsetheWindiff.exe eUtilityhttp p://support.m microsoft.com m/kb/159214

3.1.3 M Memory Co onfigurati ion

Thissectio ondescribesthememorymodelofAna alysisServices sandprovide esguidanceo onhowtoinitially configurememorysett tingsforaser rver.Aswillb becomeclearintheMonitoringandTun ningsection,itis oodideatore eadjustthem memorysettin ngsasyoulearnmoreabou uttheworklo oadthatisrun nning oftenago onyourse erver. Tounders standthetrad deoffsyouwi illmakedurin ngserverconf figuration,itisusefultounderstandab bit moreabo outhowAnaly ysisServicesu usesmemory y.

3.1.3.1 Understanding Memory Usa age

Apartfrom mtheexecuta ableitself,An nalysisService eshassevera aldatastructu uresthatcons sumephysica al memory,includingcac chesthatboosttheperform manceofuse erqueries.Me emoryisalloc catedusingth he emoryAPI(wi iththeexcept tionofaspec cialcasedescribedlater)andispartoft the standardWindowsme .exeprocessprivatebytes sandworking gset.Analysis sServicesdoe esnotusethe eAWEAPIto Msmdsrv. allocatem memory,whic chmeansthat ton32bitsy ystemsyouar rerestrictedt toeither2GB Bor3GBof memoryf forAnalysisServices.Ifyou uneedmorememory,werecommendthatyoumov vetoa64bit t system.Thetotalmem moryusedbyAnalysisServ vicescanbem monitoredinP PerformanceMonitorusin ng SOLAP:Memo ory\Memory yUsageKbor rProcess\Priv vateBytesm msmdsrv.exe e. either:MS Memorya allocatedbyA AnalysisServi icesfallsintotwobroadca ategories:shr rinkableandn nonshrinkable memory.

Figure 4 - Memor Structur 47 ry res

Nonshrin nkablememor ryincludesallthestructur resthatarere equiredtokeeptheserver rrunning:act tive usersessions,workingmemoryofr runningqueries,metadata aabouttheob bjectsontheserver,andt the 112

processitself.AnalysisServicesdoesnotreleasememoryinthenonshrinkablememoryspacesthough thismemorycanseestillbepagedoutbytheoperatingsystem(butseethesectiononPreAllocate). Shrinkablememoryincludesallthecachesthatgraduallybuilduptoincreaseperformance:Theformula enginehasacalculationcachethattriestokeepfrequentlyaccessedandcalculatedcellsinmemory. Dimensionsthatareaccessedbyusersarealsokeptincache,tospeedupaccesstodimensions, attributes,andhierarchies.Thestorageenginealsocachescertainaccessedsubcubes.(Notallsubcubes arecached.Forexample,theonesusedbyarbitraryshapesarenotkept.)AnalysisServicesgradually trimsitsworkingsetiftheshrinkablememorycachesarenotusedfrequently.Thismeansthatthetotal memoryusageofAnalysisServicesmayfluctuateupanddown,dependingontheserverstatethisis expectedbehavior. OutsideoftheAnalysisServicesprocessspace,youhavetoconsiderthememoryusedbytheoperation systemitself,otherservices,andthememoryconsumedbythefilesystemcache.Ideally,youwantto avoidpagingimportantmemoryandaswewillsee,thismayrequiresomeconfigurationtweaks. References: Russinovich,MarkandDavidSolomon:WindowsInternals,5thedition. http://technet.microsoft.com/enus/sysinternals/bb963901 Webb,Chris,MarcoRusso,andAlbertoFerrari:ExpertCubeDevelopmentwithMicrosoftSQL Server2008AnalysisServicesChapter11 ArbitraryShapesinAS2005https://kejserbi.wordpress.com/2006/11/16/arbitraryshapesin as2005/

3.1.3.2 Memory Setting for the Analysis Service Process

TheAnalysisServicesmemorybehaviorcanbecontrolledbyusingparametersavailableinthe msmdsrv.inifile.ExceptfortheLimitFileSystemCacheMBsetting,allthememorysettingsbehavelike this: Ifthesettinghasavaluebelow100,therunningvalueisthatpercentoftotalRAMonthe machine. Ifthesettinghasavalueabove100,therunningvalueisthatamountofKB.

Foreaseofuse,werecommendthatyouusethepercentagevalues.Thetotalmemoryallocationsof AnalysisServicesprocessarecontrolledbythefollowingsettings. LowMemoryLimitThisistheamountofmemorythatAnalysisServiceswillholdonto,without trimmingitsworkingset.Aftermemorygoesabovethisvalue,AnalysisServicesbeginstoslowly deallocatememorythatisnotbeingused.TheconfigurationvaluecanbereadfromthePerformance Monitorcounter:MSOLAP:Memory\LowMemoryLimit.AnalysisServicesdoesnotallocatethismemory atstartup;however,afterthememoryisallocated,AnalysisServicesdoesnotreleaseit.

113

PreAllocateThisoptionalparameter(withadefaultof0)enablesyoutopreallocatememoryatservice startup.Itisdescribedinmoredetaillaterinthissection.PreAllocateshouldbesettoavaluelessthan orequaltoLowMemoryLimit. TotalMemoryLimitWhenAnalysisServicesexceedsthismemoryvalue,itstartsdeallocatingmemory aggressivelyfromshrinkablememoryandtrimmingitsworkingset.Notethatthissettingisnotanupper memorylimitfortheservice;ifalargequeryconsumesalotofresources,theservicecanstillconsume memoryabovethisvalue.ThisvaluecanalsobereadfromthePerformanceMonitorcounter: MSOLAP:Memory\TotalMemoryLimit. HardMemoryLimitThissettingisonlyavailableinSQLServer2008AnalysisServicesandSQLServer 2008R2AnalysisServicesnotSQLServer2005AnalysisServices.Itisamoreaggressiveversionof TotalMemoryLimit.IfAnalysisServicesgoesaboveHardMemoryLimit,usersessionsareselectively killedtoforcememoryconsumptiondown. LimitSystemFileCacheMBTheWindowsfilesystemcache(describedlater)isactivelyusedbyAnalysis Servicestostorefrequentlyusedblocksondisk.Forsomescanintensiveworkloads,thefilesystem cachecangrowsolargethatAnalysisServicesisforcedtotrimitsworkingset.Toavoidthis,Analysis ServicesexposestheWindowsAPItolimitthesizeofthefilesystemcachewithaconfigurationsetting inMsmdsrv.ini.Ifyouchoosetousethissetting,notethatitlimitsthefilesystemcachefortheentire server,notjustfortheAnalysisServicesfiles.Thisalsomeansthatifyourunmorethanoneinstanceof AnalysisServicesontheserver,youshouldusethesamevalueforLimitSystemFileCacheMBforeach instance. 3.1.3.2.1 PreAllocate and Locked Pages ThePreAllocatesettingfoundinmsmdsrv.inicanbeusedtoreservevirtualand/orphysicalmemoryfor AnalysisServices.ForinstallationswhereAnalysisServicescoexistswithotherservicesonthesame machine,settingPreAllocatecanprovideamorestablememoryconfigurationandsystemthroughput. NotethatiftheserviceaccountusedtorunAnalysisServicesalsohastheLockpagesinMemory privilege,PreAllocatecausesAnalysisServicestouselargememorypages.Largememorypagesare moreefficientonabigmemorysystem,buttheytakelongertoallocate.LockpagesinMemoryisset usingGpedit.msc.Bearinmindthatlargememorypagescannotbeswappedouttothepagefile.While thiscanbeanadvantagefromaperformanceperspective,ahighnumberofallocatedlargepagescan causethesystemtobecomeunresponsive. Important:PreAllocatehasthelargestimpactontheWindowsServer2003operatingsystem.Withthe introductionofWindowsServer2008,memorymanagementhasbeenmuchimproved.Wehavebeen testingthissettingonWindowsServer2008,buthavenotmeasuredanysignificantperformance benefitsofusingPreAllocateonthisplatform.However,youmaywantstillwanttomakeuseofthe lockedpagesfunctionalityinWindowServer2008. TolearnmoreabouttheeffectsofPreAllocate,seethefollowingtechnicalnote:

114

RunningMicrosoftSQLServer2008AnalysisServicesonWindowsServer2008vs.Windows Server2003andMemoryPreallocation:LessonsLearned (http://sqlcat.com/technicalnotes/archive/2008/07/16/runningmicrosoftsqlserver2008 analysisservicesonwindowsserver2008vswindowsserver2003andmemorypreallocation lessonslearned.aspx)

References: Howto:EnabletheLockPagesinMemoryOption(Windows)http://msdn.microsoft.com/en us/library/ms190730.aspx

3.1.3.2.2 AggregationMemory Max/Min UndertheserverpropertiesofSQLServerManagementStudioandinmsmdsrv.ini,youwillfindthe followingsettings: OLAP\Process\AggregationMemoryLimitMin OLAP\Process\AggregationMemoryLimitMax

Thesetwosettingsdeterminehowmuchmemoryisallocatedforthecreationofaggregationsand indexesineachpartition.WhenAnalysisServicesstartspartitionprocessing,parallelismisthrottled basedontheAggregationMemoryMin/Maxsetting.Thesettingisperpartition.Forexample,ifyou startfiveconcurrentpartitionprocessingjobswithAggregationMemoryMin=10,anestimated50 percent(5x10%)ofreservedmemoryisallocatedforprocessing.Ifmemoryrunsout,newpartition processingjobsareblockedwhiletheywaitformemorytobecomeavailable.Onalargememory system,allocating10percentofavailablememoryperpartitionmaybetoomuch.Inaddition,Analysis Servicesmaysometimesmisestimatethemaximummemoryrequiredforthecreationofaggregatesand indexes.Ifyouprocessmanypartitionsinparallelonalargememorysystem,loweringthevalueof AggregationMemoryLimitMinandAggregationMemoryMaxmayincreaseprocessingspeed.Thisworks becauseyoucandriveahigherdegreeofparallelismduringtheprocessindexphase. LiketheotherAnalysisServicesmemorysettings,ifthissettinghasavaluegreaterthan100itis interpretedasafixedamountofkilobytes,andifislowerthan100,itisinterpretedasapercentageof thememoryavailabletoAnalysisServices.Formachineswithlargeamountsofmemoryandmany partitions,usinganabsolutekilobytevalueforthesesettingsmayprovideabettercontrolofmemory thanusingapercentagevalue. 3.1.3.2.3 BufferMemoryLimit and BufferRecordLimit OLAP\Process\BufferMemoryLimitdeterminesthesizeofthefactdatabuffersusedduringpartition dataprocessing.WhilethedefaultvalueoftheOLAP\Process\BufferMemoryLimitissufficientformost deployments,youmayfinditusefultoalterthepropertyinthefollowingscenario. Ifthegranularityofyourmeasuregroupismoresummarizedthantherelationalsourcefacttable,you maywanttoconsiderincreasingthesizeofthebufferstofacilitatedatagrouping.Forexample,ifthe sourcedatahasagranularityofdayandthemeasuregrouphasagranularityofmonth;AnalysisServices 115

mustgroupthedailydatabymonthbeforewritingtodisk.Thisgroupingoccurswithinasinglebuffer anditisflushedtodiskafteritisfull.Byincreasingthesizeofthebuffer,youdecreasethenumberof timesthatthebuffersareswappedtodisk.Becausetheincreasedbuffersizesupportsahigher compressionratio,thesizeofthefactdataondiskisdecreased,whichprovideshigherperformance. However,beawarethathighvaluesfortheBufferMemoryLimitusemorememory.Ifmemoryrunsout, parallelismisdecreased. Youcanuseanotherconfigurationsettingtocontrolthisbehavior:BufferRecordLimit.Thissettingis expressedinreceivedrecordsfromthedatasourceinsteadofaMemory%/Kbsize.Thelowerofthe twotakesprecedence.Forexample,ifBufferMemoryLimitissetto10percentofa32GBmemory spaceandBufferRecordLimitissetto10millionrows,either3.2GBor10,000,000timestherowsizeis allocatedforthemergebuffer,whicheverissmaller. 3.1.3.2.4 Heap Settings, DataStorePageSize, and DataStoreHashPageSize Duringqueryexecution,AnalysisServicesgenerallytouchesalotofdataandperformsmanymemory allocations.AnalysisServiceshashistoricallyreliedonitsownheapimplementationtoachievethebest performance.However,sinceWindowsServer2003,advancesintheWindowsServeroperatingsystem meanthatmemorycannowbemanagedmoreefficientlybytheoperatingsystem.Thisturnsouttobe especiallytrueformultiuserscenarios.Becauseofthis,serversthathavemanyusersshouldgenerally applythefollowingchangestothemsmdsrv.inifile.
Default Multi-user, faster heap Setting 1 2 <MemoryHeapType> 1 0 <HeapTypeForObjects> ItisalsopossibletoincreasethepagesizethatisusedformanagingthehashtablesAnalysisServices usestolookupvalues.Especiallyonmodernhardware,wehaveseenthefollowingchangeyielda significantincreaseinthroughputduringqueryexecution.

Setting <DataStorePageSize> <DataStoreHashPageSize> References:

Default 8192 8192

Bigger pages 65536 65536

KB2004668Youexperiencepoorperformanceduringindexingandaggregationoperations whenusingSQLServer2008AnalysisServiceshttp://support.microsoft.com/kb/2004668

3.1.3.2.5 Clean Up Idle Sessions Clienttoolsmaynotalwayscleanupsessionsopenedontheserver.Thememoryconsumedbyactive sessionsisnonshrinkable,andhenceisnotreturnedtheAnalysisServicesoroperatingsystemprocess forotherpurposes.Afterasessionhasbeenidleforsometime,AnalysisServicesconsidersthesession expiredandmovethememoryusedtotheshrinkablememoryspace.Butthesessionstillconsumes memoryuntilitiscleanedup. 116

Becauseidlesessionsconsumevaluablememory,youmaywanttoconfiguretheservertobemore aggressiveaboutcleaningupidlesessions.Therearesomemsmdsrv.inisettingsthatcontrolhow AnalysisServicesbehaveswithrespecttoidlesessions.Notethatavalueofzeroforanyofthefollowing settingsmeansthatthesessionsorconnectioniskeptaliveindefinitely. Setting <MinIdleSessionTimeOut> Description Thisistheminimumamountoftimeasessionisallowedtobeidle beforeAnalysisServicesconsidersitreadytodestroy.Sessionsare destroyedonlyifthereismemorypressure. Thisisthetimeafterwhichtheserverforciblydestroysthesession, regardlessofmemorypressure. Thisisthetimeoutthatcontrolssessionsthatnolongerhavea connectionassociatedwiththem.Examplesoftheseareusers runningaqueryandthendisconnectingfromtheserver. Thistimeoutcontrolshowlongaconnectioncanbekeptopenand idleuntilAnalysisServicesdestroysit.Notethatiftheconnectionhas noactivesessions,MaxIdleSessionTimeouteventuallymarksthe sessionforcleaningandtheconnectioniscleanedwithit.

<MaxIdleSessionTimeout> <IdleOrphanSessionTimeout>

<IdleConnectionTimeout>

Ifyourserverisundermemorypressureandhasmanyusersconnected,butfewexecutingqueries,you shouldconsiderloweringMinIdleSessionTimeOutandIdleOrphanSessionTimeouttocleanupidle sessionsfaster.

3.1.4 Thread Pool and CPU Configuration

AnalysisServicesusesthreadpoolstomanagethethreadsusedforqueriesandprocessing.Thethread managementsubsystemthatAnalysisServicesusesenablesyoutofinetunethenumberofthreadsthat arecreated.TuningthethreadpoolisabalancingactbetweenCPUoverutilizationandunderutilization: Iftoomanythreadsarecreated,unnecessarycontextswitchesandcontentionforsystemresources lowerperformance,andtoofewthreadscanleadtoCPUanddiskunderutilization,whichmeansthat performanceisnotoptimalonthehardwareallocatedtotheprocess.Whenyoutuneyourthreadpools, itisessentialthatyoubenchmarkyourperformancebeforeandafteryourconfigurationchanges; misconfigurationofthethreadpoolcanoftencauseunforeseenperformanceissues.

3.1.4.1 Thread Pool Sizes

Thissectiondiscussesthesettingsthatcontrolthreadpoolsizes. ThreadPool\Process\MaxThreadsdeterminesthemaximumnumberofavailablethreadstoAnalysis ServicesduringprocessingandforaccessingtheI/Osystemduringqueries.Onlarge,multipleCPU machines,thedefaultvalueinSQLServer2008AnalysisServicesandSQLServer2008R2Analysis Servicesof64or10multipliedbythenumberofCPUcores(whicheverishigher)maybetoolowtotake advantageofallCPUcores.InSQLServer2005AnalysisServicesthesettingsforprocessthreadpool weresettothestaticvalueof64.IfyouarerunninganinstallationofSQLServer2005AnalysisServices,

117

orifyouupgradedfromSQLServer2005AnalysisServicestoSQLServer2008AnalysisServicesorSQL Server2008R2AnalysisServices,itmightbeagoodideatoincreasethethreadpool. ThreadPool\Query\MaxThreadsdeterminesthemaximumnumberofthreadsavailabletotheAnalysis Servicesformulaengineforansweringqueries.InSQLServer2008AnalysisServicesandSQLServer 2008R2AnalysisServices,thedefaultis2multipliedbythenumberoflogicalCPUor10,whicheveris higher.InSQLServer2005AnalysisServices,thedefaultvaluewasfixedat10.Again,ifyouarerunning onSQLServer2005AnalysisServicesoranupgradedSQLServer2005AnalysisServices,youmaywant tousethenewsettings.

3.1.4.2 CoordinatorExecutionMode and CoordinatorQueryMaxThreads

AnalysisServicesusesacentralizedstorageenginejobarchitectureforbothqueryingandprocessing operations.Whenasubcuberequestorprocessingcommandisexecuted,acoordinatorthreadis responsibleforgatheringthedataneededtosatisfytherequest. ThevalueofCoordinatorExecutionModelimitsthetotalnumberofcoordinatorjobsthatcanbe executedinparallelbyasubcuberequestinthestorageengine.Anegativevaluespecifiesthemaximum numberofparallelcoordinatorjobsthatcanstartperprocessingcorepersubcuberequest.Bydefault CoordinatorExecutionModeissetto4,whichmeanstheserverislimitedto4jobsinparallelpercore. Forexample,ona16coremachinewiththedefaultvaluesofCoordinatorExecutionMode=4,atotal of64concurrentthreadscanexecutepersubcuberequest. Whenasubcubeisrequested,acoordinatorthreadstartsuptosatisfytherequest.Thecoordinatorfirst queuesuponejobforeachpartitionthatmustbetouched.Eachofthosejobsthencontinuestoqueue upmorejobs,dependingonthetotalnumberofsegmentsthatmustbescannedinthepartition.The valueofCoordinatorQueryMaxThreads,withadefaultof16,limitsthenumberofpartitionjobsthat canbeexecutedinparallelforeachpartition.Forexample,iftheformulaenginerequestsasubcube thatrequirestwopartitionstobescanned,thestorageengineislimitedtoamaximumof32threadsfor scanning.Alsonotethatboththecoordinatorjobsandthescanthreadsarelimitedbythemaximum numberofthreadsconfiguredinThreadPool/Processing/MaxThreads.Thefollowingdiagramillustrates therelationshipbetweenthedifferentthreadpoolsandthecoordinatorsettings.

118


Figure 48 - Coordinator Queries

Ifyouincreasetheprocessingthreadpool,youshouldmakesurethattheCoordinatorExecutionMode settings,aswellastheCoordinatorQueryMaxThreadssettings,havevaluesthatenableyoutomakefull useofthethreadpool. Ifthetypicalqueryinyoursystemtouchesmanypartitions,youshouldbecarefulwiththe CoordinatorQueryMaxThreads.Forexample,ifeveryquerytouches10partitions,atotal160threads canbeusedjusttoanswerthatquery.Itwillnottakemanyqueriestorunthethreadpooldryunder thoseconditions.Insuchacase,considerloweringthesettingofCoordinatorQueryMaxThreads.

3.1.4.3 Multi-User Process Pool Settings

Inmultiuserscenarios,longrunningqueriescanstarveotherqueries;specificallytheycanconsumeall availablethreads,blockingexecutionofotherqueriesuntilthelongerrunningqueriescomplete. Youcanreducetheaggressivenessofhoweachcoordinatorjoballocatesthreadspersegmentby modifyingCoordinatorQueryBalancingFactorandCoordinatorQueryBoostPriorityLevelasfollows. Setting 119 Default Multiusernonblocking settings

CoordinatorQueryBalancingFactor 1 CoordinatorQueryBoostPriorityLevel 3

1 0

Ifyouwanttounderstandexactlywhatthesesettingsdo,youneedtoknowalittleabouttheinternals ofAnalysisServices.First,awordofwarning:TheremainderofthissectionlooksatAnalysisServicesat averydetailedlevel.Itisperfectlyacceptabletotakeaqueryworkloadandtestwiththedefault settingsandthenwiththemultiusersettingstodecideifitisworthmakingthesechanges. Withthedisclaimeroutoftheway,letslookatanexampletoexplainthisbehavior.Ona45core WindowsServer2008serverwithdefault.inifilesettings,youhavealongrunningquerythatappearsto beblockingmanyoftheotherqueriesbeingexecutedbyotherusers.BehindthescenesinAnalysis Services,multiplesegmentjobs(differentfromcoordinatorjobs)arecreatedtoquerytherespective segments.AsegmentofdatainAnalysisServicesiscomposedofroughly64,000records,whichare subdividedintopages.Thereare256pagesinasegmentand256recordsinapage.Theserecordsare broughtintomemoryinchunksuponrequestbyqueries.AnalysisServicesdetermineswhichpages needtobescannedtoretrievetherelevantrecordsforthedatarequested.Thesejobsarechained, meaningJob1queuesJob2tothethreadpool,Job1performsitsownjob,Job2queuesJob3tothe threadpool,Job2performsitsownjob,andsoon.Eachjobhasitsownthread,orsegmentjob. Inourexample,usingthedefaultsettingsthefirstqueryfiresoff720jobs,scanningalotofdata.The secondqueryfiresoff1job.Thefirst720jobsfireofftheirown720jobs,usingupallofthethreads.This preventsthesecondqueryfromexecuting,becausenothreadsareavailable.Thisbehaviorcauses blockingofthesecondandsubsequentqueriesthatneedthreadsfromtheprocesspool.Themultiuser settings(CoordinatorQueryBalancingFactor=1,CoordinatorQueryBoostPriorityLevel=0)preventallof thethreadsfrombeingallocatedsothesecondandthirdqueriescanexecutetheirjobs. Again,eachsegmentjobusesonethread.Ifthelongrunningqueryrequiresscanningofmultiple segments,AnalysisServicescreatesthenecessarynumberofthreads.Insingleusermode,thefirst queryfiresoff720jobs,andthesecondqueryfiresoff1job.Eachsegmentjobisimmediatelyqueued upbeforethepriorsegmentjobbeginsscanningdata.Thefirst720jobsfireofftheirown720jobs, preventingsecondqueryfromexecuting.Inmultiusermode,notallthreadsareallocated,allowing secondquery(andthird)toexecutetheirjobs. Becarefulmodifyingthesesettings;althoughthesesettingsreduceorstoptheblockingofshorter runningqueriesbylongerrunningones,itmayslowtheresponsetimesofindividualqueries.(Inthe figure,SSASstandsforSQLServerAnalysisServices.)

120


Figure 49 - Default settings

Figure 50 - Multi-user settings

3.1.4.4 Hyperthreading
Wereceivenumerousquestionsfromcustomersaroundhyperthreading.TheAnalysisServices developmentteamhasnoofficialrecommendationonhyperthreading;itisincludedinthisbookonly 121

becauseourcustomersaskaboutitfrequently.Withthatsaid,wehavemadethefollowingobservations ininstallationsinwhichhyperthreadingisusedwithAnalysisServices: IftheloadonyourserverismoreCPUbound,turningonhyperthreadingoffersno improvement,inourexperience. IftheloadonyourserverismoreI/Obound,theremaybesomebenefittoturningon hyperthreading. Processorsareconstantlychanging,andmanycharacteristicsofbehaviorandperformance relatedtohyperthreadingaregoingtobespecifictotheprocessor.

3.1.5 Partitioning, Storage Provisioning and Striping

WhenyouconfigurethestorageofaserveryouareoftenpresentedwithaseriesofLUNsfordata storage.TheseareprovisionedfromeitheryourSAN,internaldrives,orNANDmemory.NANDmemory, insomeconfigurationalsoknownasSolidStateDisks/Devices(SSD)istypicallyagoodfitforlargecubes, becausethelatencyandI/OpatternfavoredbythesedrivesareagoodmatchfortheAnalysisServices storageengineworkload. ThequestionishowtomakethebestuseofthestorageforyourAnalysisServicesinstance. Apartfromperformanceandcapacity,thefollowingfactorsmustalsobeconsideredwhendesigningthe diskvolumesforAnalysisServices: IsclusteringoftheAnalysisServicesinstancerequired? Willyoubebuildingascaledoutenvironment?

Consultthesubsectionsthatfollowforguidanceinthesespecificsetups.However,thefollowinggeneral guidanceapplies. SANMegaLUNs:IfyouareusingaSAN,yourstorageadministratormaybeabletoprovisionyoua single,largeLUNforyourcubes.Unlessyouplantobuildaconsolidationenvironment,havingsucha single,megaLUNisprobablytheeasiestandmostmanageablewaytoprovisionyourstorage.Firstof all,itprovidestheIOPSasageneralresourcetotheserver.Secondly,andadditionaladvantageofa megaLUNisthatyoucaneasilydiskalignthisinWindowsServer2003.ForWindowsServer2008you donotneedtoworryaboutdiskalignmentonnewlycreatedvolumes. WindowsServerdynamicdiskstripes:UsingDiskManageritispossibletocombinemultipleLUNsintoa singleWindowsvolume.Thisisaverysimplewaytocombinemultiple,similarlysized,similarly performingLUNsintoasinglemountpointordriveletter.Wehavetesteddynamicdiskstripesin WindowsServer2008R2allthewayupto100.000IOPSandtheperformanceoverheadtothatlevelis negligible. Note:Youcannotusedynamicdiskstripesinacluster.Thisisdiscussedingreaterdetaillaterin Part2.

122

Drivelett tersversusmountpoints:Bothdrivele ettersandmo ountpointswillworkwithAnalysisServ vices cubes.Ift theserverisd dedicatedtoa asingleAnaly ysisServicesinstance,adr rivelettermaybethesimp plest solution.C Choosingdriv velettersvers susmountpo ointsisoftenamatterofp personalprefe erence,oritc can bedrivenbythestandardsofyouroperationste eam.Fromtheperspective eofperforma ance,oneisnot totheother. superiort AllowedB BrowsingFolders:Rememb berthatinord derforadirectorytobevi isibletoadmi inistratorsof AnalysisS ServicesinSQ QLServerMan nagementStu udio,itmustb belistedinAl llowedBrows singFolders,w which isavailabl leinSQLServ verManagem mentStudioby yclickingServ verPropertie esandthenA Advanced Propertie es.

Figure 5 Allowe 51 edBrowsin ngFolders

TheAnaly ysisServicesa accountmustalsohavepe ermissiontob bothreadand dwritetothesefolders.Ju ust addingthemtotheAll lowedBrowsingFolderslist tisnotenoug ghyoumus stalsoassignfilesystem permissio ons. MFTversu usGPTdisk:Ifyouexpectyourcubestobelargerth han2terabytes,youshoul lduseaGPTd disk. Thedefau ult,anMFTdi isk,onlyallow ws2terabytepartitions. TempDirFolder:TheTe empDirfolde er,configured dintheAdvan ncedPropertiesoftheserv ver,shouldbe e movedtothefastestv volumeyouha ave.Thismay ymeanyouw willshareTem mpDirwithcub bedata,whic chis uplancapacit tyaccordingly y.Formorein nformation,seethesectionabouttheT TempDirfolde er. fineifyou NTFSAllo ocationUnitS Size(AUS):W Withcarefulop ptimization,it tissometime espossibletogetslightlyb better performancebysmartlychoosingb between32Ko or64K(afew wpercent).Bu ut,unlessyouarehuntingfor benchmar rkperforman nce,justgowith32K.Ifyou uhavestanda ardizedon64 4KforotherSQLServer services,t thatwillwork ktoo. 123

Dontsuboptimizestorage:Thereareafewcaseswhereitmakessensetosplityourdataintomultiple volumesforexampleifyouhavedifferentstoragetypesattachedtotheserver(suchasNANDfor latestdataandSATAdrivesforhistoricaldata).However,itisgenerallynotagoodideatosuboptimize thestoragelayoutofAnalysisServicesbycreatingcomplicateddatadistributionsthatspandifferent storagetypes.Theruleofthumbforoptimaldisklayoutistoutilizealldiskdrivesforallcubepartitions. Createlargepoolsofdisk,presentedassingle,largevolumes. ExcludeAnalysisServicesfoldersfromvirusscanners:Ifyouarerunningavirusscannerontheserver, makesuretheDatafolder,TempDir,andbackupfoldersarenotbeingscannedortouchedbythefilter driversoftheantivirustool.TherearenoexecutablefilesintheseAnalysisServicesfolders,andenabling virusscannersonthefoldermayslowdownthediskaccessspeedssignificantly. ConsiderDefragmentingtheDataFolder:AnalysisServicescubesgetveryfragmentedovertime.We haveseencaseswheredefragmentingcubedatafolders,usingthestandarddiskdefragmentutility, yieldedasubstantialperformancebenefit.Notethatdefragmentingadrivealsohasimpactonthe performanceofthatdriveasitmovesthefilesaround.IfyouchoosetodefragmentanAnalysisServices drive,dosoinbatchwindowwheretheservicecanbetakenofflinewhilethedefragmentationhappens, orplandiskspeedsaccordinglytomakesuretheperformanceimpactisacceptable. References: Whitepaper:ConfiguringDynamicVolumeshttp://technet.microsoft.com/en us/library/bb727122.aspx

3.1.5.1 I/O Pattern

BecauseAnalysisServicesusesbitmapindexestoquicklylocatefactdata,theI/Ogeneratedismostly randomreads.I/Osizeswilltypicallyaveragearound32KBblocksizes. AswithallSQLServerdatabasessystems,werecommendthatyoutestyourI/Osubsystembefore deployingthedatabase.Thisallowsyoutomeasurehowclosetheproductionsystemistoyour maximumpotentialthroughput.NotrunningpreproductionI/OtestingofanAnalysisServicesserveris theequivalentofnotknowinghowmuchmemoryorhowmanyCPUcoresyourserverhas. Duetothethreadingarchitectureofthestorageengine,theI/Opatternwillalsobehighlymulti threaded.BecausetheNTFSfilesystemcacheissuessynchronousI/O,eachthreadwillhaveonlyoneor twooutstandingI/Orequests.Incidentally,thismeansthatcubeswilloftenrunverywellonNAND storagethatfavorthistypeofI/Opattern. TheAnalysisServicesI/OpatterncanbeeasilysimulatedandtestedusingSQLIO.exe.Thefollowing commandlineparameterwillgiveyouagoodindicationoftheexpectedperformance: sqlio -b32-frandom-o1-s30-t256-kR<path of file> Makesureyourunonasufficientlylargetestfile.Formoreinformation,refertothelinksinthe Referencessection. 124

Reference es: SQ QLIOdownload:http://ww ww.microsoft t.com/downlo oads/en/deta ails.aspx?familyid=9a8b005b 84 4e44f248d6 65cb53442d9 9e19&display ylang=en Whitepaper:A W AnalyzingI/Ocharacteristi icsandsizingstorageforS SQLServerDa atabase Applicationshttp://sqlcat t.com/whitep papers/archiv ve/2010/05/10/analyzingiocharacteri istics ndsizingstor ragesystemsforsqlserve erdatabasea applications.a aspx an Blog:TheMem moryShellGame ttp://blogs.m msdn.com/b/n ntdebugging/archive/2007 7/10/10/them memoryshellgame.aspx ht

3.1.6 N Network Configuration C

Duringthe eProcessDat taphaseofAn nalysisServicesprocessing g,rowsaretransferredfro omtherelatio onal databasespecifiedint thedatasourc cetoAnalysis sServices.Ify yourdatasou urceisonare emoteserver, ,the datashou uldberetrieve edoverTCP/IP.Itisimpor rtanttomake esureallnetw workingcomp ponentsare configuredtosupporttheoptimalt throughput.If fyourEthern netthroughpu utisconsisten ntlycloseto8 80 percento ofyourmaxim mumcapacity, ,addingmore enetworkcap pacitytypicallyspeedsupProcessData. Creatinga ahighspeednetworkisfa airlyeasywith hthenetwork kinghardware eavailableto oday.Specifica ally mostserv verscomewit thaGigabitN NICsoutofthe ebox.Infinibandand10g gigabitNICsar realsoavailable forevenm morethrough hput.Withtha atsaid,youroverallthroughputcanbe elimitedbyro outersandother hardware einyournetw workthatdontsupportthe espeedofyo ourNIC.Youc canusetheN Networkingta abin TaskManagertoquick klydetermine eyourlinkspe eed.Inthefollowingscree enshotyoucanseethattheNIC maximumof1 1Gbps. andswitchsupportam

Figure 5 - Viewin NIC spe 52 ng eed

Ifyouhav vea1GbpsN NIC,butonlya a100Mbpsswitch,TaskM Managerdispl lays100Mbp ps.Depending gon yournetw worktopology ytheremayb bemoretode eterminingyo ourlinkspeed dthanthis,bu utusingTask Managerisasimplewaytogetaro oughideaoft theconfigurat tion. Inadditiontocreatingahighspeed dnetwork,the erearesome eadditionalco onfigurations syoucanchan nge tofurther rspeedupne etworktraffic.

3.1.6.1 Use Shar Memory for Loc SQL Se red cal erver Data Sources a s
IfAnalysis sServicesiso onthesamep physicalmach hineastheda atasource,an ndthedataso ourceisSQL Server,yo oushouldmakesureyoua areexchangin ngdataovert thesharedme emoryprotoc col.Shared memoryismuchfasterthanTCP/IP P,asitavoidstheoverhead dofthetrans slationlayersinthenetwork stack.Sha aredmemoryisonlypossib bleiftheSQL LServerdatab baseisonthe esamephysic calmachineas AnalysisS Services.Ifyoucannotgetahighspeednetworksetupinyouror rganization,y youcansometimes benefitfro omrunningS SQLServeran ndAnalysisSe ervicesonthe esamephysic calmachine.

125

Tomodify yyourdataso ourceconnec ctiontospecif fysharedmem mory: 1. First,m makesuretha attheShared dMemorypro otocolisenab bledinSQLSe erver Config gurationMana ager.

2. Next,a addlpc:toyo ourdatasourc ceintheconn nectionstring gtoforcesha aredmemory.

3. Aftery youstartproc cessing,youc canverifyyou urconnection nisusingshar redmemoryb by execut tingthefollow wingSELECTs statement.
SELECT session_i T id, net_tr ransport, n net_packet t_size FROMsy ys.dm_exec_ _connectio ons

Thenet_t transportfortheAnalysisServicesSPID Dshouldshow w:Sharedmemory. Formoreinformationaboutshared dmemorycon nnections,see eCreatingaValidConnec ctionStringU Using msdn.microso oft.com/enus s/library/ms1 187662.aspx). . SharedMemoryProtocol(http://m Tocompa areTCP/IPand dsharedmem mory,weranthefollowing gtwoprocess singcomman nds.
<!--SQL Server Na ative Clien with TC nt CP--> <Batchxm mlns="http p://schemas s.microsof ft.com/anal lysisservi ices/2003/e engine"> <Paralle el> <Process s <Object> > <Databas seID>Adven nture Works DW 2008R s R2</Databas seID> <CubeID> >Adventure Works</Cu e ubeID> </Object t> <Type>Pr rocessFull l</Type> <DataSou urcexsi:ty ype="Relati ionalDataS Source"> <ID>Adve enture Wor rks DW</ID> > <Name>Ad dventure Works DW</N W Name>

126

<ConnectionString> Provider=SQLNCLI10.1;Data Source=tcp:johnsi5\kj; Integrated Security=SSPI;Initial Catalog=AdventureWorksDW2008R2; </ConnectionString> <ImpersonationInfo> <ImpersonationMode>ImpersonateCurrentUser</ImpersonationMode> </ImpersonationInfo> <Timeout>PT0S</Timeout> </DataSource> </Process> </Parallel> </Batch> <!--SQL Native Client with shared memory--> <Batchxmlns="http://schemas.microsoft.com/analysisservices/2003/engine"> <Parallel> <Process <Object> <DatabaseID>Adventure Works DW 2008R2</DatabaseID> <CubeID>Adventure Works</CubeID> </Object> <Type>ProcessFull</Type> <DataSourcexsi:type="RelationalDataSource"> <ID>Adventure Works DW</ID> <Name>Adventure Works DW</Name> <ConnectionString> Provider=SQLNCLI10.1;Data Source=lpc:johnsi5\kj; Integrated Security=SSPI;Initial Catalog=AdventureWorksDW2008R2; </ConnectionString> <ImpersonationInfo> <ImpersonationMode>ImpersonateCurrentUser</ImpersonationMode> </ImpersonationInfo> <Timeout>PT0S</Timeout> </DataSource> </Process> </Parallel> </Batch>

ThefirstcommandusingTCP/IPmaxedoutat112,000rowspersecond.Becausethedatasourceforthe cubewasonthesamemachineasAnalysisServices,wewereabletousesharedmemoryinthesecond processingcommandandgetmuchbetterthroughput:180,000maxrows/sec.

127


Figure 5 - Compa 53 aring rows/sec throu ughput

3.1.6.2 High-Spe Netwo eed orking Fea atures

WindowsServer2008R2hasnume erousimprove ementsinnet tworkvirtuali izationsuppo ortthatenable dnetworkingsupport.Win ndowsServer2008R2hasalsoenhance edthesuppor rtofjumbo enhanced packetsandTCPoffloa adingthatwasintroducedinWindowsServer2008.AdditionallyVirtualMachine Queue(VMQ)supportwasaddedto oallownetworkroutinganddatacopy yprocessingtobeoffloade edto lNIC.Thesefeatureswere eintroducedt totakeadvan ntageofthec capabilitiesfo oundinthe10 0GbE aphysical serverNIC Cs. 3.1.6.2.1 TCP Chim mney workingtechnologythattra ansfersTCP/IPprotocolpr rocessingfrom mtheCPUtoa TCPChimneyisanetw adapterdurin ngthenetwor rkdatatransf fer.Therehav vebeensome eissueswithenablingTCP networka Chimneyinthepast,andforthatre easonmanyp peoplerecommendedturn ningitoff.The etechnologyhas andmanyoft theissuesrep portedweres specifictothe eNICmanufa acturer.Appli icationsthath have matureda longlived dconnections stransferringalotofdatabenefitthem mostfromthe eTCPChimne eyfeature. ProcessingAnalysisServicesdatafr romaremote eserverfallsintothiscateg gory,sowere ecommendth hat esurethatTC CPChimneyisenabledandconfiguredc correctly.TCP PChimneycan nbeenabledand youmake disabledintheoperati ingsysteman ndintheadva ancedproper rtiesofthene etworkadapte er. Toenable eTCPChimneyyouneedto operformthe efollowing: 128

1. EnableTCPchimneyintheoperatingsystemusingthenetshcommands. 2. EnsurethephysicalnetworkadaptersupportsTCPChimneyoffload,andthenenableit fortheadapterinthenetworkdriver. TCPchimneyhasthreemodesofoperation:Automatic(newinWindowsServer2008R2),Enabled,and Disabled.ThedefaultmodeinWindowsServer2008R2,Automatic,checkstomakesurethe connectionsconsideredforoffloadingmeetthefollowingcriteria: 10GbpsEthernetNICinstalledandconnectionestablishedthrough10GbEadapter Meanroundtriplinklatencyislessthan20milliseconds Connectionhasexchangedatleast130KBofdata

TodeterminewhetherTCPChimneyisenabled,runthefollowingfromanelevatedcommandprompt. netshinttcpshowglobal Intheresults,checktheChimneyOffloadStatesetting. TCPGlobalParameters ReceiveSideScalingState ChimneyOffloadState NetDMAState DirectCacheAccess(DCA) ReceiveWindowAutoTuningLevel AddOnCongestionControlProvider ECNCapability RFC1323Timestamps Inthisexample,theresultsshowthatTCPChimneyoffloadingissettoautomatic.Thismeansthatitis enabledaslongastherequirementsmentionedearlieraremet. AfteryouverifythatTCPChimneyisenabledattheoperatingsystemlevel,checktheNICsettingsin devicemanager: 1. Gotostart|runandtypedevmgmt.msc. 2. InDeviceManager,expandNetworkAdapters,andrightclickthenameofthephysicalNIC adapter,andthenclickProperties. 3. OntheAdvancedtab,underProperty,clickTCPChimneyOffloadorTCPConnectionOffload, andthenunderValue,verifythatEnabledisdisplayed.YoumayneedtodothisforbothIPv4 andIPv6.NotethatTCPChecksumOffloadisnotthesameasTCPChimneyOffload. IfthesystemhasIPsecenabled,noTCPconnectionsareoffloaded,andTCPChimneyprovidesno benefit.TherearenumerousdifferentconfigurationoptionsforTCPChimneyoffloadingthatouroutside thescopeofthisbook.Seethereferencessectionforadeepertreatment. 129

:enabled :automatic :enabled :disabled :normal :none :disabled :disabled

References: WindowsServer2008R2NetworkingDeploymentGuide:DeployingHighSpeedNetworking Featureshttp://download.microsoft.com/download/8/E/D/8EDE21BC0E3B4E14AAEA 9E2B03917A09/HSN_Deployment_Guide.doc WindowsServer2008R2NetworkingDeploymentGuide http://download.microsoft.com/download/8/E/D/8EDE21BC0E3B4E14AAEA 9E2B03917A09/HSN_Deployment_Guide.doc

3.1.6.2.2 Jumbo Frames JumboframesareEthernetframeswithmorethan1,500andupto9,000bytesofpayload.Jumbo frameshavebeenshowntoyieldasignificantthroughputimprovementduringAnalysisServices processing,specificallyProcessData.NetworkthroughputisincreasedwhileCPUusageisminimized. Jumboframesareonlyavailableongigabitnetworks,andalldevicesinthenetworkpathmustsupport them(switches,routers,NICs,andsoon). MostdefaultEthernetsetupsareconfiguredtohaveMTUsizesofupto1,500bytes.Jumboframes allowyoutogoupto9,000bytesinasingletransfer.Toenablejumboframes: 1. Configureallrouterstosupportjumboframes. 2. ConfiguretheNICsontheAnalysisServicesmachineandthedatasourcemachineto supportjumboframes. a) ClicktheStartbutton,pointtoRun,andthentypencpa.cpl. b) Rightclickyournetworkconnection,andthenclickProperties. c) ClickConfigure,andthenclicktheAdvancedtab.UnderProperty,clickJumbo Frame,andthenunderValue,changethevaluefromDisabledto9kbMTUor 9014,dependingontheNIC. d) ClickOKonallthedialogboxes.Afteryoumakethechange,theNICloses networkconnectivityforafewsecondsandyoushouldreboot. 3. Afteryouconfigurejumboframes,youcaneasilytestthechangebypingingtheserver withalargetransfer: Ping<servername>fl9000 YoushouldonlymeasureonenetworkpacketperpingrequestinNetworkMonitor.

3.1.6.3 Hyper-V and Networking

IfyouarerunningAnalysisServicesandSQLServerinaHyperVvirtualmachine,thereareafewthings youshouldbeawareofspecifictonetworking. WhenyouenableHyperVandcreateanexternalvirtualnetwork,boththeguestandthehostmachine gothroughavirtualnetworkadaptertoconnecttothephysicalnetwork.Alltrafficgoesthroughyour virtualnetworkadapter,andthereisadditionallatencyoverheadassociatedwiththis. 130

3.1.6.4 Increase Network Packet Size

Underthepropertiesofyourdatasource,increasingthenetworkpacketsizeforSQLServerminimizes theprotocoloverheadrequiretobuildmany,smallpackages.ThedefaultvalueforSQLServer2008is 4096.Withadatawarehouseload,apacketsizeof32K(inSQLServer,thismeansassigningthevalue 32767)canbenefitprocessing.DontchangethevalueinSQLServerusingsp_configure;instead overrideitinyourdatasource.ThiscanbesetwhetheryouareusingTCP/IPorSharedMemory.

Figure 54 Tuning network packet size

3.1.6.5 Using Multiple NICs

Ifyouarehittinganetworkbottleneckwhenyouprocessyourcube,youcanaddadditionalNICsto increasethroughput.TherearetwobasicconfigurationsforusingmultipleNICsonaserver.Thefirstand defaultoptionistousetheNICsseparately.ThesecondoptionistouseNICteaming. MultipleNICscanbeusedindividuallytoconcurrentlyrunmanymultipartitionprocessingcommands. EachpartitioninAnalysisServicescanrefertoadifferentdatasource,anditisthisfeatureyoumakeuse of.TousemultipleNICs,followthesesteps: 1. SetupNICswithdifferentIPnumbers. 2. Addnewdatasources. 3. Changeyourpartitionsetuptoreferencethenewdatasources.

131

First,setupmultipleNICsinthesourceandthetarget,eachwithitsownIPnumber.MatcheachNICon thecubeserverwithacorrespondingNIContherelationaldatabase.Setupthesubnetmasksoonlythe desiredNIConthesourcecanbereachedfromitspartneronthecubeserver.Ifyouarelimitedin bandwidthontheswitchesorwanttocreateaverysimplesetup,youcanusethistechniquewitha crossovercable,whichwehavedonesuccessfully Second,addadatasourceforeachNIConthedatasource,andsetupeachdatasourcetopointtoits ownsourceNIC. Third,configurepartitionsinthecubesothattheyarebalancedacrossalldatasources.Forexample,if youhave16partitionsand4datasourcesperNICpoint4partitionstoeachdatasource. Schematically,thesetuplookslikethis.

Figure 55 - Using multiple NICs for processing

NICteaming:AnotheroptionistoteammultipleNICssotheyappearasoneNICtoWindowsServer.Itis difficulttomakespecificrecommendationsaroundusingteaming,becauseperformanceisspecificto thehardwareanddriversusedforteaming.TodeterminewhetherNICteamingwouldbebeneficialto yourprocessingworkload,testwithitbothenabledanddisabled,andmeasureyourperformance resultswithbothsettings.

3.1.7 Disabling Flight Recorder

FlightRecorderprovidesamechanismtorecordAnalysisServicesserveractivityintoashorttermlog. FlightRecorderprovidesagreatdealofbenefitwhenyouaretryingtotroubleshootspecificquerying andprocessingproblemsbyloggingsnapshotsofcommonDMVintothelogfile.However,itintroduces anamountofI/Ooverheardontheserveritself. IfyouareinaproductionenvironmentandyouarealreadymonitoringtheAnalysisServicesinstance usingtheinformationinPart2,youdonotneedFlightRecordercapabilitiesandyoucandisableits 132

loggingandremovetheI/Ooverhead.TheserverpropertythatcontrolswhetherFlightRecorderis enabledistheLog\FlightRecorder\Enabledproperty.Bydefault,thispropertyissettotrue.

3.2 Monitoring and Tuning the Server

Aspartofhealthyoperationsmanagement,youmustcollectdatathatallowsbothreactiveand proactivebehaviorsthatincreasesystemstability,performance,andintegrity.Thetemptationisto collectalotofdatainthebeliefthatwithmoreknowledgecomesbetterdecisions.Thisisnotalways thecase,becauseyoumayeitheroverloadtheserverwithdatacollectionoverheadorcollectdata pointsthatyoucannottakeanyactionon. Hence,foreverydatapointyoucollect,youshouldhaveanideaaboutwhatthatdatacollectionhelps youachieve.Part2ofthisbookprovidesyouwiththeknowledgeyouneedtointerpretthe measurementsyouconfigureandhowtotakeactiononthem.Foreaseofreference,thissection summarizesthedatacollectionrequired.Youcanuseitasachecklistfordatacollection. Thetoolusedtocollectthedatadoesnotmattermuch;itisoftenaquestionofpreferenceand operationalprocedures.Whatthissectionprovidesisthesourceofthedatapointshowyouaggregate andcollectthemwilldependonyourorganization.

3.2.1 Performance Monitor

OnanAnalysisServicesserver,theminimalcollectionofdataisthis. Performanceobject LogicalDisk Process MSOLAP:Memory MSOLAP:Connection MSOLAP:Threads Memory Counter Description All/All Collectsdataaboutdiskloadandutilization. PrivateBytesmsmdsrv.exe ThememoryconsumedbytheAnalysis Servicesprocess. MemoryUsageKb AlternativetoProcess. CurrentConnections Measuresthenumberofopenconnectionsto gaugeconcurrency. * Allowstuningofthreadpools. Estimatesthesizeofthefilesystemcache. CacheBytes StandbyCacheNormal PriorityBytes StandbyCacheCoreBytes StandbyCacheReserve Bytes PageFaults/sec Testsforexcessivepagingoftheserver. AvailableBytes Usedtotunememorysettings. tempfilebyteswritten Measuresthespillfromprocessingoperations. Ideally,cubeandhardwareshouldbebalanced tomakethis0. CurrentPartitions Measuresspeedandconcurrencyofprocess Rows/sec index.

Memory Memory MSOLAP:Proc Aggregations MSOLAP:Proc Indexes 133

MSOLAP:Processing MSOLAP:MDX

Rowsread/sec Rowswritten/sec *

System System NetworkInterface TCPv4andTCPv6

FileReadBytes/sec FileReadOperations/sec BytesReceived/sec BytesSent/sec Segments/sec Segments Retransmitted/sec

Measuresspeedofrelationreadandefficiency ofmergebuffers. Usedbycubetunerstodeterminewhether calculationscriptsorMDXqueriescanbe improved. Measurebytesreadfromthefilesystem cache. MeasuresIOPSfromfilesystemcache ContainsthecapacityplantofollowifNIC speedslows.Seesection2.6.5. Discoversunstableconnections.

Inmostcases,youcancollectthisinformationevery15secondswithoutcausingmeasurableimpactto thesystem. IfyouareusingSystemsCenterOperationsManager(SCOM)tomonitorservers,itisagoodideatoadd thisdatacollectiontoyourmonitoring.Alternatively,youcanusethebuiltinperformancemonitorin WindowsServer,butthatwillrequirethatyouharvestthecountersfromtheserversyourself.

3.2.2 Dynamic Management Views

StartingwithSQLServer2008AnalysisServices,thereisanewsetofmonitoringtoolsavailabletoyou: dynamicmanagementviews(DMVs).Theseviewsprovideinformationabouttheoperationsofthe servicethatyoucannotfindinSQLServerProfilerorPerformanceMonitor. ThefollowingtableliststhemostusefulDMVscollectonaregularbasisfromtheserver. DMV $system.discover_commands $system.discover_sessions $system.discover_connections $system.discover_memoryusage $system.discover_locks Description Listsallcurrentlyrunningcommandsontheserver,including commandsrunbytheserveritself. Listallsessionsontheserver.Itisusedtomapcommandsand connectionstogether. Listscurrentopenconnections. Listsallmemoryallocationsintheserver. Listscurrentlyheldandrequestedlocks.

Yourcapturerateofthedatadependsonthesystemyouarerunningandhowquicklyyouroperations teamresponsetoevents.$system.discover_locksand$system.discover_memoryusageareexpensive DMVstoqueryandgatheringthemtoooftenconsumessignificantCPUresourcesontheserver. CapturingtheDMVsonceeveryfewminutesisprobablyenoughformostoperationsmanagement purposes.Ifyouchoosetocapturethemmoreoften,measuretheimpactonyourserver,whichwill dependontheconcurrencyofexecutingsessions,memorysizes,andcubedesign. 134

DependingonhowoftenyouquerytheDMV,youcangeneratealotofdata.Itisoftenagoodideato keeptherecentdataatahighgranularityandaggregateolderdata.UsingatoollikeMicrosoft StreamInsightenablesyoutoperformsuchhistoricalaggregationinrealtime. CapturingtheDMVdataenablesyoutomonitortheprogressofqueriesandalertyoutoheavyresource consumersearly.HerearesomeexamplesofissuesyoucanidentifywhenyouuseDMVs: Queriesthatconsumealotofmemory Queriesthathavebeenblockedforalongtime Locksthatareheldforalongtime Sessionsthathaverunforalongtime,orconsumedalotofI/O Connectionsthataretransmittingalargeamountofdataoverthenetwork

OnewaytocollectthisdataistousealinkedserverfromtheSQLServerDatabaseEnginetoAnalysis ServicesandusetheDatabaseEngineasthestorageforthedatayoucollect.Youmayalsobeableto configureyourfavoritemonitoringtooltodothesame.Unfortunately,AnalysisServicesdoesnotship withafullyautomatedtoolforthisdatacollection. Thefollowingexamplecollectionscriptscangetyoustartedwithabasicdatacollectionframework:

select SESSION_SPID /* Key in commands */ /* Monitor for large values and large different with below */ , COMMAND_ELAPSED_TIME_MS , COMMAND_CPU_TIME_MS /* Monitor for large values */ , COMMAND_READS /* Monitor for large values */ , COMMAND_WRITES /* Monitor for large values */ , COMMAND_TEXT /* Track any problems to the query */ from $system.discover_commands select SESSION_SPID /* Join to SPID in Sessions */ , SESSION_CONNECTION_ID /* Join to connection_id in Connections */ , SESSION_USER_NAME /* Finds the authenticated user */ , SESSION_CURRENT_DATABASE from $system.discover_sessions select CONNECTION_ID /* Key in connections */ , CONNECTION_HOST_NAME /* Find the machine the user is coming from */ , CONNECTION_ELAPSED_TIME_MS , CONNECTION_IDLE_TIME_MS /* Find clients not closing connections */ /* Monitor the below for large values */ , CONNECTION_DATA_BYTES_SENT , CONNECTION_DATA_BYTES_RECEIVED from $system.discover_connections select SPID , MemoryName , MemoryAllocated , MemoryUsed , Shrinkable , ObjectParentPath , ObjectId from $system.discover_memoryusage

135

where SPID <> 0

References: MSDNlibraryreferenceonAnalysisServicesDMVs:http://msdn.microsoft.com/en us/library/ee301466.aspx CodePlexResMontool,whichcapturesdetailedinformationaboutcubes: http://sqlsrvanalysissrvcs.codeplex.com/wikipage?title=ResMon%20Cube%20Documentation

3.2.3 Profiler Traces

AnalysisServicesexposesalargenumberofeventsthroughtheprofilerAPI.Collectingeverysingleone duringnormaloperationsnotonlygenerateshighdatacollectionvolumes,italsoconsumessignificant CPUandnetworktraffic.Whileitispossibletodigintogreatdetailoftheserver,youshouldmeasureto impactonthesystemwhilerunningthetrace.Ifeveryeventistraced,alotoftraceinformationis generated:Makesureyoueithercleanuphistoricalrecordsorhaveenoughspacetoholdthetrace data.VerydetailedSQLServerProfilertracesarebestreservedforsituationswhereyouneedtodo diagnosticsontheserverorwhereyouhaveenoughCPUandstoragecapacitytocollectlotsofdetails. References:

ASTraceatooltocollectprofilertracesintoSQLServerforfurtheranalysis: http://msftasprodsamples.codeplex.com/wikipage?title=SS2005%21Readme%20for%20ASTrace %20Utility%20Sample&referringTitle=Home

3.2.4 Data Collection Checklist

Youcanusethefollowingchecklisttomakesureyouhavecoveredthebasicdatacollectionneedsofthe server: WindowsServerspecificPerformanceMonitorcounters AnalysisServicesspecificPerformanceMonitorcounters DataManagementViewscollected: o $system.discover_commands o $system.discover_sessions o $system.discover_connections o $system.discover_memoryusage o $system.discover_locks

3.2.5 Monitoring and Configuring Memory

TodiscoverthebestmemoryconfigurationforyourAnalysisServicesinstance,youneedtocollectsome dataaboutthetypicalusageofthesystem.

136

Firstofall,youshouldstartuptheserverwithAnalysisServicesdisabledandmakesurethatallthe otherservicesyouneedfornormaloperationsinastatethatistypicalforyourstandardserver configuration.Whattypicalisvariesbyenvironmentyouarelookingtomeasurehowmuchmemory willbeavailabletoAnalysisServicesaftertheoperatingsystemandotherservices(suchasvirus scannersandmonitoringtools)havetakentheirshare.NotedownthevalueofthePerformance MonitorcounterMemory/AvailableBytes. Secondly,startAnalysisServicesandrunatypicaluserworkloadontheserver.Youcanforexampleuse queriesfromyourtestharness.Forthistest,alsomakesurePreAllocateissetto0.Whiletheworkload isrunning,runthefollowingquery.

SELECT * FROM $SYSTEM.DISCOVER_MEMORYUSAGE

Ifyouhaveanylongrunning,highmemoryconsumingqueries,youshouldalsomeasurehowmuch memorytheyconsumewhiletheyexecute. CopythedataintoanExcelspreadsheetforfurtheranalysis.YoucanalsousetheCodePlexproject ResMoncube(seereferencesection)toperiodicallylogsnapshotsofthisDMVandbrowsememory usagetrendssummarizedinacube. Withthedatacollectedintheprevioussection,youcandefinesomevaluesneededtosetthememory: [PhysicalRAM]=Thetotalphysicalmemoryonthebox [AvailableRAM]=ThevalueofthePerformanceMonitorcounterMemory/AvailableBytesasnoted downearlier. [NonShrinkableMemory]=ThesumoftheMemoryUsedcolumnfrom $SYSTEM.DISCOVER_MEMORYUSAGEwherethecolumnShrinkableisFalse. [ValuableObjects]=Thesumofallobjectsthatyouwanttoreservememoryforfrom $SYSTEM.DISCOVER_MEMORYUSAGEwherecolumnShrinkableisTrue.Forexample,youwillmost likelywanttoreservememoryforalldimensionsandattributes.Ifyouhaveasmallcube,thisvaluemay simplybeeverythingthatismarkedasshrinkable. [WorstQueriesMemory]=Theamountofmemoryusedbytheallthemostdemandingqueriesyou expectwillrunconcurrently.YoucanusetheDMV$system.discover_memoryusagetomeasurethison aknownworkload. WiththeprecedingvaluesyoucancalculatethememoryconfigurationforAnalysisServices: LowMemoryLimit=[NonShrinkableMemory]+[ValuableObjects]/[TotalRAM]Butkeepagapofat least20percentbetweenthisvalueandHighMemoryLimittoallowthememorycleanerstorelease memoryatagoodrate.

137

HighMemoryLimit=([AvailableRAM][WorstQueriesMemory])/[TotalRAM] HardMemoryLimit=[AvailableRAM]/[TotalRAM] LimitSystemFileCacheMB=[AvailableRAM]LowMemoryLimit Notethatitmaynotalwaysbepossibletomeasureallthecomponentsthatmakeuptheseequations. Insuchcase,yourbestbetistoguesstimatethem.TheideabehindthismethodisthatAnalysisServices willalwaysholdontoenoughmemorytokeepthevaluableobjectsinmemory.Whatisvaluable dependsonyourparticularsetupandqueryset.Therestoftheavailablememoryissharedbetweenthe filesystemcacheandtheAnalysisServicescaches;ifyouusetheoperatingsystemsmemoryusage optimizations,theidealbalancebetweenAnalysisServicesandthefilesystemcacheisadjusted dynamically. IfAnalysisServicescoexistswithotherservicesonthemachine,taketheirmaximummemory consumptionintoconsideration.Whenyoucalculate[AvailableRAM],subtractthemaximummemory usebyotherlargememoryconsumers,suchastheSQLServerrelationalengine.Also,makesurethose othermemoryconsumershavetheirmaximummemorysettingsadjustedtoallowspaceforAnalysis Services. Thefollowingdiagramillustratesthedifferentusesofmemory.

Figure 56 - Memory Settings

138

References: CodePlexResMonproject http://sqlsrvanalysissrvcs.codeplex.com/wikipage?title=ResMon%20Cube%20Documentation

3.2.6 Monitoring and Tuning I/O and File System Caches

TotunetheAnalysisServicesI/Osubsystem,itishelpfultounderstandhowauserqueryistranslated intoI/Orequests.ThefollowingdiagramillustratesthecachingandI/OarchitectureofAnalysisServices.

Figure 57 - I/O stack

FromtheperspectiveoftheI/Osystem,thestorageenginerequestsdatafromanoffsetinafile.The buildingblocksofthestorageusedbyAnalysisServicesarethefilesusedtostoredimensionandfact data.Dimensiondataistypicallycachedbythestorageengine.Hence,themostfrequentlyrequested filesfromthestoragesystemaremeasuregroupdata,andtheyhavethefollowingfilenames: *.fact.data,*.aggs.rigid.dataand*.agg.flex.data.BecauseAnalysisServicesusesbufferedI/O, frequentlyrequestedblocksinfilesaretypicallyretainedbytheNTFSfilesystemcache.Notethatthis meansthatthememoryusedforthefilesystemcachingofdatacomesfromoutsidetheAnalysis Servicesworkingset. WhenablockfromafileisrequestedbyAnalysisServices,thepathtakenbytheoperatingsystem dependsonthestateofthecache. 139


Figure 58 - Accessing a file in the NTFS cache

Intheprecedingillustration,AnalysisServicesexecutestheReadFileExcalltotheWindowsAPI. Iftheblockisnotinthecache,anI/Orequestisissued(2),thefileisputinthecache(3),and thedataisreturnedtoAnalysisServices(5). Ifheblockisalreadyinthecache(3),itissimplyreturnedtoAnalysisServices(5). Ifablockisnotfrequentlyaccessedorifthereismemorypressure,theNTFSfilesystemcache maychoosetoplacetheblockintheStandbyCache(4)andeventuallyremovethefilefrom memory.Ifthefileisrequestedbeforeitisremoved,itisbroughtbackintotheSystemCache (3)andthenreturnedtoAnalysisServices(5). o NotethatinWindowsServer2003,thefileissimplyremovedfromthesystemcache, bypassingthestandbycache.

3.2.6.1 System Cache vs. Physical I/O

BecauseAnalysisServicesusestheNTFScache,noteveryI/OrequestreachestheI/Osubsystem.This meansthatevenwithaclearstorageengineandformulaenginecache,queryperformancewillstillvary dependingonthestateofthefilesystemcache.Intheworstcase,aquerywillrunonacleanfilesystem cacheandeveryI/Orequestwillhitthephysicaldisk.Inthebestcase,everyI/Orequestwillbeserved bythememorycache.Thisdependsontheamountofdatarequestedbythequery. ItisusefultoknowtheratiobetweentheI/OissuestothediskandtheI/Oservedbythefilesystem cache.Thishelpsyoucapacityplanforgrowingcube.Smallcubes,lessthanthesizeoftheserver memory,aretypicallyservedmostlyfromthefilesystemcache.However,ascubesgrowlargerand 140

exceedthefilesystemcachesize,youwilleventuallyhavetoassistcubeperformancewithafastdisk system.Measuringthecurrentcachehitratiohelpsshedlightonthis. TomeasuretheI/Oservedbythefilesystemcache,usePerformanceMonitortomeasureSystem:File ReadBytes/secandSystemFileReadOperations/sec.Thesecounterswillgiveyouanestimateofthe numberofI/Ooperationsandfilebytesthatareservedfrommemory.BycomparingthiswithLogical Disk/Diskreads/secyoucanmeasuretheratiobetweenmemorybufferedandphysicalI/O.Asthe amountofdatainthecubegrows,youwilloftenseethattheamountofdiskaccessbeginstogrowin proportiontomemoryaccess.Byextrapolatingfromhistoricaltrends,youcanthendecidewhatthe bestratiobetweenIOPSandRAMsizesshouldbegoingforward. TomeasurethetotalmemoryconsumptionofAnalysisServices,youwillalsohavetotakethefile systemcacheintoaccount.Thetotalmemoryusedbyanalysisservicesisthesumof: Processmsmdsrv.exe/PrivateBytesThememoryusedbyAnalysisServicesitself MemoryCacheBytesThefilescurrentlyliveincache MemoryStandbyCacheNormalPriorityBytesThefilesthatcanbefreedfromthecache

However,notethatthememorycountersalsomeasureotherfilescachesintheNTFSfilesystemcache. IfyouarerunningAnalysisServicestogetherwithotherservicesonthemachine,thefilesystemcounter maynotaccuratelyreflectthecachesusedbyAnalysisServices.

3.2.6.2 TempDir Folder

Whenmemoryisscarceontheserver,AnalysisServicesspillsoperationstodisk.Examplesofsuch operationare: Dimensionprocessing o ByTableprocessingcommandsthatdontfitmemory o Processingoflargedimensionwherethehashtablescreatedexceedavailablememory Aggregationprocessing ROLAPdimensionattributestores

Dimensionprocessing:Tooptimizedimensionprocessing,werecommendthatyouusethetechniques describedlaterinthisdocumenttoavoidspillsandspeedupprocessing.ByTableshouldgenerallyonly beusedifyoucankeepthedimensiondatainmemoryduringprocessing. Aggregationprocessing:Duringcubeprocessing,aggregationbuffersdescribedinconfigurationsection determinetheamountofmemorythatisavailabletobuildaggregationsforagivenpartition.Ifthe aggregationbuffersaretoosmall,AnalysisServicessupplementstheaggregationbufferswith temporaryfiles.Tomonitoranytemporaryfilesusedduringaggregation,reviewMSOLAP:Proc Aggregations\Tempfilebyteswritten/sec.Youshouldtrytodesignyourcubeinsuchawaythat memoryspillstotempfilesdoesnotoccur,thatis,keepthetempbyteswrittenatzero.Thereare severaltechniquesavailabletoavoidmemoryspillsduringaggregationprocessing:

141

Onlycreateaggregationsthataresmallenoughtofitinmemoryduringprocessing. Processfewerpartitionsinparallel. AddmorememorytothemachineorallocatingmorememorytoAnalysisServices.

ItisgenerallyagoodideatosplitProcessDataandProcessIndexoperationsintotwodifferent processingjobs.ProcessDatatypicallyconsumeslessmemorythanProcessIndexandyourunmany ProcessDataoperationsinparallel.DuringProcessIndex,youcanthendecreaseconcurrencyifyouare shortonmemory,toavoidthediskspill. ROLAPdimensions:Ingeneral,youshouldtrytoavoidROLAPdimensions;MOLAPstoresaremuch fasterfordimensionaccess.However,therequirementforROLAPdimensionsisoftendrivenbyalackof sufficientmemorytoholdtheattributestoresrequiresfordrillthroughactions,whichreturnsdatausing adegenerateROLAPdimension.Ifthisisyourscenario,youmaynotbeabletoavoidspillstothe TempDirfolder. Ifyoucannotdesignthecubetofitalloperationsinmemory,spillingtoTempDirmaybeyouronly option.Ifthatisthecase,werecommendthatyouplacetheTempDirfolderonafastLUN.Youcan eitheruseaLUNbackedbycaches(forexampleinaSAN)oronethatsitsonfaststorageforexample, NANDdevices.

3.2.7 Monitoring the Network

Onewaytoeasilymonitornetworkthroughputisthroughaperformancemonitortrace.Windows PerformanceMonitorrequeststhecurrentvalueofperformancecountersatspecifiedtimeintervals andlogsthemintoatracelog(.blgor.csv). Thefollowingcounterswillhelpyouisolateanyproblemsinthenetworklayer. Processing:Rowsread/secthisistherateofrowsreadfromyourdatasource.Thisisoneofthemost importantcounterstomonitorwhenyoumeasureyourthroughputfromAnalysisServicestoyour relationaldatasource.Whenyoumonitorthiscounter,youwillwanttoviewthetraceusingalinechart, asitgivesyouabetterideaofyourthroughputrelativetotime.Itisreasonabletoexpecttensof thousandsofrowspersecondpernetworkconnectiontoaSQLServerdatasource.Thirdpartysources mayexperiencesignificantlyslowerthroughput. Bytesreceived/secandBytessent/secThesetwocountersenableyoutomeasurehowfaryouare fromthetheoreticalNICspeeds.ItallowscapacityplanningforfasterNIC. TCPv4andTCPv6Segments/secandSegmentsretransmitted/secThesecountersenableyouto discoverunstablenetworkconnections.Theratiobetweensegmentsretransmittedandsegmentsfor bothTCPv4andTCPv6shouldnotexceed34percentonastablenetwork.

142

3.3 Security and Auditing

Cubesoftencontaintheveryheartofyourbusinessdatathesourceofyourdecisionmaking.That meansthatyouhavetoconsiderpotentialattackvectorsthatamalicioususerorintrudercoulduseto acquireaccesstoyourdata.Thefollowingtableprovidesanoverviewofthepotentialvulnerabilitiesand thecountermeasuresyoucantake.Notethatmostenvironmentswillnotneedallofthese countermeasuresitdependsonthedatasecurityandontheattackvectorsthatarepossibleonyour network. Attackvector Otherserviceslisteningontheserver SniffTCP/IPpacketstoclient SniffTCP/IPpacketsduringprocessing Stealphysicalmediacontainingcubes Compromisetheserviceaccount Countermeasure FirewallallportsotherthanthoseusedbyAnalysis Services ConfigureIPsecorSSLencryption ConfigureSQLServerProtocolEncryption Encryptfilesystemusedtostorecubes Configureminimumprivilegestotheservice account Requireastrongpassword Securefilesystemwithminimalprivileges

Accessthefilesystemasaloggedinuser

Afulltreatmentofsecurityconfigurationisoutsidethescopeofthisbook,butthefollowingsections providereferencesthatserveasastartingpointforfurtherreadingandgiveyouanoverviewofthe optionsavailabletoyou.

3.3.1 Firewalling the Server

Bydefault,AnalysisServicescommunicateswithclientsonport2383.Ifyouwanttochangethis,access thepropertiesoftheserver.

143


Figure 59 - Changing the port used for Analysis Services

BearinmindthatyouwillneedtoopentheportassignedhereinyourfirewallforTCP/IPtraffic.Ifyou leavethedefaultvalueof0,AnalysisServiceswilluseport2382. Ifyouareusingnamedinstances,yourclientapplicationmayalsoneedtoaccesstheSQLServer browserservice.Thebrowserserviceallowsclientstoresolveinstancenamestoportnumbers,andit listensonTCPport2382.NotethatitispossibletoconfiguretheconnectionstringforAnalysisServices insuchawaythatyouwillnotneedthebrowserserviceportopen.Toconnectdirectlytotheportthat AnalysisServicesislisteningon,usethisformat[Servername}:[Port].Forexample,toconnecttoan instancelisteningonport2384onserverMyServer,useMyServer:2384. AnalysisServicescanbesetuptouseHTTPtocommunicatewithclients.Ifyouchoosethisoption, followtheguidelinesforconfiguringMicrosoftInternetInformationServices.Inthiscase,youwill typicallyneedtoopeneitherport80orport443.

144

References: Howto:ConfigureWindowsFirewallforAnalysisServicesAccess http://msdn.microsoft.com/enus/library/ms174937.aspx ResolvingCommonConnectivityIssuesinSQLServer2005AnalysisServicesConnectivity Scenarioshttp://msdn.microsoft.com/enus/library/cc917670.aspx o AlsoappliestoSQLServer2008AnalysisServicesandSQLServer2008R2Analysis Services ConfiguringHTTPAccesstoSQLServer2005AnalysisServicesonMicrosoftWindows2003 http://technet.microsoft.com/enus/library/cc917711.aspx o AlsoappliestoSQLServer2008AnalysisServices,SQLServer2008R2AnalysisServices, WindowsServer2008,andWindowsServer2008R2 AnalysisServices2005protocolXMLAoverTCP/IP http://www.mosha.com/msolap/articles/as2005_protocol.htm

3.3.2 Encrypting Transmissions on the Network

AnalysisServicescommunicatesinacompressedandencryptedformatonthenetwork.Youmaystill wanttouseIPsectorestrictwhichmachinescanlisteninonthenetworktraffic,butthecommunication channelitselfisalreadyencryptedbyAnalysisServices. IfyouhaveconfiguredAnalysisServicestocommunicateoverHTTP,thecommunicationcanbesecured usingtheSSLprotocol.However,beawarethatyoumayhavetoacquireacertificatetouseSSL encryptionoverpublicnetworks.Also,notethatSSLencryptionnormallyusesport443,notport80,to communicate.Thisdifferencemayrequirechangesinthefirewallconfiguration.UsingtheHTTP protocolalsoallowsyoutorunsecuredlinestopartiesoutsidethecorporatenetworkforexample,in anextranetsetup. Dependingonyournetworkconfiguration,youmayalsobeconcernedaboutnetworkpacketsgetting sniffedduringcubeprocessing.Toavoidthis,youagainhavetoencrypttraffic.Again,youcanuseIPsec toachievethis.AnotheroptionistouseprotocolencryptioninSQLServer,whichisdescribedinthe references. References: HowtoconfigureSQLServer2008AnalysisServicesandSQLServer2005AnalysisServicesto useKerberosauthenticationhttp://support.microsoft.com/kb/917409 WindowsFirewallwithAdvancedSecurity:StepbyStepGuide:DeployingWindowsFirewalland IPsecPolicieshttp://www.microsoft.com/downloads/en/details.aspx?FamilyID=0b937897 ce39498ebb37751c00f197d9&displaylang=en HowToConfigureIPsecTunnelinginWindowsServer2003 http://support.microsoft.com/kb/816514 HowtoenableSSLencryptionforaninstanceofSQLServerbyusingMicrosoftManagement Consolehttp://support.microsoft.com/kb/316898

145

3.3.3 Encrypting Cube Files

Somesecuritystandardsrequireyoutosecurethemediathatthedataisstoredon,topreventintruders fromphysicallystealingthedata.IfyouhaveMOLAPcubesontheserver,mediasecuritymaybea concerntoyou.BecauseAnalysisServices,unliketherelationalengine,doesnotshipwithnative encryptionofMOLAPcubedata,youmustrelyonencryptionoutsidetheengine.YoucanuseWindows FileSystemencryption(WindowsServer2003)orBitLocker(onWindowsServer2008andWindows Server2008R2)toencryptthedriveusedtostorecubes.Becareful,though;encryptingMOLAPdata canhaveabigimpactonperformance,especiallyduringprocessingandschemamodificationofthe cube.Wehaveseenperformancedropsofupto50percentwhenprocessingdimensionsonencrypted drivesthoughlesssoforfactprocessing.Weightherequirementtoencryptdatacarefullyagainstthe desiredperformancecharacteristicsofthecube. References: EncryptingFileSysteminWindowsXPandWindowsServer2003 http://technet.microsoft.com/enus/library/bb457065.aspx BitLockerDriveEncryptionhttp://technet.microsoft.com/en us/library/cc731549%28WS.10%29.aspx

3.3.4 Securing the Service Account and Segregation of Duties

TosecuretheserviceaccountforAnalysisServices,itisusefultofirstunderstandthedifferentsecurity rolesthatexistataserverlevel. Theserviceaccountistheaccountthatrunsthemsmdsrv.exefile.Itisconfiguredduringinstallationand canbechangedusingSQLServerConfigurationManager.Theserviceaccountistheaccountusedto accessallfilesrequiredbyAnalysisServices,includingMOLAPstores.Foryourconvenience,alocal groupSQLServerMSASUser$<InstanceName>iscreatedthathastherightprivilegesonthebinaries requiredtorunAnalysisServices.Ifyouconfiguretheserviceaccountduringinstallation,theaccount willautomaticallybeaddedtothisgroup.IfyoulaterchoosetochangetheserviceaccountinSQLServer ConfigurationManager,youmustmanuallyupdatethegroupmembership. Theserveradministratorrolehasprivilegestochangeserversettings,andtocreate,backup,restore, anddeletedatabases.Membersofthisaccountcanalsocontrolsecurityonalldatabasesinthe instance.ItistheDBAroleoftheinstanceandalmostequivalenttothesysadminroleforSQLServer. MembershipintheserveradministratorroleisconfiguredinthepropertiesoftheserverinSQLServer ManagementStudio. Inasecureenvironment,youshouldrunAnalysisServicesunderadedicatedserviceaccount.Youcan configurethisduringinstallationoftheservice. Ifyourenvironmentrequiressegregationofdutiesbetweenthosewhoconfiguretheserviceaccount andthosewhoadministertheserver,youneedtomakesomechangestothemsmdsrv.inifile:

146

Bydefault,localadministratorsaremembersoftheserveradministratorrole.Toremovethis association,set<BuiltinAdminsAreServerAdmins>to0. Bydefault,theserviceaccountforAnalysisServicesisamemberoftheserveradministrator role.Removethisassociationbysetting<ServiceAccountIsServerAdmin>to0.

However,keepinmindthatalocaladministratorcouldstillaccessthemsmdsrv.inifileandalterthe changesyouhavemade,soyoushouldauditforthispossibility.

3.3.5 File System Permissions

Asmentionedearlier,theAnalysisServicesinstallerwillcreatetheWindowsNT groupSQLServerMSASUser$<InstanceName>andaddtheserviceaccounttothisgroup.Atinstalltime, thisgroupisalsograntedaccesstotheData,backup,log,andTempDirfolders. Ifyouatalatertimeaddmorefolderstotheinstancetoholddata,backups,orlogfiles,youwillneedto granttheSQLServerMSASUser$<InstanceName>groupreadandwriteaccesstothenewfolders.Ifyou movetheTempDirfolder,youwillalsoneedtoassignthegroupread/writepermissiontothenew location.NootherusersneedpermissionsonthesefoldersforAnalysisServicestooperate. IntheconfigurationoftheserveryouwillalsofindtheAllowedBrowsingFolderssetting.Thissetting,a pipeseparatedlistofdirectories,limitsthevisiblefoldersthatserveradministratorscanseewhenthey configurestoragelocationsAnalysisServicesdata.AllowedBrowsingFoldersisnotasecurityfeature, becauseserveradministratorcanchangethevaluesinittoreflectanythingthattheserviceaccountcan see.However,itdoesserveasaconvenientfiltertodisplayonlyasubsetofthefoldersthattheservice accountcanaccess.Notethatserveradministratorscannotdirectlyaccessthedatavisiblethroughthe AllowedBrowsingFolderssetting,buttheycanwritebackupsinthefolders,restoreformthefolders, moveTempDirthere,andsetthestoragelocationofdatabases,dimensions,andpartitionstothose folders.

3.4 High Availability and Disaster Recovery

HighavailabilityanddisasterrecoveryarenotrobustlyintegratedaspartofAnalysisServicesforlarge scalecubes.Inthissection,youlearnaboutthecombinationofmethodsyoucanusetoachievethese goalswithinanenterpriseenvironment. Toensuredisasterrecovery,usethebuiltinBackup/Restoremethod,AnalysisServicesSynch,or Robocopytoensurethatyouhavemultiplecopiesofthedatabase.Youcanachievehighavailabilityby deployingyourAnalysisServicesenvironmentontoclusteredservers.Alsonotethatbyusinga combinationofmultiplecopies,clustering,andscaleout(section7.4),youcanachievebothhigh availabilityanddisasterrecoveryforyourAnalysisServicesenvironment.Inalargescaleenvironment, thescaleoutmethodgenerallyprovidesthebestofusehardwareresourceswhilealsoprovidingboth backupandhighavailability.

147

3.4.1 Backup/Restore
Cubesaredatastructuresondiskandassuch,theycontaininformationthatyoumaywanttobackup. AnalysisServicesbackupAnalysisServiceshasabuiltinbackupfunctionalitythatgeneratesasingle backupfilefromacubedatabase.AnalysisServicesbackupspeedshavebeensignificantlyimprovedin SQLServer2008andformostsolutions,youcansimplyusethisbuiltinbackupandrestore functionality.Aswithallbackupsolutions,youshouldofcoursetesttherestorespeed. SANbasedbackupIfyouuseaSAN,youcanoftenmakebackupsofaLUNusingthestoragesystem itself.ThisbackupprocessistransparenttoAnalysisServicesandtypicallyoperatesontheLUNlevel. YoushouldcoordinatewithyourSANadministratortomakesurethecorrectLUNsarebackedup, includingallrelevantdatafoldersusedtostorethecube.YoushouldalsomakesurethattheSAN backuputilityusesaVDI/VSScomplianttooltocalltoWindowsbeforethebackupistaken.Ifyoudonot useaVDI/VSStool,youriskgettingchkdskerrorswhentheLUNisrestored. FilebasedbackupcopyInSQLServer2008AnalysisServicesandSQLServer2008R2AnalysisServices, youcanattachdatabaseifyouhaveacopyofthedatafolderthedatabaseresidesin.Thismeansthata detachedcopyoffilesinastalecubecanserveasabackupofthedatabase.Thisoptionisavailableonly withSQLServer2008AnalysisServicesandSQLServer2008R2AnalysisServices.Youcanusethesame toolsthatyouuseforscaleoutcubes(forexample,RobocopyorNiceCopy).Restoringinthiscase meanscopyingthefilesbacktotheserverandattachingthedatabase. DontbackupWhilethismaysoundlikeasillyoption,itdoeshavemeritinsomedisasterrecovery scenarios.Cubesarebuiltonrelationaldatasources.Ifthesesourcesareguaranteedtobeavailableand securelybackedup,itmaybefastertosimplyreprocessthecubethantorestoreitfrombackup.Ifyou usethisoption,makesurethatnodataresidesinthecubesthatcannotberecreatedfromthe relationalsource(forexample,dataloadedviatheSQLServerIntegrationServicesAnalysisServices destination).Ofcourse,youshouldalsomakesurethatthecubestructureitselfisavailable,includingall aggregationandpartitiondesignsthathavebeenchangedontheproductionserverfromthestandard deploymentscript. ThisisparticularlytrueforROLAPcubes.Inthiscase(asinallthepreviousscenarios),youshouldalways maintainanupdatedbackupofyourAnalysisServicesprojectthatallowsforaredeploymentofthe solution(withthesubsequentrequiredprocessing),incaseyourbackupmediasuffersanykindof physicalcorruption.

3.4.1.1 Synchronization as Backup

Ifyouarebackingupsmallormediumsizedatabases,theAnalysisServicessynchronizationfeatureisan operationallyeasymethod.ItsynchronizesdatabasesfromsourcetotargetAnalysisServiceservers.The processscansthedifferencesbetweenthetwodatabasesandtransfersonlythefilesthathavebeen modified.Thereisoverheadassociatedwithscanningandverifyingthemetadatabetweenthetwo differentdatabases,whichcanincreasethetimeittakestosynchronize.Thisoverheadbecomes increasinglyapparentinrelationtothesizeandnumberofpartitionsofthedatabases. 148

Herearesomeimportantfactorstoconsiderwhenyouworkwithsynchronization: Attheendofthesynchronizationprocess,aWritelockmustbeappliedtothetargetserver.If thislockisqueuedupbehindalongrunningquery,itcanpreventusersfromqueryingthetarget database. Duringsynchronization,areadcommitlockisappliedtothesourceserver,whichpreventsusers fromprocessingnewdatabutallowsmultiplesynchronizationstooccurfromthesamesource server. Forenterprisesizedatabases,thesynchronizationmethodcanbeexpensiveandlow performing.Becausesomeoperationsaresinglethreaded(suchasthedeleteoperation),having highperformanceserversanddisksubsystemsmaynotresultinfastersynchronizationtimes. Becauseofthis,werecommendedthatforenterprisesizedatabasesyouusealternatemethods, suchasRobocopy(discussedlaterinthisbook)orhardwarebasedsolutions(forexample,SAN clonesandsnapshots). Whenexecutingmultiplesynchronizationoperationsagainstasingleserver,youmaygetthe bestperformance(byminimizinglockcontentions)byqueuingupthesynchronizationrequests andthenrunningthemserially. Forthesamelockingcontentionreasons,planyoursynchronizationsfordowntimes,when queryingandprocessingarenotoccurringontheaffectedservers.Whiletherearelocksinplace topreventoverwrites,processessuchaslongrunningqueriesandprocessingmaypreventthe synchronizationfromcompletinginatimelyfashion.

Formoreinformation,seetheAnalysisServicesSynchMethodsectioninAnalysisServices SynchronizationBestPracticestechnicalnote (http://sqlcat.com/technicalnotes/archive/2008/03/16/analysisservicessynchronizationbest practices.aspx).

3.4.1.2 Robocopy
ThebasicprinciplebehindtheRobocopymethodistouseafastcopyutility,suchasRobocopy,tocopy theOLAPdatafolderfromoneservertoanother.Byfollowingthesamplescriptnotedinthetechnical note,SampleRobocopyScripttocustomersynchronizeAnalysisServicesdatabases (http://sqlcat.com/technicalnotes/archive/2008/01/17/samplerobocopyscripttocustomer synchronizeanalysisservicesdatabases.aspx),youcanperformdeltacopies(thatis,copyonlythedata thathaschanged)oftheOLAPdatafolderfromonesourcetoanothertargetsourceinparallel.This methodisoftenemployedinenterpriseenvironmentsbecausethekeyfactorhereisfast,robustdata filetransfer. However,thekeydisadvantagesofusingthisapproachinclude: Youmuststopandthenrestart(inSQLServer2005AnalysisServices)ordetach(inSQLServer 2008AnalysisServicesandSQLServer2008R2AnalysisServices)yourAnalysisServicesservers whenyouuseafastcopyutility. YoucannotusethesynchronizationfeatureandRobocopytogether.

149

Thisapproachmakestheassumptionthatthereisonlyonedatabaseonthatinstance;thisis usuallyokaybecauseofitssize. Somefunctionalityislostifyouusethismethod,including,butnotlimitedto,writeback,ROLAP, andrealtimeupdates.

Nevertheless,thisisanespeciallyeffectivemethodforqueryserver/processingserverarchitectures thatinvolveonlyonedatabaseperserver. References: ScaleOutQueryingwithAnalysisServices (http://sqlcat.com/whitepapers/archive/2007/12/16/scaleoutqueryingwithanalysis services.aspx)

3.4.2 Clustered Servers

AnalysisServicescanparticipateinaWindowsfailoverclusterusingashareddisksubsystem,suchasa SAN.Whenprovisioningstorageforacluster,therearesomethingsyoushouldbeawareof. Itisnotpossibletousedynamicdiskstripes.Inthiscase,youhavetwooptionsforspreadingthecubes overallavailableLUN: HaveyourSANadministratorconfigureamegaLUN. SelectivelyplacepartitionsondifferentLUNsandthenmanuallybalancetheloadbetween theseLUNs.

MegaLUN:MostSANstodaycanstripemultiple,smallerLUNsintoalargemegaLUN.TalktoyourSAN administratoraboutthisoption. Selectiveplacement:IfyouneedtousemultipleLUNsforasinglecube,youshouldtrytomanually balanceI/OtrafficbetweentheseLUNs.Onewaytoachievethisistopartitionthecubeintoroughly equalsizedslicesbasedonadimensionkey.Veryoften,theonlywaytoachieveroughlyequalbalance likethisistoimplementatwolayerpartitioningonbothdateandthesecondary,balancingkey.For moreinformation,seetheRepartitioningsectioninPart1.

3.5 Diagnosing and Optimizing

Thissectiondiscusseshowtotroubleshootproblemsandimplementchangesthatcanbemade transparentlyinthecubestructurestoimproveperformance.Manyofthesechangesarealready documentedinPart1,butsomeadditionalconsiderationsapplyfromanoperationsperspective.

3.5.1 Tuning Processing Data

DuringProcessDataoperations,AnalysisServicesusestheprocessingthreadpoolforworkerthreads usedbythestorageengine. ProcessDataoperationsusethreeparallelthreadstoperformthefollowingtasks: 150

Querythedatasourceandextractdata Lookupdimensionkeysindimensionstoresandpopulatetheprocessingbuffer Writetheprocessingbuffertodiskwhenitisfull

YoucanusuallyincreasethroughputofProcessDatabytuningthenetworkstackandprocessingmore partitionsinparallel.However,youmaystillbenefitfromtuningtheprocesspool. TooptimizethesesettingsfortheProcessDataphase,checkyourPerformanceMonitorcounteronthe objectMSOLAP:Threadsandusethefollowingtableforguidance. Situation Processingpooljobqueuelength>0and Processingpoolidlethreads=0 forlongerperiodsduringprocessing. BothProcessingpooljobqueuelength>0and Processingpoolidlethreads>0atsametime duringprocessing. Action IncreaseThreadPool\Process\MaxThreadsand thenretest. DecreaseCoordinatorExecutionMode(for example,changeitfrom4to8)andthenretest.

YoucanusetheProcessor%ProcessorTimeTotalcounterasaroughindicatorofhowmuchyou shouldchangethesesettings.Forexample,ifyourCPUloadis50percent,youcandouble ThreadPool\Process\MaxThreadstoseewhetherthisalsodoublesyourCPUusage.Itispossibletoget to100percentCPUloadinasystemwithoutbottlenecks,thoughyoumaywanttoleavesome headroomforgrowth.Keepinmindthatincreasedparallelismofprocessingalsohasaneffecton queriesrunningatthesystem.Ideally,useaseparateprocessingserveroradedicatedprocessingtime windowwherenooneisqueryingAnalysisServices.Ifthisisnotanoption,asyoudedicatemoreCPU powerandthreadstoprocessing,lessCPUwillbeusedforqueryresponses.Becauseprocessing consumesthreadsfromthesamepoolasquerysubcuberequests,youshouldalsobecarefulthatyou dontruntheprocessthreadpooldryifyouprocessandqueryatthesametime.Ifyouareprocessing thecubesduringasetprocessingwindowwithnousersonthebox,thiswillofcoursenotbeanissue. References: SQLServer2005AnalysisServices(SSAS)ServerProperties(http://technet.microsoft.com/en us/library/cc966526.aspx)

3.5.1.1 Optimizing the Relational Engine

InadditiontolookingatAnalysisServicesconfigurations,andsettings,youcanalsolookattherelational enginewhenyouareplanningimprovementstoyourAnalysisServicesinstallation.Thissectionfocuses onworkingwithrelationaldatafromSQLServer2005,SQLServer2008,orSQLServer2008R2. AlthoughAnalysisServicescanbeusedwithanyOLEDBor.NETdriverenableddatabase(suchasOracle orTeradata),theadviceheremaynotapplytosuchthirdpartyenvironments.However,ifyouarea thirdpartyDBA,youmaybeabletotranslatethetechniquesdiscussedheretosimilaronesinyourown environment. 151

3.5.1.1.1 Relational Indexing for Partition Processing Whileyougenerallywanteachcubepartitiontotouchatmostonerelationalpartition,thereverseis nottrue.Itisperfectlyviabletohavetohavemorethanonecubepartitionaccessingthesame relationalpartition.Asanexample,arelationalsourcethatispartitionedbyyearwithacubethatis partitionedbymonthcanstillprovideoptimalprocessingperformance. Ifyoudonothaveaonetoonerelationshipbetweenrelationalandcubepartitions,yougenerallywant anindextosupportthefactprocessingquery.Thebestchoiceofindexforthispurposeisaclustered index;ifyourloadstrategyallowsyoutomaintainsuchanindex,thisiswhatyoushouldaimfor. Whenapartitionprocessingqueryissupportedbyanindextheplanshouldlooklikethis.

Figure 17 Supporting Measure Group processing with an index

References: Top10SQLServer2005PerformanceIssuesforDataWarehouseandReportingApplications (http://sqlcat.com/top10lists/archive/2007/11/21/top10sqlserver2005performanceissues fordatawarehouseandreportingapplications.aspx) BenGan,ItzikandLuborKollar,InsideMicrosoftSQLServer2005:TSQLQuerying.Redmond, Washington:MicrosoftPress,2006.

3.5.1.1.2 Relational Indexing for Dimension Processing Ifyoufollowadimensionalstarschemadesign(whichwerecommendforlargecubes),mostdimension processingqueriesshouldrunrelativelyfastandtakeonlyatinyportionofthetotalcubeprocessing time.Butifyouhaveverylargedimensionswithmillionsofrowsordimensionswithlotsofattributes, someperformancecanbestillbegainedbyindexingtherelationaldimensiontable.Tounderstandthe bestindexingstrategy,itisusefultoknowwhichformdimensionsprocessingqueriestake.Thenumber ofqueriesgeneratedbyAnalysisServicesdependsontheattributerelationshipsdefinedinthe dimension.Foreachattributeinthedimension,thefollowingqueryisgeneratedduringprocessing. SELECTDISTINCT<attribute>,[<relatedattribute>[n]] FROM<dimensiontable> Considerthefollowingexampledimension,withCustomerIDasthekeyattribute. 152

Country

State Age Gender City

Name Customer ID

Zip

Figure 60 Example Customer Dimension Attribute relationships

Thefollowingqueriesaregeneratedduringdimensionprocessing.
SELECTDISTINCT Country FROM Dim.Customer SELECTDISTINCTState, Country FROM Dim.Customer SELECTDISTINCT City,StateFROM Dim.Customer SELECTDISTINCT Zip, City FROM Dim.Customer SELECTDISTINCT Name FROM Dim.Customer SELECTDISTINCT Gender FROM Dim.Customer SELECTDISTINCT Age FROM Dim.Customer SELECTDISTINCT CustomerID, Name, Gender, Age, Zip FROM Dim.Customer

Theindexingstrategyyouapplydependsonwhichattributeyouaretryingtooptimizefor.Toillustratea tuningprocess,thissectionwalksthroughsometypicalcases. Keyattribute:thekeyattributeinadimension,inthisexampleCustomerID,canbeoptimizedby creatingaunique,clusteredindexonthekey.Typically,usingsuchaclusteredindexisalsothebest strategyforrelationaluseraccesstothetablesoyourDBAwillbehappyifyoudothis.Inthisexample, thefollowingindexhelpswithkeyprocessing.

CREATEUNIQUECLUSTEREDINDEX CIX_CustomerID ON Dim.Customer(CustomerID)

Thiswillcreatethefollowing,optimalqueryplan.

153


Figure 61 - A good key processing plan

Highcardinalityattributes:Forhighcardinalityattributes,likeName,youneedanonclusteredindex. NoticetheDISTINCTintheSELECTquerygeneratedbyAnalysisServices.
SELECTDISTINCT Name FROM Dim.Customer

DISTINCTgenerallyforcestherelationalenginetoperformasorttoremoveduplicatesinthereturned dataset.Thesortoperationresultsinaplanthatlookslikethis.

Figure 20 - Expensive sort plan during dimension processing

Ifthisplantakesalongtimetorun,whichcouldbethecaseforlargedimension,considercreatingan indexthathelpsremovethesort.Inthisexample,youcancreatethisindex.
CREATEINDEX IX_Customer_Name ON Dim.Customer(Name)

Thisindexgeneratesthefollowing,muchbetterplan.

Figure 62 - A fast high cardinality attribute processing plan

Lowcardinalityattributes:Forattributesthatarepartofalargedimensionbutlowgranularity,even theprecedingindexoptimizationmayresultinanexpensiveplan.ConsidertheCityattributeinthe customerdimensionexample.Thereareveryfewcitiescomparedtothetotalnumberofcustomersin thedimension.Forexample,youwanttooptimizeforthefollowingquery.

154

SELECTDISTINCT City,StateFROM Dim.Customer

CreatingamulticolumnindexonCityandStateremovesthesortoperationrequiredtoreturnDISTINCT rowsresultinginaplanverysimilartotheoptimizationperformedearlierwiththeNameattribute. Thisisbetterthanrunningthequerywithnoindexedaccess.Butitstillresultsintouchingonerowper customerwhichisfarfromoptimalconsideringthatthereareveryfewcitiesinthetable. IfyouhaveSQLServerEnterprise,youcanoptimizetheSELECTqueryevenfurtherbycreatingan indexedviewlikethis.

CREATEVIEW Dim.Customer_CityState WITHSCHEMABINDING AS SELECT City,State,COUNT_BIG(*)AS NumRows FROM Dim.Customer GROUPBY City,State GO CREATEUNIQUECLUSTEREDINDEX CIX_CityState ON Dim.Customer_CityState(City,State)

SQLServermaintainsthisaggregateviewondisk,anditstoresonlythedistinctvaluesofCityandState exactlythedatastructureyouarelookingfor.Runningthedimensionprocessingquerynowresultsin thefollowingoptimalplan.

Figure 63 - Using indexed views to optimize for low-cardinality attributes

3.5.1.1.3 Overoptimizing and Wasting Time Itispossibletotunetherelationalenginetocutdowntimeonprocessingsignificantly,especiallyfor partitionprocessingandlargedimensions.However,bearinmindthateverytimeyouaddanindextoa table,youaddonemoredatastructurethatmustbemaintainedwhenusersmodifyrowsinthattable. Relationalindexing,likeaggregationdesigninacubeandmuchofBIanddatawarehousing,istradeoff betweendatamodificationspeedanduserqueryperformance.Thereistypicallyasweetspotinthis spacethatwilldependonyourworkload.Differentpeoplehavedifferentskills,andtheperceptionof wherethatsweetspotlieswillchangewithexperience.Asyougetclosertotheoptimalsolution, increasedthetuningeffortwilloftenreachapointofdiminishingreturnswherethespeedofthesystem movesasymptoticallytowardstheoptimumbalance.Monitoryourowntuningeffortsandtryto understandwhenyouaregettingclosetothatflatlinebehavior.Astemptingasfulltuningexercisescan betothetechnicallysavvy,noteverysystemneedsbenchmarkperformance. 155

3.5.1.1.4 Using Index FILLFACTOR = 100 and Data Compression Ifpagesplittingoccursinanindex,thepagesoftheindexmayenduplessthan100percentfull.The effectisthatSQLServerwillbereadingmoredatabasepagesthannecessarywhenscanningtheindex. YoucancheckforindexpagesarenotfullbyqueryingtheSQLServerDMV sys.dm_db_index_physical_stats.Ifthecolumnavg_page_space_used_in_percentissignificantlylower than100percent,aFILLFACTOR100rebuildoftheindexmaybeinorder.Itisnotalwayspossibleto rebuildtheindexlikethis,butthistrickhastheabilitytoreduceI/O.Forstaledata,rebuildingthe indexesonthetableisoftenagoodideabeforeyoumarkthedataasreadonly. InSQLServer2008youcanuseeitherroworpagecompressiontofurtherreducetheamountofI/O requiredbytherelationaldatabasetoservethefactprocessquery.CompressionhasaCPUoverhead, butreductioninI/Ooperationsisoftenworthit. References: DataCompression:Strategy,CapacityPlanningandBestPractices http://msdn.microsoft.com/enus/library/dd894051%28v=sql.100%29.aspx

3.5.1.1.5 Eliminating Database Locking Overhead WhenSQLServerscansanindexortable,pagelocksareacquiredwhiletherowsarebeingread.This ensuresthatmanyuserscanaccessthetableconcurrently.However,fordatawarehouseworkloads, thispagelevellockingisnotalwaystheoptimalstrategyespeciallywhenlargedataretrievalqueries likefactprocessingaccessthedata. BymeasuringthePerfmoncounterMSSQL:LocksLockRequests/SecandlookingforLCKeventsin sys.dm_os_wait_stats,youcanseehowmuchlockingoverheadyouareincurringduringprocessing. Toeliminatethislockingoverhead,youhavethreeoptions: Option1:SettherelationaldatabaseinReadOnlymodebeforeprocessing. Option2:BuildthefactindexeswithALLOW_PAGE_LOCKS=OFFandALLOW_ROW_LOCKS=OFF. Option3:Processthroughaview,specifyingtheWITH (NOLOCK)orWITH (TABLOCK)queryhint.

Option1maynotalwaysfityourscenario,becausesettingthedatabasetoreadonlymoderequires exclusiveaccesstothedatabase.However,itisaquickandeasywaytocompletelyremoveanylock waitsyoumayhave. Option2isoftenagoodstrategyfordatawarehouses.BecauseSQLServerReadlocks(Slocks)are compatiblewithotherSlocks,tworeaderscanaccessthesametabletwice,withoutrequiringthefine granularityofpageandrowlocking.Ifinsertoperationsareonlydoneduringbatchtime,relyingsolely ontablelocksmaybeaviableoption.Todisablerowandpagelockingonatableandindex,rebuildALL byusingastatementlikethisone. 156

ALTERINDEXALLON FactInternetSales REBUILD WITH (ALLOW_PAGE_LOCKS=OFF,ALLOW_ROW_LOCKS=OFF)

Option3isaveryusefultechnique.Processingthroughaviewprovidesyouwithanextralayerof abstractionontopofthedatabaseagooddesignstrategy.Intheviewdefinitionyoucanadda NOLOCKorTABLOCKhinttoremovedatabaselockingoverheadduringprocessing.Thishasthe advantageofmakingyourlockingeliminationindependentofhowindexesarebuiltandmanaged.

CREATEVIEW vFactInternetSales AS SELECT [ProductKey], [OrderDateKey], [DueDateKey] ,[ShipDateKey], [CustomerKey], [PromotionKey] ,[CurrencyKey], [SalesTerritoryKey], [SalesOrderNumber] ,[SalesOrderLineNumber], [RevisionNumber], [OrderQuantity] ,[UnitPrice], [ExtendedAmount], [UnitPriceDiscountPct] ,[DiscountAmount], [ProductStandardCost], [TotalProductCost] ,[SalesAmount], [TaxAmt], [Freight] ,[CarrierTrackingNumber] ,[CustomerPONumber] FROM [dbo].[FactInternetSales] WITH (NOLOCK)

IfyouusetheNOLOCKhint,bewareofthedirtyreadsthatcanoccur.Formoreinformationabout lockingbehaviors,seeSETTRANSACTIONISOLATIONLEVEL(http://technet.microsoft.com/en us/library/ms173763.aspx)inSQLServerBooksOnline. 3.5.1.1.6 Forcing Degree of Parallelism in SQL Server DuringProcessData,apartitionprocessingtaskislimitedbythespeedachievablefromasinglenetwork connectionthedatasource.Thesespeedscanvarybetweenafewthousandrowspersecondforlegacy datasources,toaround100,000rowspersecondfromSQLServer.Perhapsthatisnotfastenoughfor you,andyouhavefollowedtheguidanceinthisbooktoallowmultiplepartitionstoprocessinparallel scalingProcessDatanearlylinearly. Wehaveseencustomerreach6.5millionrowspersecondintoacubebyprocessing64partitions concurrently.Bearinmindwhatittakestotransportmillionsofrowsoutofarelationaldatabaseevery second.Eachprocessingquerywillbeselectingdatafromabigtable,aggressivelyfetchingdataasfast asthenetworkstackandI/Osubsystemcandeliverthem.WhenSQLServerreceivesjustasingle requestforallrowsinalargefacttable,itwillspawnmultiplethreadsinsidethedatabaseengineto servethatitasfastaspossibleutilizingallserverresources.ButwhattheDBAoftherelationalengine maynotknow,isthatinafewmilliseconds,yourcubedesignissetuptoaskforanotheroneofthose largetablesconcurrently.Thispresentstherelationalenginewithaproblem:howmanythreadseach queryshouldbeassignedtooptimizethetotalthroughputofthesystem,withoutsacrificingoverall performanceofindividualprocessingcommands.Itiseasytoseethatraceconditionsmaycreateall sortsoninterestingsituationstofurthercomplicatethis.IfparallelismoverloadstheDatabaseEngine, SQLServermustresorttocontextswitchingbetweentheactivetasks,continuouslyredistributingthe scarceCPUresourcesandwastingCPUtimewithscheduling.Youcanmeasurethishappeningasahigh SOS_SCHEDULER_YIELDwaitinsys.dm_os_wait_stats. 157

Fortunately,youcandefendyourselfagainstexcessiveparallelismbyworkingtogetherwiththecube designertounderstandhowmanypartitionsareexpectedtoprocessatthesametime.Youcanthen assignasmallersubsetoftheCPUcoresintherelationaldatabaseforeachpartition.Carefullydesigning forthisparallelismcanhavealargeimpact.Wehaveseencubeprocessdataspeedsmorethandouble whentheassignedCPUresourcesarecarefullycontrolled. Youhaveoneortwowaystopartitionserverresourcesforoptimalprocessdataspeeds,dependingon whichversionofSQLServeryourun. InstancereconfigurationUsingsp_configure,youcanlimitthenumberofCPUresourcesasingle querycanconsume.Forexample,consideracubethatprocesseseightpartitionsinparallelfroma computerrunningSQLServerwith16cores.Todistributetheprocessingtasksequallyacrosscores,you wouldconfigurelikethis.
EXECsp_configure'show advanced options', 1 RECONFIGURE EXECsp_configure'max degree of parallelism', 2 RECONFIGURE

Unfortunately,thisisabruteforceapproach,whichhasthesideeffectofchangingthebehaviorofthe entireinstanceofSQLServer.Itmaynotbeaproblemiftheinstanceisdedicatedforcubeprocessing, butitisstillacrudetechnique.Unfortunately,thisistheonlyoptionavailabletoyouonSQLServer 2005. ResourceGovernorIfyourunSQLServer2008orSQLServer2008R2,youcanuseResourceGovernor tocontrolprocessingqueries.Thisisthemostelegantsolution,becauseitcanoperateoneachdata sourceviewindividually. ThefirststepistocreatearesourcepoolandaworkloadgrouptocontroltheAnalysisServices connections.Forexample,tothefollowingstatementlimitseachProcessDatataskto2CPUcoresanda maximummemorygrantof10percentperquery.
CREATERESOURCEPOOL [cube_process] WITH( min_cpu_percent=0 , max_cpu_percent=100 , min_memory_percent=0 , max_memory_percent=100) GO CREATEWORKLOADGROUP [process_group] WITH( group_max_requests=0 , importance=Medium , request_max_cpu_time_sec=0 , request_max_memory_grant_percent=10 , request_memory_grant_timeout_sec=0 , max_dop=2) USING [cube_process] GO

158

 Thenextstepistodesignaclassifierfunctionthatrecognizestheincomingcubeprocessrequests.There areseveralwaystorecognizecubeconnection.Oneistousethehostname.Anotheristouse applicationnames(whichyoucansetintheconnectionstringinthedatasourceviewinthecube).The followingexamplerecognizesallAnalysisServicesconnectionsthatusethedefaultvaluesinthe connectionstring.

USE [master] GO CREATEFUNCTION fnClassifier() RETURNSsysname WITHSCHEMABINDING AS BEGIN DECLARE @group SYSNAME IFAPP_NAME()LIKE'%Analysis Services%'BEGIN SET @group='process_group' END ELSEBEGIN SET @group ='default' END RETURN (@group) END

Makesureyoutesttheclassifierfunctionbeforeapplyingitinproduction.Afteryouarecertainthatthe functionworks,enableit.
ALTERRESOURCEGOVERNORWITH (CLASSIFIER_FUNCTION = [dbo].[fnClassifier]); GO ALTERRESOURCEGOVERNORRECONFIGURE;

3.5.1.1.7 Loading from Oracle AnalysisServicesiscommonlydeployedinheterogeneousenvironments,withOraclebeingoneofthe maindatasources.Becausecubesoftenneedtoreadalotofdatafromthesourcesystemduring processing,itisimportantthatyouusehighspeeddriverstoextractthisdata.Wehavefoundthatthe nativeOracledriversprovideareasonableperformance,especiallyifmanypartitionsareprocessedin parallel. However,wehavealsofoundthatdatacanbeextractedfromaSQLServerdatasourceat510times thespeedofthesameOraclesource.TheSQLServerdriverSQLNLCIisoptimizedforveryhigh extractionspeedssometimesreachingupto80,000100,000rowspersecondfromasingleTCP/IP connection,dependingonthesourceschema).WehavetestedprocessingspeedsontopofSQLServer allthewayto6.1millionrowspersecond. ConsiderthatSQLServerIntegrationServices,partofthesameSKUasAnalysisServices,hasahigh speedOracledriveravailablefordownload.Thisdriverisoptimizedforhighspeedextractionfrom

159

Oracle.Wehavefoundthatthefollowingarchitectureoftenprovidesfasterprocessingperformance thanprocessingdirectlyontopofOracle.

Figure 64 - Fast Processing on Oracle

References: MicrosoftConnectorsVersion1.1forOracleandTeradatabyAttunity http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6732934c2eea4a7f 85a88ba102e6b631 TheDataLoadingPerformanceGuidehttp://msdn.microsoft.com/enus/library/dd425070.aspx o DescribeshowtomovelotsofdataintoSQLServer

3.5.2 Tuning Process Index

AswithProcessDataworkloads,youcanoftenincreasespeedofProcessIndexbyrunningmore partitionsinparallel.However,ifthisoptionisnotavailabletoyou,thethreadsettingscanprovidean extrabenefit.WhenyoumeasureCPUutilizationwiththecounterProcessor%ProcessorTimeTotal andyoufindutilizationislessthan100percentwithnoI/Obottlenecks,thereisagoodchancethatyou canincreasethespeedoftheProcessIndexphasefurtherbyusingthetechniquesinthissection.

3.5.2.1 CoordinatorBuildMaxThreads
Whenasinglepartitionisscanned,theamountofthreadsusedtoscanextentsislimitedbythe CoordinatorBuildMaxThreadssetting.Thesettingdeterminesthemaximumnumberofthreads allocatedperaggregationprocessingjob.Itistheabsolutenumberofthreadsthatcanberunbyan aggregationprocessingjob.Keepinmindthatyouarestilllimitedbythenumberofthreadsinthe processthreadpoolwhenprocessing,soincreasingthisvaluemayrequireincreasingthethreads availabletotheprocessthreadpooltoo.

160


Figure 24 CoordinatorBuildMaxThreads

Ifyouarenotabletodrivehighparallelismbyusingmorepartitions,youcanchangethe CoordinatorBuildMaxThreadsvalue.Increasingthisallowsyoutousemorethreadsperpartitionwhile buildingaggregationsforeachpartitionduringtheProcessIndexphase

3.5.2.2 AggregationMemoryMin and Max

AsyouincreaseparallelismofProcessIndex,youmayrunintomemorybottlenecks.Onaserverwith morethanaroundtenProcessndexjobsrunninginparallel,youmayneedtoadjust AggregationMemoryMinandAggregationMemoryMaxtogetoptimalresults.Formoreinformation, seetheMemorysectioninPart2.

3.5.3 Tuning Queries

QueryoptimizationismostlycoveredinPart1andwillgenerallyinvolvedesignorapplicationlevel changes.However,therearesomeoptimizationsthatcanbemadeinaproductioncubethatare transparenttousersandBIdevelopers.Thesearedescribedhere.

3.5.3.1 Thread Pool Tuning

AnalysisServicesusestheparsingthreadpools,thequerythreadpool,andtheprocessthreadpoolfor queryworkloadsonAnalysisServices.AlistenerthreadlistensforaclientrequestontheTCP/IPport specifiedintheAnalysisServicesproperties.Whenaquerycomesin,thelistenerthreadbrokersthe requesttooneoftheparsingthreadpools.Theparsingthreadpoolseitherexecutetherequest immediatelyorsendtherequestofftothequeryorprocessthreadpool.

161

Workerthreadsfromthequerypoolcheckthedataandcalculationcachestoseewhethertherequest canbeservedfromcache.Iftherequestcontainscalculationsthatneedtobehandledbytheformula engine,workerthreadsinthequerypoolareusetoperformthecalculationandstoretheresults.Ifthe queryneedstogotodisktoretrieveaggregationsorscanapartition,workerthreadsfromthe processingpoolareusedtosatisfytherequestandstoretheresultsincache. Therearesettingsinthemsmdsrv.inifilethatallowuserstotunethebehaviorofthethreadpools involvedinqueryworkloads.Guidanceonusingthemisprovidedinthissection. 3.5.3.1.1 Parsing Thread Pools TheAnalysisServicesprotocolusesSimpleObjectAnalysisProtocol(SOAP)andXMLAforAnalysis (XMLA)withTCP/IPorHTTP/HTTPSastheunderlyingtransportmechanism.Afteraclientconnectsto AnalysisServicesandestablishesaconnection,commandsarethenforwardedfromtheconnections inputbufferstooneofthetwoparsingthreadpoolswheretheXMLAparserbeginsparsingtheXMLA whileanalyzingtheSOAPheaderstomaintainthesessionstateofthecommand. Therearetwoparsingthreadpools:theshortcommandparsingpoolandthelongcommandparsing pool.Theshortcommandparsingpoolisusedforcommandsthatcanbeexecutedimmediatelyandthe longcommandparsingpoolisusedforcommandsthatrequiremoresystemresourcesandgenerally takelongertocomplete.Requestslongerthanonepackagearedispatchedtothelongparsingthread pool,andonepackagerequestsgototheshortparsingpool.Ifarequestisaquickcommand,likea DISCOVER,theparsingthreadisusedtoexecuteit.Iftherequestisalargeroperation,likeanMDX queryoranMDSCHEMA_MEMBERS,itisqueueduptothequerythreadpool. FormostAnalysisServicesconfigurationsyoushouldnotmodifyanyofthesettingsintheshortparsing orlongparsingthreadpools.However,withtheinformationprovidedhere,youcanimagineaworkload whereeithertheshortparsingorlongparsingthreadpoolsrundry.Wehaveonlyseenonesuch workload,atveryhighconcurrency,soitisunlikelythatyouwillneedtotunetheparsingthreadpool.If youdohaveaproblemwithonetheparsingthreadpools,itwillshowupasvaluesconsistentlyhigher thanzeroinMSOLAP/ThreadShortparsingjobqueuelengthorMSOLAP/ThreadLongparsingjob queuelength. 3.5.3.1.2 Query Thread Pool Settings AlthoughmodifyingtheThreadPool\Process\MaxThreadsandThreadPool\Query\MaxThreads propertiescanincreaseparallelismduringquerying,youmustalsotakeintoaccounttheadditional impactofCoordinatorExecutionModeasdescribedintheconfigurationsection. Inpracticalterms,thebalancingofjobsandthreadscanbetricky.Ifyouwanttoincreaseparallelism,it isimportanttofirstnarrowdownparallelismasthebottleneck.Tohelpyoudeterminewhetherthisis thecase,itishelpfultomonitorthefollowingperformancecounters: Threads\QuerypooljobqueuelengthThenumberofjobsinthequeueofthequerythread pool.Anonzerovaluemeansthatthenumberofqueryjobshasexceededthenumberof availablequerythreads.Inthisscenario,youmayconsiderincreasingthenumberofquery

162

threads.However,ifCPUutilizationisalreadyveryhigh,increasingthenumberofthreadsonly addstocontextswitchesanddegradesperformance. Threads\QuerypoolbusythreadsThenumberofbusythreadsinthequerythreadpool.Note thatthiscounterisbrokeninsomeversionsofAnalysisServicesanddoesnotdisplaythecorrect value.ThevaluecanalsobederivedfromthesizeofthethreadpoolminustheThreads\Query poolidlethreadscounter. Threads\QuerypoolidlethreadsThenumberofidlethreadsinthequerythreadpool.

AnyindicationofqueuesinanyofthesethreadpoolswithaCPUloadlessthan100percentindicatea potentialoptionfortuningthethreadpool. References AnalysisServicesQueryPerformanceTop10BestPractices (http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/ssasqptb.mspx)

3.5.3.2 Aggregations
Aggregationsbehaveverymuchlikeindexesinarelationaldatabase.Theyarephysicaldatastructures thatareusedtoanswerfrequentlyaskedqueriesinthedatabase.Addingaggregationstoacube enablesyoutospeeduptheoverallthroughput,atthecostofmorediskspaceandincreasedprocessing times. Notethatjustaswithrelationalindexing,noteverysingle,potentiallyhelpfulaggregateshouldbe created.Whenthespaceofpotentialaggregatesgrowslarge,oneaggregatebeingreadmaypush anotheroutofmemoryandcasediskthrashing.AnalysisServicesmayalsostruggletofindthebest aggregateamongmanymatchingagivenquery. Agoodruleofthumbisthatnomeasuregroupshouldhavemorethan30percentofitsstoragespace dedicatedtoaggregates.Todiscoverwhichaggregatesaremostuseful,youshouldcollectthequery subcubeverboseeventusingSQLServerProfilerwhilerunningarepresentativeworkload.Bysumming theruntimeofeachuniquesubcubeyoucangetagoodoverviewofthemostvaluableaggregationsto create.Formoreinformationabouthowtocreateaggregates,seePart1. Therearesomeaggregatesthatyougenerallyalwayswanttocreate,namelytheonesthatareused directlybysetsdefinedinthecalculationscript.Youcanidentifythosebytracingqueriesgoingtothe cubeafteraprocessevent.Thefirstusertorunaqueryorconnecttothecubeafteraprocessingevent maytriggerquerysubcubeverboseeventsthathaveSPID=0thisisthecalculationscriptitself requestingdatatoinstantiateanynamedsetsdefinedthere. References:

163

AnalysisServices2005AggregationDesignStrategy http://sqlcat.com/technicalnotes/archive/2007/09/11/analysisservices2005aggregation designstrategy.aspx o AlsoappliestoSQLServer2008AnalysisServicesandSQLServer2008R2Analysis Services AggregationManageronCodePlex http://bidshelper.codeplex.com/wikipage?title=Aggregation%20Manager&referringTitle=Home o Makesthetaskofdesigningaggregationseasier BIDSHelperhttp://bidshelper.codeplex o Assistswithmanycommontasks,includingaggregationdesign

3.5.3.3 Optimizing Dimensions

Gooddimensiondesigniskeytogoodperformanceoflargecubes.Yourdevelopersshoulddesign dimensionsthatfittheneedsofthebusinessusers.However,therearesomeminorchangesyoucan oftenmakeonalivesystemthatcanyieldasignificantperformancebenefit.Thechangesarenot100 percenttransparenttobusinessusers,buttheymaywellbeacceptableorevenimprovethebehaviorof thecube.BesuretocoordinatewithyourBIdeveloperstounderstandwhetheryoucanmakeanyofthe changesdiscussedhereinthecube. Removingthe(All)LevelInsomedimensions,itdoesnotmakesensetoaskforthe(All)levelina query.Theclassicexampleisadatedimension.Ifthe(All)levelispresent,andnodefaulthasbeenset,a userconnectingtothecubemayinadvertentlyaskforthesumofallyears.Thisnumberisrarelyauseful valueandjustwastesresourcesattheserver. SettingdefaultmembersWheneverauserissuesanMDXstatement,everyhierarchynotexplicitly mentionedinthequeryusesitsdefaultmembervalue.Bydefault,thismemberisthe(All)levelinthe hierarchy.The(All)levelmaynotbethetypicalusagescenario,whichwillcausetheusertoreissuethe querywithanewvalue.Itissometimesusefultosetanotherdefaultvalueinthedimensionthatmore accuratelyreflectsthemostcommonusagescenario.Thiscanbedonewithasimplemodificationtothe calculationscriptofthecube.Forexample,thefollowingcommandsetsanewdefaultmemberinthe Datedimension.
ALTERCUBE [Adventure Works]UPDATE DIMENSION [Date], DEFAULT_MEMBER='[Date].[Date].&[2000]'

ScopetheAllLeveltoNULLRemovingthe(All)levelandsettingdefaultmemberscanbeconfusingto usersofExcel.Anotheroptionistoforcethe(All)leveltobeNULLforthedimensionswherequerying thatlevelmakesnosense.YoucanuseaSCOPEstatementforthat.Formoreinformation,seePart1. AttributeHierarchyEnabledThisproperty,whensettofalse,makesthepropertyinvisibleasan attributehierarchytousersbrowsingthecube.Thisreducesthemetadataoverheadofthecube.Notall attributescanbedisabledlikethis,butyoumaybeabletoremovesomeofthemafterworkingwiththe users. 164

Notethattherearemanyotheroptimizationsthatcanbedoneondimensions.Part1ofthisbook containsmoredetailedinformation.

3.5.3.4 Calculation Script Changes

ThecalculationscriptofacubecontainstheMDXstatementsthatmakeupthenonmaterializedpartof thecube.Optimizingthecalculationscriptisaverylargetopicthatisoutsidethescopeofthis document.Whatyoushouldknowitthatsuchchangescanoftenbemadetransparentlytotheuserand thatthegains,dependingontheinitialdesign,canoftenbesubstantial. YoushouldgenerallycollecttheperformancecountersunderMSOLAP:MDXbecausethesecanbeused bycubedeveloperstodeterminewhethertherearepotentialgainstobehad.Atypicalindicatorofa slowcalculationscriptisalargeratiobetweentheMSOLAP:MDX/Numberofcellbycellevaluation nodesandMSOLAP:MDX/Numberofbulkmodeevaluationnodes.

3.5.3.5 Repartitioning
Partitioningofcubescanbeusedtobothincreaseprocessingandqueryspeeds.Forexample,ifyou strugglewithprocessdataspeeds,splittingupthenightlybatchintomultiplepartitionscanincrease concurrencyoftheprocessingoperation.ThistechniqueisdocumentedinPart1ofthisbook. PartitionscanalsobeusedtoselectivelymovedatatodifferentI/Osubsystems.Anexampleofthisisa customerthatkeepsthelatestdatainthecubeonNANDdevicesandmovesoldandinfrequently accesseddatatoSATAdisks.YoucanmovepartitionaroundusingSQLServerManagementStudiointhe propertiespaneofthepartition.Alternatively,youcanuseXMLorscriptingtomovepartitiondataby executingaquerylikethis.
<AlterObjectExpansion="ExpandFull" xmlns="http://schemas.microsoft.com/analysisservices/2003/engine"> <Object> <DatabaseID>Adventure Works DW</DatabaseID> <CubeID>Adventure Works DW</CubeID> <MeasureGroupID>Fact Reseller Sales</MeasureGroupID> <PartitionID>Reseller_Sales_2001</PartitionID> </Object> <ObjectDefinition> <Partition> <ID>Reseller_Sales_2001</ID> <Name>Reseller_Sales_2001</Name> <StorageLocation>D:\MSAS10_50.KILIMANJARO\OLAP\2001\</StorageLocation> </Partition> </ObjectDefinition> </Alter>

Beawarethatafteryouhavechangedthestoragelocationofthepartitionthatmarksthepartitionas unprocessedandempty,youmustreprocessthepartitiontophysicallymovethedata.Notethat AnalysisServicescreatesafolderunderthepathyouspecify,andthisfolderisnamedusingaGUIDnot

165

easytodecodeforahuman.Tokeeptrackofwhereyourmovedpartitionsare,itisthereforean advantagetoprecreatefolderswithhumanreadablenamestoholdthedata. Forlargecubes,itisoftenagoodideatoimplementamatrixpartitioningscheme:partitiononboth dateandsomeotherkey.Thedatepartitioningisusedtoselectivelydeleteormergeoldpartitions.The otherkeycanbeusedtoachieveparallelismduringpartitionprocessingandtorestrictcertainuserstoa subsetofthepartitions.Forexample,consideraretailerthatoperatesinUS,Europe,andAsia.You mightdecidetopartitionlikethis.

ParallelProcessData Queryfilterngbasedonuserregion

Process

Process

US 201001 US 200912 US 200911

Europe 2010 Europe 200912 Europe 200911

Asia 2010 Asia 200912

MonthlyPartition

Asia 200911

US 200901

Europe 200901

Asia 200901

US 2008

Europe 2008

Asia 2008
YearlyPartitiong MonthsMergedat yearend

US 2007

Europe 2007

Asia 2007

RegionPartitioning

Process

Figure 65 - Example of matrix partitioning

Iftheretailergrows,theymaychoosetosplittheregionpartitionsintosmallerpartitionstoincrease parallelismofloadfurtherandtolimittheworstcasescansthatausercanperform.Forcubesthatare expectedtogrowdramatically,itisagoodideatochooseapartitionkeythatgrowswiththebusiness 166

andgivesyouoptionsforextendingthematrixpartitioningstrategyappropriately.Thefollowingtable containsexamplesofsuchpartitioningkeys. Industry WebRetail StoreRetail DataHosting Telecommunications Computerized manufacturing InvestmentBanking RetailBanking OnlineGaming Examplepartitionkey Customerkey Storekey HostIDorracklocation SwitchIDorcountrycodeor areacode ProductionLineIDorMachine ID StockExchangeorfinancial instrument CreditCardNumberor CustomerKey GameKeyorPlayerKey Sourceofdataproliferation Addingcustomersandtransactions Addingnewstores Addinganewserver Expandingintonewgeographical regionsoraddingnewservices Addingproductionlinesor(for machines)sensors Addingnewfinancialinstruments, products,ormarkets Increasingcustomertransactions Addingnewgamesorplayers

Sometimesitisnotpossibletocomeupwithagooddistributionofthekeysacrossthepartitions. Perhapsyoujustdonthaveagoodkeycandidatethatfitsthedescriptioninthepreviousparagraph,or perhapsthedistributionofthekeyisunknownatdesigntime.Insuchcases,abruteforceapproachcan beused:Partitiononthehashvalueofakeythathasahighenoughcardinalityandwherethereislittle skew.Userswillhavetotouchallthehashbucketsastheyquerythecube,butatleastyoucanperform parallelloading.Ifyouexpecteveryquerytotouchmanypartitions,itisimportantthatyoupayspecial attentiontotheCoordinatorQueryBalancingFactordescribedearlier. Asyouaddmorepartitions,themetadataoverheadofmanagingthecubegrowsexponentially.Asarule ofthumb,youshouldthereforeseektokeepthenumberofpartitionsinthecubeinthelowthousands. ThisaffectsProcessUpdateandProcessAddoperationsondimensions,whichhavetotraversethe metadatadependenciestoupdatethecubewhendimensionschange.Forlargecubes,preferlarger partitionsovercreatingtoomanypartitions.ThisalsomeansthatyoucansafelyignoretheAnalysis ManagementObjectswarning(AMO)inVisualStudiothatpartitionsizesshouldnotexceed20million rows.Wehavemeasuredtheeffectoflargepartitionsizes,andfoundthattheyshownegligible performancedifferencescomparedtosmallerpartitionsizes.Therefore,thereductioninpartition managementoverheadjustifiesrelaxingtheguidelinesonpartitionsizes,particularlywhenyouhave largenumbersofpartitions. 3.5.3.5.1 Adding and Managing Partitions Managingpartitionsonalargecubequicklybecomesalargeadministrativetaskifdonemanually throughSQLServerManagementStudio.Werecommendthatyoucreatescriptstohelpyoumanage partitioningofthecubeinanautomatedmannerinstead. Thebestwaytomanagealargecubeistousearelationaldatabasetoholdthemetadataaboutthe desiredpartitioningschemeofthecubeanduseXMLAtokeepthecubestructureinsyncwiththe 167

metadataintherelationalsource.Everytimeyourequestmetadatafromacube,youhavetoruna DISCOVERcommandsomeofthesecommandsareexpensiveonalargecubewithmanypartitions. Youshoulddesignpartitionmanagementcodetoextractthemetadatafromthecubeinasinglebulk operationinsteadofmultiplesmallDISCOVERcommands.NotethatwhenyouuseAMO(andtheSQL ServerIntegrationServicestask)toaccessthecube,someamountoferrorhandlingisbuiltintothe.NET library.ThiserrorhandlingcodeexecutesDISCOVERcommandsandbequitechattywiththeserver.A goodexampleofthisisthepartitionaddfunctioninAMO.Thiscodewillfirstchecktoseewhetherthe partitionalreadyexists(itraisesanerrorifitdoes)usingaDISCOVERcommand.Afterthatitwillissue theCREATEcommand,creatingthepartitioninthecube.Thistechniqueturnsouttobeahighly inefficientwaytoaddmanynewpartitionsoncubewiththousandsofexistingpartitions.Instead,itis fastertofirstreadallthepartitionsusingasingleXMLAcommand,discoverwhichpartitionsaremissing ,andthenrunanXMLAcommandtocreateeachmissingpartitiondirectlyavoidingtheerrorchecking thatAMO.NETdoes. Insummary,designpartitionmanagementcodecarefullyandtestitusingSQLServerProfilertotrace thecommandsyougenerateasthecoderuns.Becauseitprovidessomuchfeedbacktotheserver, partitionmanagementcodeisveryexpensive,anditisoftensurprisingtocustomershoweasyitisto createinefficientoperationsonabigcube.ThatisthepriceofthesafetynetthatADOMD.NETgives you.UsingXMLAtocarefullymanagepartitionsisoftenabettersolutionforalargecube.

3.5.4 Locking and Blocking

Locksareusedtomanageconcurrentoperations.InanAnalysisServicesdeployment,youllfindthat locksareusedpervasivelyindiscoveryandexecutiontoensurethestabilityofunderlyingdata structureswhileaqueryorprocessisrunning. Occasionally,lockstakenforoneoperationcanblockotherprocessesorqueriesfromrunning.A commonexampleiswhenaquerytakesalongtimetoexecute,andanothersessionissuesawrite operationinvolvingthesameobjects.Thequeryholdslocks,causingthewriteoperationtowaitforthe querytocomplete.Additionalnewqueriesissuedonothersessionsarethenblockedastheywaitfor theoriginalqueryoperationtocomplete,potentiallycausingtheentireservicetostophandling commandsuntiltheoriginallongrunningquerycompletes. Thissectiontakesacloserlookathowlocksareusedindifferentscenarios,andhowtodiagnoseand addressanyperformanceproblemsorservicedisruptionsthatarisefromlockingbehaviors.Oneofthe toolsatyourdisposalisthenewSQLServerProfilerlockeventsthatwereintroducedinSQLServer2008 R2ServicePack1(SP1).Finally,thissectionlooksatdeadlocksandprovidessomerecommendationsfor resolvingthemiftheybegintooccurtoofrequently.

3.5.4.1 Lock Types

InAnalysisServices,themostcommonlyusedlocktypesareRead,Write,Commit_Read,and Commit_Write.Theselocksareusedfordiscoveryandexecutionontheserver.SQLServerProfiler, queries,anddynamicmanagementviews(DMVs)usethemtosynchronizetheirwork.Otherlocktypes 168

(suchasLOCK_NONE,LOCK_SESSION_LOCK,LOCK_ABORTABLE,LOCK_INPROGRESS,LOCK_INVALID)are usedperipherallyashelperfunctionsforconcurrencymanagement.Becausetheyareperipheraltothis discussion,thissectionfocusesonthemainlocktypesinstead. Thefollowingtableliststheselocksandbrieflydescribeshowtheyareused. Locktype Read Object All Usage Readlocksareusedforreadingmetadataobjects,ensuringthat theseobjectscannotbemodifiedwhiletheyarebeingused. Readlocksareshared,meaningthatmultipletransactionscantake Readlocksonthesameobject. Writelocksareusedforcreate,alter,andupdateoperations. Writelocksareexclusive.Exclusivelockspreventconcurrent transactionsfromtakingReadorWritelocksontheobjectatthe sametime. Commit_Readlocksareshared,buttheyblockwriteoperations thatarewaitingtocommitchangestodisk. DatabaseObjects:Commit_Readlocksareusedforthefollowing: Queryoperationstoensurethatnocommittransactions overwriteanyofthemetadataobjectsusedinthequery. Thelockisheldforthedurationofaquery. Duringprocessing,whileacquiringReadandWritelockson otherobjects. Atthebeginningofasessiontocalculateuserpermissions foragivendatabase. Duringsomediscoveroperations,suchasthosethatread fromadatabaseobject. ServerObjects:Commit_Readlocksareusedatthebeginningofa sessiontocomputesessionsecurity.Commit_Readisalsousedat thestartofatransactiontoreadthemasterversionmap. ServerProxyObjects:Commit_Readlocksareusedatthe beginningofSQL,DMX,andMDXqueriestopreventchangesto assembliesoradministrativerolewhiletheobjectsareretrieved. Commit_Writelocksareexclusivelocksonanobject,takento preventaccesstothatobjectwhileitisbeingupdated. Commit_Writeistheprimarymechanismforensuringoneversion ofthetruthforqueries.

Write

All

Commit_Read

Database, Server, ServerProxy

Commit_Write

Database, Server, ServerProxy

169

DatabaseObjects:ACommit_Writeisusedinacommittransaction thatcreates,updates,ordeletesDDLstructuresinthedatabase.In acommittransaction,Commit_Writelockscanbetakenon multipledatabasesatthesametime,assumingthetransaction includesthem(forexample,deletingmultipledatabasesinone transaction). ServerObject:ACommit_Writeisusedtoupdatethemaster versionmap.Moreinformationaboutthemasterversionmapisin thenextsectiononserverlocks. ServerProxyObject:ACommit_Writeistakenwheneverthereare changestoassembliesormembershipoftheserverrole.

3.5.4.2 Server Locks

InAnalysisServices,therearetwoserverlockobjects,theServerandtheServerProxy,eachusedfor veryspecificpurposes.

Figure 66 - Server Lock Hierarchies

TheServerobjecttakesacommitlockforoperationssuchasprocessingorqueryingthattraversethe objecthierarchy.Typically,commitlocksontheserverareverybrief,takenprimarilywheneverthe serverreadsthemasterversionmap(Master.vmp)fileorupdatesitsothatitcontainsnewerversionsof objectschangedbyatransaction. Themasterversionmapisalistofobjectidentifiersandcurrentversionnumberforallofthemajor objectsrecognizedbytheserver(thatis,Database,Cube,MeasureGroups,Partitions,Dimensionsand Assemblies).Minorobjects,suchashierarchylevelsorattributes,donotappearinthelist.Themaster versionmapidentifieswhichobjectversiontouseatthestartofaqueryorprocess.Theversionnumber ofamajorobjectchangeseachtimeyouprocessorupdateit.OnlytheServerobjectreadsandwritesto theMaster.vmpfile. Whenreadingthemasterversionmap,theservertakesaCommit_Readtoprotectagainstchangesto thefilewhileitisbeingread.ACommit_WriteistakenontheServerobjecttoupdatethemaster versionmap,bymergingnewerversioninformationthatwascreatedinatransaction. 170

TheServerProxyobjectisusedforadministrativelocks,forexamplewhenyoumakechangestothe membershipoftheServerroleortoassembliesusedbythedatabase.

3.5.4.3 Lock Fundamentals

ThissectionexploressomeofthefoundationalconceptsthatdescribelockingbehaviorinAnalysis Services.Likealllockingmechanisms,thepurposeoflocksinAnalysisServicesistoprotectmetadataor datafromtheeffectsofconcurrency. Therulesorprinciplesofhowlocksareusedcanbedistilledintothefollowingpoints: Locksprotectmetadata;latchesprotectdata.Latchesarelightweightlocksthatarerapidly acquiredandreleased.Theyareusedforatomicoperations,likereadingadatavalueoutof systemdata. Readlocksaretakenforobjectsthatfeedintoatransaction;Writelocksaretakenfor objectsthatarechangedbythetransaction.Throughouttherestofthissection,wellsee howthisprincipleisappliedindifferentoperations.

Considerthecaseofdimensionprocessing.Objectslikedatasourcesanddatasourceviewsthatprovide datatothedimension(thatis,objectsthatthedimensiondependson)takeaReadlock.Objectsthatare changedbytheoperation,suchaspartitions,takeaWritelock.Whenadimensionisprocessed,the partitionsneedtobeunprocessedandthenreprocessed,soaWritelockistakenonthepartitionsto updatethedimensionrelateddatawithineachpartition.

Figure 67 - Locks during dimension processing

Comparetheprecedingillustrationtothefollowingone,whichshowsonlyReadlocksusedforpartition processing.Becausenomajorobjectsdependonapartition,theonlyWritelockinplayisonthe partitionitself.

171


Figure 68 - Locks during partition processing

3.5.4.4 Locks by Operation

Thissectionexplainsthelockstakenbydifferentoperationsontheserver.Thislistoflocksisnot exhaustive: BeginSession Queries Discover Processing DDLoperations Rollback

Otheroperationssuchaslazyprocessing,sessions,andproactivecachingposeinterestingchallengesin termsofunderstandinglocks.Thereasonisdiscussedattheendofthissection,buttheseother operationsarenotdiscussedindetail.Alsomissingfromthelistiswriteback,Becausewritebackisa hybridofqueryandprocessoperations,whicharecoveredindividually. 3.5.4.4.1 Begin Session Whenasessionstarts,AnalysisServicesdetermineswhichdatabasestheuserhaspermissiontouse. PerformingthistaskrequirestakingaCommit_Readlockonthedatabase.Ifadatabaseisnotspecified, sessionsecurityiscomputedforeachdatabaseontheserveruntiladatabaseisfoundthatthesession hasreadaccessto.Locksarereleasedaftersessionsecurityiscomputed.Occasionally,thecommitlock isacquiredandreleasedsoquicklythatitnevershowsupinatrace. 3.5.4.4.2 Queries Allqueriesrunonadatabaseandinasession.ACommit_Readlockistakensimultaneouslyonthe ServerProxyobjectanddatabasewhenthequerystarts.Thelockisheldforthedurationofthequery. TheCommit_Readlockonadatabaseheldbyqueryissometimesblocksprocessingoperations.

172


Figure 69 - Write locks blocking queries

Intheprecedingexample,thelongrunningquerypreventstheCommit_Writelockrequiredbythe processingoperation(yellowarrow)frombeingtaken.NewreadersrequiringtheCommit_Readlockon thedatabasethenbecomeblockedbehindtheprocessingcommand.Itisthiscombinationofevents thatcanturnathreesecondqueryintoathreeminute(orlonger)query. 3.5.4.4.3 Discovers Therearetwotypesofdiscoveroperationsthatuselocks:thosethatrequestmetadataaboutinstances orobjects(suchasMDSCHEMA_CUBE),andthosethatarepartofadynamicmanagementview(DMV) query. TheDiscovermethodonmetadataobjectsbehavesverysimilarlytoaqueryinthatittakesa Commit_Readlockonthedatabase. DiscoversissuedforDMVoperationsdonttakeReadlocks.Instead,DMVsuseinterlockingmethodsto synchronizeaccesstosystemdatathatiscreatedandmaintainedbytheserver.Thisapproachis sufficientbecauseDMVdiscoversdonotreadfromthedatabasestructure.Assuch,theprotection offeredbylocksisoverkill.ConsiderDISCOVER_OBJECT_MEMORY_USAGE.ItisaDMVquerythat returnsshrinkableandunshrinkablememoryusedbyallobjectsontheserveratgivenpointintime. Becauseitisafastreadofsystemdatadirectlyontheserver,DISCOVER_OBJECT_MEMORY_USAGEuses interlockedaccessovermemoryobjectstoretrievethisinformation. 3.5.4.4.4 Processing Forprocessing,locksareacquiredintwophases,firstduringobjectacquisitionandagainduring scheduleprocessing.Afterthesefirsttwophases,thedatacanbeprocessedandthechanges committed. Phase1:ObjectAcquisition 1. ACommit_Readlockisacquiredonthedatabasewhiletheobjectsareretrieved. 2. Theobjectsusedinprocessingarelookedup. 3. Theobjectsareidentified,andthentheCommit_Readlockonthedatabaseisreleased. Phase2:ScheduleProcessing Thisphasefindstheobjectsonwhichthedimensionorpartitiondepends,aswellasthoseobjectsthat dependonit.Inpractice,buildingthescheduleisaniterativeprocesstoensurethatallofthe dependenciesareidentified.Afterallofthedependenciesareunderstood,theexecutionworkflow 173

movesforwardtotheprocessingphase.Thefollowinglistsummarizestheeventsinschedule processing. 1. Schedulerbuildsadependencygraphthatidentifiesalloftheobjectsinvolvedintheprocessing command.Basedonthisgraph,itbuildsascheduleofoperations.

2. ScheduleracquiresaCommit_Readonthedatabase,andthenitacquiresReadandWritelocks ontheobjects. 3. Schedulergeneratesthejobsandisnowfreetostartexecutingthejobs. Phase3:Process Afterscheduling,therearenoCommit_Readlocks,justWritelocksontheaffectedobjects,andread locksonobjects.OnlyReadandWritelocksareusedtodothework.ReadandWritelockspreventother transactionsfromupdatinganyobjectsusedinthetransaction.Atthesametime,objectsthatare relatedbutnotincludedinthecurrenttransactioncanbeupdated.Forexample,supposetwodifferent dimensionsarebeingprocessedinparallel.Ifeachdimensionisreferencedbydifferentcubesandif eachdimensionisfullyindependentoftheother,thereisnoconflictwhenthetransactioniscommitted. HadaCommit_Writelockbeenheldonthedatabase,thistypeofparallelprocessingwouldnotbe possible. Phase4:Commit WhenaCommittransactionstarts,aCommit_Writelockistakenonthedatabaseandheldforthe durationofthetransaction.NonewCommit_ReadlocksareacceptedwhileCommit_Writeispending. AnynewBeginSessionsmayencounteraconnectiontimeout.Newqueriesarequeuedorcanceled, dependingontimeoutsettings. TheCommittransactionwaitsforthequeryqueuetodrain(thatis,itwaitsforCommit_Readlockstobe released).Atthispoint,readingfromthepropertysettingsdefinedontheserver,itmightuseoneofthe followingpropertiestocancelaqueryifitistakingtoolong: 174

ForceCommitTimeoutcancelstransactionsthatholdCommit_Readlocks.Bydefault,this propertyissetto30seconds.However,itisimportanttorememberthatcancelationsarenot alwaysinstantaneous.Sometimesittakesseveralminutestoreleasecommitlocks. CommitTimeoutcancelstransactionsrequestingCommit_Writelocks,ineffectprioritizing queriesoverprocessing.Theserveruseswhichevertimeoutoccursfirst.If ForceCommitTimeoutoccurssoonerthanCommitTimeout,cancelationiscalledonlong runningqueriesinsteadofthewriteoperation. Note:Iftheclientapplicationhasitsownretrylogic,itcanreissueacommandinresponsetoa connectiontimeoutandtheconnectionwillsucceedifthereareavailablethreads. AcommittransactionalsotakesaCommit_Writelockontheserver.Thisisveryshortandoccurswhen thetransactionversionmapismergedintothemasterversionmapthatkeepstrackofwhichobject versionsarethemasterversions.BecausethisisaWritelock,itmustwaitforReadlockstoclearthis waittimeiscontrolledbyForceCommitTimeoutandCommitTimeout.Acommittransactioncreates newversionsofthemasterobjects,ensuringconsistentresultsforallqueries. AftertheMaster.vmpfileisupdated,theserverdeletesunusedversiondatafiles.Atthispoint,the Commit_Writelockisreleased. 3.5.4.4.5 DDL Operations Foroperationsthatcreate,alter,ordeletemetadataobjects,locksareacquiredonce.Itissimilartothe processingworkflowbutwithouttheschedulingphase.Itfindstheobjectsthattheobjectdependson andlocksthemusingReadlocks,andWritelocksaretakenontheobjectsthatdependontheobject thatisbeingmodified.Writelocksarealsoacquiredontheobjectsthatarecreated,updated,ordeleted inthetransaction. 3.5.4.4.6 Rollback Forrollback,thereisaserverlatchthatprotectsMaster.vmp,butarollbackbyitselfdoesnottakea Commit_ReadorCommit_Writelockonthedatabase.Inarollback,nothingischanging,sonocommit locksarerequired.However,anyReadandWritelocksthatweretakenbythetransactionarestillheld duringtherollback,buttheyarereleasedaftertherollbackhasbeencompleted.Rollbackdeletesthe unusedversionsofanyobjectscreatedbytheoriginaltransaction. 3.5.4.4.7 Session Transactions, Proactive Caching, and Lazy Processing Intermsoflocking,sessions,proactivecachingandlazyprocessingaretrickybecausetheyhave breakablelocks.Pendingawriteoperationfromanadministrator,LockManagercancelsWritelocksina session,proactivecaching,orlazyprocessing,anditletstheWriteCommitprevail. SessionscanhaveWritelocksthatdontconflict.Snapshotsandcheckpointsareusedtomanagewrite operationsforsessioncubes.TheclassicexampleisthegroupingbehaviorinExcelwherenew dimensionsarecreatedinsession,onthefly,tosupportadhocdatastructures.Thenewdimensions arealwaysrolledbackeventually,buttheycanbeproblematicduringtheirlifetime.Ifanerroroccursin

175

session,theservermightperformapartialrollbacktoachieveastablestate;sometimestheserollback operationshaveunintendedconsequences. 3.5.4.4.8 Synchronization, Backup and Restore, and Attach Thelockingmechanismforsynchronization,restore,andattacharealreadydocumentedintheAnalysis ServicesSynchronizationBestPracticesarticleontheSQLCATwebsite(http://sqlcat.com).Restated fromthatarticle,thebasicworkflowoflockacquisitionandreleaseforsynchronizationisasfollows. Duringsynchronization,aWritelockisappliedbeforethemergeofthefilesonthetargetserver,anda readcommitlockisappliedtothesourcedatabasewhilefilesaretransferred. TheWritelockisappliedonthetargetserver(whenatargetdatabaseexists),preventingusers fromqueryingand/orwritingindatabase.Thelockisreleasedfromthetargetdatabaseafter themetadataisvalidated,themergehasbeenperformedandthetransactioniscommitted. Thereadcommitlockistakenonthesourceservertopreventprocessingfromcommittingnew data,butitallowsqueriestorun,includingothersourcesynchronization.Becauseofthis, multipleserverscanbesynchronizedatthesametimefromthesamesource.Thelockis releasedataboutthesamemomentastheWritelock,asitistakeninthisdistributed transaction.

Forsynchronizationonly,thereisalsoanadditionalCommit_Writelockwhilethedatabasesaremerged. Thislockcanbeheldforalongperiodoftimewhilethedatabaseiscreatedandoverwritten. Forbackup,thereisonlyaCommit_Readonthedatabase. Arestoreoperationusesamorecomplexseriesoflocks.ItacquiresaWritelockonthedatabasethatis beingrestored,ensuringtheintegrityofthedatabaseasitsbeingrestored,butblockingquerieswhile Restoreisbeingprocessed.Bestpracticerecommendationssuggestusingtwodatabaseswithdifferent namessothatyoucanminimizequerydowntime. Thebasicworkflowforalloftheseoperationsisasfollows: 1. Writelocksaretakenonthedatabasethatisbeingsynchronized,restored,orattached. 2. Filesareextractedandchangescommittedwhileprocessing. 3. Thelocksarereleased.

3.5.4.5 Investigating Locking and Blocking on the Server

Thissectiondescribesthetoolsandtechniquesformonitoringlocksandremovinglockingproblems. OneofthetoolsdiscussedisthenewlockeventsthatareintroducedinSQLServer2008R2SP1.The othertoolisDMVs(andtheDISCOVER_LOCKSschemarowsetinparticular),whicharediscussedwith emphasisonhowitfitsintoatroubleshootingscenario. Lockingandblockingproblemsmanifestthemselvesinaserverenvironmentindifferentways.Aswith allperformanceproblems,thisoneisamatterofdegree.Youmighthaveablockingissueinyour environmentthatresolvessoquickly,itneverregistersasaproblemthatrequiresasolution.Atthe 176

otherendofthespectrum,blockingcanbecomesoseverethatusersarelockedoutofthesystem. Connectionrequestsfail;querieseithertimeoutorfailtostartaltogether. Intheseextremescenarios,itisdifficulttoclearlydiagnosetheproblemifyoudonothaveanavailable threadtoconnectSQLServerProfilerorifyouareunabletorunaDMVquerythattellsyouwhatis goingon.Inthiscase,theonlywaytoconfirmalockingproblemistoworkwithaMicrosoftsupport engineertoanalyzeamemorydump.Theengineercantellyouwhetheryourserverunavailabilityisdue tolocks,indicatedbyPCLockManager::Waitonmultiplethreads. 3.5.4.5.1 Using DMV Queries and XMLA to Cancel a Blocked Transaction Ifyoursituationislessdire,youcanuseaDMVqueryandXMLforAnalysis(XMLA)tounblockyour server.Givenafreethreadforprocessinganewconnectionrequest,youcanconnecttotheserverusing anadministratoraccount,runaDMVquerytogetthelistoflocks,andthentrytokilltheblockingSPID byrunningthisXMLAcommand.
<Cancel xmlns="http://schemas.microsoft.com/analysisservices/2003/engine"> <SPID>nnnn</SPID> </Cancel>

TogettheSPID,youcanuseSQLServerManagementStudiotoconnecttotheserverandthenissuea DISCOVER_LOCKSstatementasanMDXquery.
Select * from $SYSTEM.DISCOVER_LOCKS

ThisDMVqueryreturnsasnapshotofthelocksusedatspecificpointintime.DISCOVER_LOCKSreturns arowsetdirectlyfromLockManager.Assuch,thedatayougetbackmightnotbeasclearoreasyto followasthetraceeventsinSQLServerProfiler.Thefollowingexamplecontainsasnapshotofthelocks heldduringaquerybutnoinformationaboutthequeryitself.

Figure 70 - Output of $system.discover_locks

Unfortunately,thereisnomitigationforthis.YoucannotrunDISCOVER_LOCKSinconjunctionwith otherDMVstatementstogetadditionalinsightintothetimingofthelockacquisitionreleaselifecycle

177

relativetothetransactionsrunningonyourserver.YoucanonlyruntheMDXSELECTstatementand thenactontheinformationitprovides. 3.5.4.5.2 Using SQL Server Profiler to Analyze Lock Contention SQLServerProfileroffersnumerousadvantagesoverDMVqueriesintermsofdepthandbreadthof information,butitcomesatacost.ItisoftennotfeasibletorunSQLServerProfilerinaproduction environmentbecauseoftheadditionalresourcedemandsitplacesonserver.Butifyoucanuseit,andif youarerunningSQLServer2008R2SP1,youcanaddthenewLockAcquired,LockReleased,andLock Waitingeventstoatracetounderstandthelockingactivityonyourserver. SQLServer2008R2SP1addsthefollowingeventstotheLockseventcategoryinSQLServerProfiler: LockAcquired,LockReleased,LockWaiting. Likeotherlockevents,theyarenotenabledbydefault.YoumustselecttheShowallcolumnscheckbox beforeyoucanselecteventsintheLockscategory.Togetanideaofhowlocksareacquiredand releasedinthecourseofatransactionbesuretoaddCommandBeginandCommandEndtothetrace.If youwanttoviewtheMDXexecuted,youshouldalsoincludetheQueryBeginandQueryEnd. InAnalysisServices,itisnormaltoseealargenumberofacquiredandreleasedlockevents.Every transactionthatincludesReadorWriteoperationsonamajorobjectrequestsalocktoperformthat action.LockWaitingislesscommon,butbyitselfisnotsymptomaticofaproblem.Itmerelyindicates thataqueuehasformedfortransactionsthatarerequestingthesameobject.Ifthequeueisnotlong andoperationscompletequickly,thequeuedrainsandqueryorprocessingtasksproceedwithonlya smalldelay. ThefollowingillustrationshowsaLockWaitingeventandtheXMLstructuresthatidentifywhich transactioncurrentlyholdsalockontheobject,andwhichtransactionsarewaitingforthatsameobject. Intheillustration: <HoldList>showswhichtransactioniscurrentlyholdingaCommit_Writelockonthedatabase. <WaitList>indicatesthatanothertransactioniswaitingforaCommit_Writelockonthesame database.

178


Figure 31 SQL Server Profiler output of lock events

3.5.4.6 Deadlocks
AdeadlockinAnalysisServicesislockcontentionbetweentwoormoresessions,whereoperationsin eithersessioncantmoveforwardbecauseeachiswaitingtoacquirealockheldbytheothersession. AnalysisServiceshasdeadlockdetectionbutitonlyworksforlocks(Read,Write,Commit_Read, Commit_Write).Itwontdetectdeadlocksforlatches.Whenadeadlockisdetected,AnalysisServices stopsthetransactioninoneofthesessionssothattheothertransactioncancomplete. Deadlocksaretypicallyaboundarycase.Ifyoucanreproduceit,youcanrunaSQLServerProfilertrace andusetheDeadlockeventtodeterminethepointofcontention.InSQLServerProfiler,adeadlock lookslikethefollowing:

179


Figure 71 - Deadlock events in profiler

AdeadlockeventusesanXMLstructurecalledadeadlockgraphtoreportonwhichsessions, transactions,andobjectscreatedtheevent.The<VICTIM>nodeinthefirstfewlinesidentifieswhich transactionwassacrificedforthepurposeofendingthedeadlock.Theremainderofthegraph enumeratesthelocktypeandstatusforeachobject.Inthefollowingexample,youcanseewhich objectsarerequestedandwhetherthelockwasgrantedorwaiting.

<DeadlockGraph> <VICTIM> <LOCK_TRANSACTION_ID>E0BF8927-F827-4814-83C1-98CA4C7F5413</LOCK_TRANSACTION_ID> <SPID>29945</SPID> </VICTIM> <LOCKS> <Lock>s <LOCK_OBJECT_ID><Object><DatabaseID>FoodMart 2008</DatabaseID><DimensionID>Promotion</DimensionID></Object></LOCK_OBJECT_ID> <LOCK_ID>326321DF-4C08-43AA-9AEC-6C73440814F4</LOCK_ID> <LOCK_TRANSACTION_ID>E0BF8927-F827-4814-83C1-98CA4C7F5413</LOCK_TRANSACTION_ID> <SPID>29945</SPID> <LOCK_TYPE>2</LOCK_TYPE> ---- Lock_Type 2 is a Read lock <LOCK_STATUS>1</LOCK_STATUS> ---- Lock_Status 1 is acquired </Lock> <Lock> <LOCK_OBJECT_ID><Object><DatabaseID>FoodMart 2008</DatabaseID><DimensionID>Product</DimensionID></Object></LOCK_OBJECT_ID> <LOCK_ID>21C7722B-C759-461A-8195-1C4F5A88C227</LOCK_ID> <LOCK_TRANSACTION_ID>E0BF8927-F827-4814-83C1-98CA4C7F5413</LOCK_TRANSACTION_ID> <SPID>29945</SPID> <LOCK_TYPE>2</LOCK_TYPE> ---- Lock_Read <LOCK_STATUS>0</LOCK_STATUS> ---- waiting on Product, which is locked by 29924 </Lock>

180

<Lock> <LOCK_OBJECT_ID><Object><DatabaseID>FoodMart 2008</DatabaseID><DimensionID>Promotion</DimensionID></Object></LOCK_OBJECT_ID> <LOCK_ID>7F15875F-4CCB-4717-AE11-5F8DD48229D0</LOCK_ID> <LOCK_TRANSACTION_ID>1D3C42F3-E875-409E-96A0-B4911355675D</LOCK_TRANSACTION_ID> <SPID>29924</SPID> <LOCK_TYPE>4</LOCK_TYPE> ---- Lock_Write <LOCK_STATUS>0</LOCK_STATUS> ---- waiting on Promotion, which is locked by 29945 </Lock> <Lock><LOCK_OBJECT_ID><Object><DatabaseID>FoodMart 2008</DatabaseID><DimensionID>Product</DimensionID></Object></LOCK_OBJECT_ID> <LOCK_ID>96B09D08-F0AE-4FD2-8703-D33CAD6B90F1</LOCK_ID> <LOCK_TRANSACTION_ID>1D3C42F3-E875-409E-96A0-B4911355675D</LOCK_TRANSACTION_ID> <SPID>29924</SPID> <LOCK_TYPE>4</LOCK_TYPE> ---- Lock_Write <LOCK_STATUS>1</LOCK_STATUS> ---- granted </Lock>< /LOCKS> </DeadlockGraph>

Deadlocksshouldberareevents,iftheyoccuratall.Persistentdeadlocksindicateaneedforredesigning yourprocessingstrategy.Youmightneedtospeedupprocessingbymakinggreateruseofpartitions,or youmightneedtotakeacloserlookatotherprocessingoptionsorschedulestoseewhetheryoucan eliminatetheconflict.Formoreinformationabouttheserecommendations,seePart1ofthisbookor theAnalysisServicesPerformanceGuide (http://www.microsoft.com/downloads/en/details.aspx?FamilyID=3be0488de7aa4078a050 ae39912d2e43&displaylang=en). References: AnalysisServicesSynchronizationBestPractices http://sqlcat.com/technicalnotes/archive/2008/03/16/analysisservicessynchronizationbest practices.aspx DeadlockTroubleshootinginSQLServerAnalysisServices(SSAS) http://blogs.msdn.com/b/sql_pfe_blog/archive/2009/08/27/deadlocktroubleshootinginsql serveranalysisservicesssas.aspx SSAS:Processing,ForceCommitTimeoutand"theoperationhasbeencancelled" http://geekswithblogs.net/darrengosbell/archive/2007/04/24/SSASProcessing ForceCommitTimeoutandquottheoperationhasbeencancelledquot.aspx LockingandUnlockingDatabases(XMLA)http://msdn.microsoft.com/en us/library/ms186690.aspx

181

3.5.5 Scale Out

AnalysisServicescurrentlysupportsupto64coresinascaleupconfiguration.Ifyouwanttogobeyond thatscale,youwillhavetodesignforscaleout.Therearealsootherdriversforscaleout;forexample,it maysimplybecheapertousemultiple,smallermachinestoachievehighuserconcurrency.Another considerationisprocessingandqueryworkloads.Ifyouexpecttospendalotoftimeprocessingorifyou aredesigningforrealtime,itisoftenusefultoscaleouttheprocessingonadifferentserverthanthe queryservers. Scaleoutarchitecturescanalsobeusedtoachievehighavailability.Ifyouhavemultiplequeryserversin ascaleoutfarm,someofthemcanfailbutthesystemwillremainonline. AlthoughthefulldetailsofdesigningascaleoutAnalysisServicesfarmisoutsidethescopeofthisbook, itisusefultounderstandthetradeoffsandpotentialarchitecturesthatcanbeapplied.

3.5.5.1 Provisioning in a Scaled Out Environment

InascaleoutenvironmentyoucanuseeitherreadonlyLUNandtheattachanddetachfunctionalityof SQLServer2008AnalysisServicesorSANsnapshots(whichcanalsobeusedinSQLServer2005Analysis Services)tohostmultiplecopiesofthesamedatabaseonmanymachines. Whenyouupdateascaleoutreadonlyfarm,aWindowsvolumehastobedismountedandmounted everytimeyouupdateanAnalysisServicesdatabase.Thismeansthatnomatterwhichdisktechnology youusethescaleout,thesmallestunityoucanupdatewithoutdisturbingothercubesisaWindows volume.Ifyouhavemultipledatabasesinthescaleoutenvironment,itisthereforeanadvantageto haveeachdatabaseliveonitsownWindowsvolume.Thisseparationallowsyoutoupdatethe databasesindependentlyofeachotherifyouarerunningSQLServer2008AnalysisServicesandSQL Server2008R2AnalysisServices.

3.5.5.2 Scale-out Processing Architectures

Therearebasicallythreedifferentarchitecturesyoucanuseinascaleoutconfiguration: Dedicatedprocessingarchitecture Query/processingflippingarchitecture ROLAP

Thethreearchitectureshavedifferenttradeoffsthatyouhavetoconsider,whichthissectiondescribes. Note:Itisalsopossibletocombinethesearchitecturesindifferenthybrids,butthatisoutsidethescope ofthisdocument.Understandingthetradeoffsforeachwillhelpyoumaketherightdesigndecisions. 3.5.5.2.1 Dedicated Processing Architecture Inthededicatedprocessingarchitecture,aninstanceofAnalysisServicesisreservedtoprocessallnew, incomingdata.Afterprocessingisdone,theresultiscopiedtoqueryservers.Theadvantageofthis architectureisthatthequeryserverscanrespondtothequerieswithoutbeingaffectedbythe processingoperation.Alockisrequiredonlywhendataisupdatedoraddedtothecube. 182

NetworkandLoadBalancer

Query InstanceA

Query InstanceB

Query InstanceC

Co py

Figure 72 - Dedicated Processing Architecture

Inadedicatedprocessingarchitecture,considerhowtogetthedatafromtheprocessinginstancetothe queryservers.Thereareseveralwaystoachievethis. AnalysisServicesCubeSynchronization:ByusingthisbuiltinAnalysisServicesfunctionality,youcan movethedeltadatadirectlytothequeryservers. RobocopyorNiceCopy:Byusingahighspeedcopyingprogram,youcanquicklysynchronizeeachquery instancewithitsowncopyofthechangeddata.Thismethodisgenerallyfasterthancube synchronization,butitrequiresyoutosetupyourowncopyscripts. SANSnapshotsorStorageMirrors:UsingSANtechnology,itispossibletoautomaticallymaintaincopies ofthedataLUNontheprocessingservers.Thesecopiescanthenbemountedonthequeryservers whenthedataisupdated. SANReadOnlyLUN:Usingthistechnique,whichisavailableonlyinSQLServer2008andSQLServer 2008R2,youcanusereadonlyLUNtomovethedatafromtheprocessinginstancetothequeryservers. AreadonlyLUNcanbesharedbetweenmultipleservers,andhenceenablesyoutousemorethan queryserveronthesamephysicaldisk. BothSANsnapshotsandreadonlyLUNstrategiesmayrequirecarefuldesignofthestoragesystem bandwidth.Ifyourcubeissmallenoughtofitinmemory,youwillnotseemuchI/Oactivityandthis 183

py Co

Process

Processing Instance

SourceData

techniquewillworkverywelloutofthebox.However,ifthecubeislargeandcannotfitinmemory, AnalysisServiceswillhavetodoI/Ooperations.Asyouaddmoreandmorequeryserverstothesame SAN,youmayendupcreatingabottleneckinthestorageprocessorsontheSANtoserveallthisI/O. YoushouldmakesurethattheSANiscapableofsupportingtherequiredthroughput.IftheI/O throughputisnotsufficient,youmayendupwithascaleoutsolutionthatperformsworsethanascale up. IfyouareworriedaboutI/Obandwidth,theRobocopy/NiceCopysolutionorthecubesynchronization solutionmayworkbetterforyou.Inthesesolutionsyoucanhavededicatedstorageoneachquery server.However,youhavetomakesurethereisenoughbandwidthonthenetworktorunmultiple copiesoverthenetwork.Youmayhavetousededicatednetworkcardsforsuchasetup. Thededicatedprocessingarchitecturescanalsobeusedtoachievehighavailability.However,youneed awaytoprotecttheprocessingserver,toavoidasinglepointoffailure.Youcanhaveeitherastandby processingserveroranextra(disabled)instanceononeofthequeryserversthatcanbeusedtotake overtheroleofprocessingserviceinthecaseofhardwarefailure.Anotheralternativeistouse clusteringontheprocessingserver,althoughthismaywastehardwareresourcesonthepassivenode. References: ScaleOutQueryingforAnalysisServiceswithReadOnlyDatabases http://sqlcat.com/whitepapers/archive/2010/06/08/scaleoutqueryingforanalysisservices withreadonlydatabases.aspx SampleRobocopyScripttoSynchronizeAnalysisServicesDatabases http://sqlcat.com/technicalnotes/archive/2008/01/17/samplerobocopyscripttocustomer synchronizeanalysisservicesdatabases.aspx ScaleOutQueryingwithAnalysisServices http://sqlcat.com/whitepapers/archive/2007/12/16/scaleoutqueryingwithanalysis services.aspx ScaleOutQueryingwithAnalysisServicesUsingSANSnapshots http://sqlcat.com/whitepapers/archive/2007/11/19/scaleoutqueryingwithanalysisservices usingsansnapshots.aspx AnalysisServicesSynchronizationBestPractices http://sqlcat.com/technicalnotes/archive/2008/03/16/analysisservicessynchronizationbest practices.aspx SQLVelocityScalableSharedDatabase http://sqlvelocity.typepad.com/blog/2010/09/scalableshareddatabasepart1.html

3.5.5.2.2 Query/Processing Flipping Architecture Thededicatedprocessingserverarchitecturesolvesmostscaleoutcases.However,thetimerequiredto movethedatafromtheprocessingservertothequeryserversmayberestrictiveifupdateshappenat intervalsshorterthanafewhours.EvenwithSANsnapshotsorreadonlyLUN,itwillstilltakesometime todismounttheLUN,setitonlineonthequeryserver,andfinallymounttheupdatedcubeonthequery 184

servers.Inthequery/processingflippingarchitecture,eachinstanceofAnalysisServicesperformsits ownprocessing,asillustratedinthefollowingfigure.

Process

Pr oc es s

Figure 73 - Query and processing flipping

Becauseeachserverdoesitownprocessing,itispossiblethatsomeserverswillhavemorerecentdata thanothers.Thismeansthatoneuserexecutingaquerymaygetalaterversionofthedatathan anotheruseexecutingthesamequeryconcurrently.However,formanyscenarioswhereyouaregoing nearrealtime,suchastateofloosesynchronizationmayisanacceptabletradeoff.Ifthetradeoffisnot acceptable,youcanworkarounditwithcarefulloadbalancingatthepriceofaddingsomeextra latencytotheupdates. Intheprecedingdiagram,youcanseethatthesourcesystemreceivesmoreprocessingrequeststhanin thededicatedprocessingarchitecture.Youshouldscalethesourceaccordinglyandconsiderthe networkbandwidthrequiredtoreadthesourcedatamorethanonce.

s es oc Pr

185

Thequery/processingflippingarchitecturealsohasabuildinhighavailabilitysolution.Ifoneofthe serversfails,thesystemcanremainonlinebutwithadditionalloadontherestoftheservers.

3.5.5.3 Query Load Balancing

Inanyscaleoutarchitecturewithmorethanonequeryserver,youneedtohavealoadbalancing mechanisminplace.Theloadbalancingmechanismservestwopurposes.First,itenablesyouto distributequeriesequallyacrossallqueryservers,achievingthescaleouteffect.Second,itenablesyou toselectivelytakequeryserversoffline,gracefullydrainingthem,whiletheyarebeingrefreshedwith newdata. Whenyouuseanyloadbalancingsolution,beawarethatthedatacachesoneachoftheserversinthe loadbalancingarchitecturewillbeindifferentstatesdependingontheclientsitiscurrentlyserving.This resultsindifferencesinresponsetimesforthesamequery,dependingonwhereitexecutes. Thereareseveralloadbalancingstrategiestoconsider.Thesearetreatedinthefollowingsubsections. Asyouchoosetheloadbalancer,bearinmindthegranularityoftheloadbalancingandhowthisaffects theprocesstoqueryserverswitching.Thisisespeciallyimportantifyouuseadedicatedprocessing architecture.Forexample,theWindowsNetworkLoadBalancingsolutionbalancesusersbetweeneach AnalysisServicesInstanceinthescaleoutfarm.Thismeansthatwhenyouhavetodrainusersfroma queryserverandupdatetheserverwiththelatestversionofthecube,theentireinstancehastobe drained.Ifyouhostmorethanonedatabaseperinstance,thismeansthatifonedatabaseisupdated theotherdatabasesinthesameinstancemustalsobetakenoffline.ClientloadbalancingandAnalysis ServicesLoadBalancermaybebettersolutionsforyouifyouwanttoloadbalancedatabases individually. 3.5.5.3.1 Client Load Balancing Intheclientloadbalancing,eachclientknowswhichqueryserveritwilluse.Implementingthisstrategy requiresclientsidecodethatcanintelligentlychoosetherightqueryserverandthenmodifythe connectionstringaccordingly.Exceladdinsareanexampleofthistypeofclientsidecode.Notethat youwillhavetodevelopyourownloadbalancertoachievethis. 3.5.5.3.2 Hardware-Level Load Balancing UsingtechnologiessuchasloadbalancersfromF5,itispossibletoimplementloadbalancingdirectlyin thenetworklayerofthearchitecture.ThismakestheloadbalancingtransparenttobothAnalysis Servicesandtheclientapplication.Ifyouchoosetogodownthisroute,makesurethattheloadbalance applianceenablesyoutoaffinitizeclientconnections.Whenclientscreatesessionobjects,stateis storedonAnalysisServices.Ifalaterclientrequest,relyingonthesamesessionstate,isredirectedtoa differentserver,theOLEDBproviderthrowsanerror.However,evenifyourunaffinity,youmaystill havetoforceclientsofftheserverwhenprocessingneedstocommit.Formoreinformationaboutthe ForceCommitTimeoutsetting,seethelockingsection.

186

3.5.5.3.3 Windows Network Load Balancing TheMicrosoftloadbalancingsolutionisNetworkLoadBalancing(NLB),whichisafeatureofthe WindowsServeroperatingsystem.WithNLB,youcancreateanNLBclusterofAnalysisServicesservers runninginmultiplehostmode.WhenanNLBclusterofAnalysisServicesserversisrunninginmultiple hostmode,incomingrequestsareloadbalancedamongtheAnalysisServicesservers. 3.5.5.3.4 Analysis Services Load Balancer AnalysisServicesisusedextensivelyinsideMicrosofttoserveourbusinessuserswithdata.Aspartof theinitiativetoscaleoutourAnalysisServicesfarmsanewloadbalancingsolutionwasbuilt.The advantagesofthissolutionarethatyoucanloadbalanceonthedatabaselevelandthatyouuseaweb APItocontroleachdatabaseandtheusersconnectedtoit.ThiscustomizedAnalysisServicesload balancingsolutionalsoallowsfinecontrolovertheloadbalancingalgorithmused.Beawarethatmoving largedatasetsoverthewebAPIhasabandwidthoverhead,dependingonhowmuchdataisrequested byuserqueries.Measurethisbandwidthoverheadaspartofthecubetestphase. References: AnalysisServicesLoadBalancingSolution http://sqlcat.com/technicalnotes/archive/2010/02/08/aslbsetup.aspx

187

3.5.5.4 ROLAP Scale Out

Withthequery/processingarchitectureyoucangettheupdatelatencyofcubesdowntoaround30 minutes,dependingonworkload.Butifyouwanttorefreshdatafasterthanthat,youhavetoeithergo fullyROLAPoruseahybridofoneofthestrategiesdiscussedearlierandROLAPpartitions. InapureROLAPsetup,youonlyprocessthedimensionsandredirectallmeasuregroupqueriesdirectly toarelationaldatabase.Thefollowingdiagramillustratesthis.

Figure 74 - ROLAP scale-out

InaROLAPsystemlikethis,youhavetoconsiderthespecialrequirementsmentionedlaterinthis document.Youshouldalsomakesurethatyourrelationaldatastoreisscaledtosupportmultiple AnalysisServicesqueryservers.

188

AninterestinghybridbetweenMOLAPandROLAPcanbebuiltbycombiningtheROLAPscaleoutwith eitherthededicatedprocessingarchitectureorthequery/processingflippingarchitecture.Youcanstore datathatchangeslessfrequentlyinMOLAPpartitions,whichyoueitherprocessorcopytothequery servers.DatathatchangesfrequentlycanbestoredinROLAPpartitions,redirectingqueriesdirectlyto therelationalsource.Suchasetupcanachieveverylowupdatelatencies,allthewaydowntoafew seconds,whilemaintainingthebenefitsofMOLAPcompression. References:

AnalysisServicesROLAPforSQLServerDataWarehouses http://sqlcat.com/whitepapers/archive/2010/08/23/analysisservicesrolapforsqlserverdata warehouses.aspx

3.6 Server Maintenance

WhenyoumoveanAnalysisServicesinstancetoproduction,therearesomeregularmaintenancetasks youshouldconfigure.Thissectiondescribesthosetasks.

3.6.1 Clearing Log Files and Dumps

Duringserveroperations,AnalysisServicesgeneratesalogfilecontainingdataabouttheoperation.This logfileislocatedinthefolderdescribedbytheLogDirinMsmdsrv.inithedefaultlocationbeing <Installdir>\OLAP\Log.Thisloggrowsextremelyslowlyandyoushouldgenerallynotneedtocleanitup. Ifyoudoneedtoreclaimthediskspacedusedbythelogfile,youhavetostoptheservicetodeleteit. Inthe<InstallDir>\OLAP\logfolder,youmayalsofindfileswithextension*.mdmp.Theseareminidump filesgeneratedbytheAnalysisServicesandaretypicallyafewmegabyteseach.Thesefilesget generatedwhenundetecteddeadlockshappeninsidetheprocessorwhenthereisaproblemwiththe service.ThefilesareusedbyMicrosoftSupporttoinvestigatestabilityissuesanderrorsintheserver.If youareexperiencinganysuchbehavior,youshouldcollecttheseminidumpfilesforuseduringcase investigation.Periodicallycheckforthesefiles,andcleanthemupafteranyMicrosoftSupportcaseyou haveopenisresolved.

3.6.2 Windows Event Log

AnalysisServicesusestheWindowseventlogstoreportservererrors,warnings,andinformation.The ApplicationLogisusedformostmessages,buttheSystemLogisalsousedforeventsthatarerelatedto ServiceManager. Dependingonyourserverconfiguration,eventlogsmaybeconfiguredtobecleanedmanually.Make surethatthisisaregularpartofyourmaintenance.Alternatively,youcanconfiguretheeventlogto overwriteoldereventswhenthelogisclosetofull.Inbothcases,makesureyouhaveenoughdisk spacetoholdthefulleventlog.Thefollowingillustrationshowshowtoconfiguretheeventlogto overwriteolderevents.

189


Figure 75 - Recycling the event log

3.6.3 Defragmenting the File System

AsdescribedintheI/Osection,therecanbeanadvantageindefragmentingthefilesstoringacube, especiallyafteralotofchangestothepartitionsanddimensions.Runningdiskdefragwillhavea measurableimpactonyourdisksubsystemperformance,anddependingonthehardwareyourunon, thismayaffectuserresponsetime.Youcanconsiderrunningdefragmentationontheserverduringoff peakhoursorinbatchwindows. Notethatthedefragutilityretainstheworkdone,evenwhenitdoesnotruntocompletion.Thismeans thatyoucandopartialdefragmentationspreadovertime.

3.6.4 Running Disk Checks

Runningdiskchecks(usingChkDsk.exe)ontheAnalysisServicesvolumegivesyoutheconfidencethat noundetectedI/Ocorruptionhasoccurred.Howoftenyouwanttodothisdependsonhowoftenyou expecttheI/Osubsystemtocreatesucherrorswithoutdetectingthemthisvariesbyvendoranddisk model. NotethatChkDsk.execanrunforaverylongtimeonalargediskvolume,andthatitwillhaveanimpact onyourI/Ospeeds.Becauseofthis,youmaywanttouseaSANsnapshotoftheLUNandrun ChkDsk.exeonanothermachinethatmountsthesnapshot.

190

Inbothcases,youshouldbeabletodetectdiskcorruptionwithouttouchingthelivesystem.Ifyou detectirreparablecorruption,youshouldconsiderrestoringthebackupsaspertheprevioussection. References: Chkdsk(http://technet.microsoft.com/enus/library/bb491051.aspx)

191

3.7 Special Considerations

UsingcertainfeaturesofAnalysisServicescubescanleadyoudownsomedesignpathsthatrequire extraattentiontosucceed.Thissectiondescribesthesespecialscenariosandtheconsiderationsthat applywhenyouencounterthem.

3.7.1 ROLAP and Real Time

ThissectiondealswithissuesthatarespecifictoBIenvironmentsthatdonothaveclearlydefinedbatch windowsforloadingdata.IfyouhaveacubethatupdatesdataatthesametimethatETLjobsare runningonthesourcedataorwhileusersareconnected,youneedtopayspecialattentiontocertain configurationparameters. AsdescribedintheLockingandBlockingsection,processingoperationsgenerallytakeaninstancewide lock.ThiswilltypicallypreventyoufromdesigningMOLAPsystemsthatareupdatedmorefrequently thanapproximatelyevery30minutes.Sucharefreshfrequencymaynotbeenough,andifthisisyour scenario,ROLAPisthepathforward,andyoushouldbeawareofthespecialconsiderationsthatapply. YoushouldalsobeawarethataROLAPpartitioncanputsignificantloadontheunderlyingrelational source,whichmeansyoushouldinvolvetheDBAfunctiontounderstandthisworkloadandtuneforit.

3.7.1.1 Cache Coherency

Bydefault,thestorageengineofAnalysisServicescachesROLAPsubcubesinthesamewayitcaches MOLAPsubcubes.Iftherelationaldatachangesfrequently,thismeansthatqueriesthattouchROLAP partitionsmayuseacombinationoftherelationalsourceandthestorageenginecachetogeneratethe response.Iftherelationalsourcehaschangedsincethecacheentrywasaddedtothestorageengine, thiscombinationofsourcedatacanleadtoresultsthataretransactionallyinconsistentfromthe perspectiveoftheuserbecausetheyrepresentanintermediatestateofthesystem.Thereareofcourse waystoresolvethiscoherencyissue,dependingonyourscenario. ChangingtheconnectionstringItispossibletoaddtheparameterRealTimeOLAP=truetothe connectionstringwhenthecubeisaccessed.Settingthisparametertotruecausesallrelevantstorage cachestoberefreshedforeveryqueryrunonthatconnectionincludingthecachesgeneratedby MOLAPqueries.Notethatthischangecancauseasignificantimpactonbothqueryperformanceand concurrency.Youshouldtestitcarefully.However,itgivesyouthemostuptodateresultspossible fromthecube,becauseAnalysisServicesisessentiallyusedasathinMDXwrapperontopofthe relationalsourceinthismode. BlowingawaycachesatregularintervalsyoucaneitheruseanXMLAscripttoclearthecacheoryou canusequerynotifications(setintheProActivecachingpropertiesofthepartition).Thisallowsyouto clearthestorageenginecachesatregularintervals.Assumingyoutimethiscacheclearingwith relationaldataloads,thisgivesusersaconsistentviewthatisupdatedeverytimethecacheiscleared. AlthoughthisdoesnotgiveyouthesamerefreshfrequencyastheRealTimeOLAP=truesetting,itdoes haveasmallerimpactonuserqueryperformanceandconcurrency.

192

Asyoucanseefromthesetwooptions,goingtowardsrealtimecubesrequiresyoutocarefullyconsider tradeoffsbetweenrefreshfrequenciesandperformance.Fullcoherencyispossiblebutexpensive. However,youcangetaloosecoherencythatismuchcheaper.AnalysisServicessupportsboth paradigms.

3.7.1.2 Manually Set Partition Slicers on ROLAP Partitions

WhenAnalysisServicesprocessestheindexonaMOLAPpartition,itcollectsdataabouttheattributesin thatpartition.Assumingthedatamatchesonlyoneattributevalue,anautomaticslicerissetonthe partition,eliminatingitfromscansthatdonotincludethatattributevalue.Forexample,ifyouprocessa partitionthathasdatafromDecember2008only,AnalysisServicesdetectsthisslicingandonlyaccesses thatpartitionwhenqueriesrequestdatainthattimerange. BecauseROLAPdataresidesoutsideofAnalysisServices,theautomaticslicerfunctionalityisnotused. Unlessyousettheslicermanually(whichcanbedonefrombothVisualStudioandSQLServer ManagementStudio)everyqueryhastotoucheveryROLAPpartition.Itisthereforeagoodpracticeto alwayssetslicesonROLAPpartitionswhentheyareavailable.

3.7.1.3 UDM Design

WhenyoudesignforROLAPaccess,itisgenerallyagoodideatokeeptheUDMassimpleaspossible. Thegivestherelationalenginethebestpossibleconditionsforoptimizingforthequeryworkload.The followingtablelistssomeoptimizationsyoushouldconsiderwhenswitchingacubetoROLAPmode. Existingfeatureusage Referencedimensions ROLAPredesign Switchtoapurestarschematoeliminateunnecessaryjoinsand providerelationalenginewithoptimalconditionsforquery execution. Normalizetheparentchilddimension(formoreinformation abouthowtodothis,seethereferencesforthissection). Reduceintermediatetablesizesusingmatrixcompression. Switchtotablebinding.Considerbindingtoaviewifqueriesare needed. Implementtheresultofthequeryintherelationalsource instead. Considerreducingthenumberofaggregates.Beawareof conditionsandfeaturesintherelationalenginesothatyoufully understandthetradeoffs.Forexample,inSQLServer2008R2,it isoftenagoodideatofocusonaggregatesthataretargetedonly attheleafleveland/or[all]levelofattributes). Optimizecarefully,andavoidcellbycelloperationsinlarge ROLAPpartitions. Ifatallpossible,useMOLAPdimensions.MOLAPdimensions havemuchbetterperformancethanROLAPdimensionsandif yourunregularProcessAddoperations,youcankeepthemupto dateatshortrefreshintervals.

Parent/childdimensions Manytomanydimensions Querybindingofpartitions Querybindingofdimensions Aggregates

MDXcalculations ROLAPdimensions

193

 YoushouldworkcloselywiththeBIdeveloperswhentroubleshootingROLAPcubes.Itisimperativeto getthedesignrightandfollowtheguidanceinPart1ofthisbook. References: AnalysisServicesParentChildDimensionNaturalizeronCodePlex http://pcdimnaturalize.codeplex.com/ o AlsoavailablefromBIDShelper:http://bidshelper.codeplex.com AnalysisServicesManytoManyDimensions:QueryPerformanceOptimizationTechniques http://www.microsoft.com/downloads/en/details.aspx?FamilyID=3494E712C90B4A4EAD45 01009C15C665&displaylang=en AnalysisServicesROLAPforSQLServerDataWarehouses http://sqlcat.com/whitepapers/archive/2010/08/23/analysisservicesrolapforsqlserverdata warehouses.aspx

194

3.7.1.4 Dimension Processing

Realtimedimensionprocessingcanpresentaspecialchallenge.Inabatchstylewarehouse,youarein controlofwheninsertsandupdateshappenwhichmeansyoucantypicallyprocessthedimension aftertherelationalsourceisdonerefreshingdata.Butifyouaredesigningacubeontopofarealtime source,therelationaldatamaychangewhileyouareprocessingadimension.Dimensionprocessingis bydefaultexecutedasmanyconcurrentSQLServerqueries,asdescribedintheOptimizingProcessing section.Considerthissequenceofevents: 1. ThecustomerdimensioncontainscustomersfromalloftheUnitedStates,butnocustomers fromtherestoftheworld. 2. DimensionProcessAddstarts. 3. AnalysisServicessendsquerySELECTDISTINCTZip,CityFROMDimensiontotherelational source.ThisqueryreadsallcurrentattributeZipandCityvalues. 4. Therelationalsourceinsertsanewrow,City=Copenhagen,inCountry=Denmark. 5. AnalysisServices,readingthenextlevelofthehierarchy,sendsthequerySELECTDISTINCTCity, Country. 6. TheCitymemberCopenhagenIsreturnedinthesecondquery,butbecauseitwasnotreturned inthefirst,AnalysisServicesthrowsanerror. Whilethisscenariomaysounduncommon,wehaveseenitatseveralcustomersthatdesignrealtime systems.Therearesomewaystoavoidtheseconditions. ByTableprocessingBysettingtheProcessingGrouppropertyofthedimensiontobeByTableyouwill changehowAnalysisServicesbehavesduringdimensionprocessing.InsteadofsendingmultipleSELECT DISTINCTqueries,theprocessingtaskwillinsteadrequesttheentiretablewithonequery.Thisallows youtogetaconsistentviewofthedimension,evenunderconcurrencyintherelationalsource. However,thissettinghasadrawback,namelythatyouwillneedtokeepallthesourcedimensiondata inmemorywhilethedimensionisprocessing.Iftheserverisundermemorywhilethishappens,paging canoccur,whichmaycauseaslowdownoftheentiresystem. MARSandSnapshotIfyouareprocessingontopofaSQLServerdatasource,youcanuseMultiple ActiveResultSets(MARS)andsnapshotisolationtoprocessthedimensionandgetaconsistentviewof thedataevenunderupdates. ConfiguringMARSandSnapshotprocessingrequiresafewconfigurationchangestothedatasource viewandrelationaldatabase.First,inthedatasourceproperties,changethedatasourceviewtouse snapshotisolation.

195


Figure 76 - Setting the Data Source to Snapshot

Second,enableMARSintheconnectionstringofthedatasourceview.

Figure 77 - Setting MARS in a DSV

Andfinally,enableeithersnapshotorreadcommittedsnapshotisolationintheSQLServerdatabase.
ALTERDATABASE [Database] SETREAD_COMMITTED_SNAPSHOTON

ProcessingnowusesMARS,andsnapshotsgenerateaconsistentviewofthedimensionduring processing. Understandthatmaintainingthesnapshotduringprocessing,aswellasstreamingthedatathrough MARS,doesnotprovidethesameperformanceasthedefaultprocessingoption. 196

MaintainingconsistencyrelationallyIfyouwanttobothmaintaintheprocessingspeedandavoid memoryconsumptionintheAnalysisServicesservice,youhavetodesignyourdatamodeltosupport realtimeprocessing.Thereareseveralwaystodothis,includingthefollowing: Addatimestamptotherowsinthedimensiontablethatshowswhentheyareinserted.During processing,onlyreadtherowshigherthanacertaintimestamp. Createadatabasesnapshotoftherelationalsourcebeforeprocessing. Manuallycreateacopyofthesourcetablebeforeprocessingontopofthecopy.Theoriginal canthenbeupdatedwhilethecopyisbeingaccessedbythecube.

References: MultipleActiveResultSets(MARS)http://msdn.microsoft.com/en us/library/ms345109(v=sql.90).aspx UsingByAttributeorByTableProcessingGroupPropertywithAnalysisServices2005 http://blogs.msdn.com/b/sqlcat/archive/2007/10/19/usingbyattributeorbytableprocessing grouppropertywithanalysisservices2005.aspx?wa=wsignin1.0

3.7.2 Distinct Count

Distinctcountmeasuresbehavedifferentlythanothermeasures.Becausedatainadistinctcount measureisnotadditive,alotmoreinformationmustbestoredondisk.Accordingtobestpractice,a measuregroupthathasadistinctcountmeasureshouldonlyhavethatsinglemeasureandnoothers. Whileadditivemeasurescompressverywell,thesameisnottruefordistinctcountmeasures.This meansthatleafleveldataofthemeasuregrouptakesupmorediskspace. Targetinggoodaggregatesfordistinctcountmeasurescanalsobedifficult.Althoughaggregatesfor additivemeasurescanbeusedbyqueriesathighergranularitiesthantheaggregate,thesamedoesnot applyfordistinctcountmeasures. Thecombinedeffectsofbigmeasuregroupsandlessusefulaggregatesmeansthatqueriesthatrun againstdistinctcountdataoftencauseasignificantamountofI/Ooperationsandsimplyrunlongerthan otherqueries.Thisisexpectedandpartofthenatureofdistinctcountdata.However,therearesome optimizationsyoucanmakethatcangreatlyspeedupbothqueriesandprocessingofdistinctcount measures.

3.7.2.1 Partitioning for Distinct Count

Recallthatforadditivemeasures,itisgenerallyrecommendedthatyoupartitionbytimeandsometimes byanotherdimension.ThispartitionstrategyisdescribedintheNonbreakingCubeChangessection. Distinctcountmeasuresareanexceptiontothisruleofthumb.Whenitcomestodistinctcount,itis oftenagoodideatopartitionbythevaluesofthedistinctcountmeasureitself.AnalysisServiceskeeps trackofthemeasurevaluesineachpartition,andassumingtheintervalsarenotoverlapping,itcan 197

benefitfromsomeparallelismoptimizations.Thebasicideaistocreatepartitions,typicallyoneperCPU coreinthemachine,thateachcontainanequalsized,nonoverlappingintervalofmeasurevalues.The followingpictureillustratesthesepartitions.

Figure 78 - Distinct Count Partitioning on a 4-Core Server

Youcanstillapplyadatebasedpartitionschemainadditiontothedistinctcountpartitioning.Butifyou do,makesurethatqueriesdonotcrossthegranularitylevelofthisdaterange,oryoulosepartofthe optimization.Forexample,ifyoudonothavequeriesacrossyears,youmaybenefitbypartitioningby bothyearandthedistinctcountmeasure.Conversely,ifyouhavequeriesthataskfordataattheyear level,youshouldnotpartitionbymonthandthedistinctcountmeasure. ThewhitepaperintheReferencessectiondescribesthepartitionstrategyfordistinctcountmeasuresin muchmoredetail. References: AnalysisServicesDistinctCountOptimization http://www.microsoft.com/downloads/en/details.aspx?FamilyID=65df6ebf9d1c405f84b1 08f492af52dd&displaylang=en o DescribesthepartitionstrategythatspeedsupqueriesandprocessingforDistinctCount Measuregroups

3.7.2.2 Optimizing Relational Indexes for Distinct Count

AnalysisServicesaddsanORDERBYclausetothedistinctcountprocessingquerieshave.Forexample,if youcreateadistinctcountmeasureonCustomerPONumberinFactInternetSales,yougetthisquery whileprocessing.
SELECTFROM FactInternetSales ORDERBY [CustomerPONumber]

198

Ifyourpartitioncontainsalargeamountofrows,orderingthedatacantakealongtime.Without supportingindexes,thequeryplanlookssomethinglikethis.

Figure 40 Relational sorting caused by distinct count

NoticethelongtimespentontheSortoperation?Bycreatingaclusteredindexsortedonthedistinct countcolumn(inthiscaseCustomerPONumber),youcaneliminatethissortoperationandgetaquery planthatlookslikethis.

Figure 41 Distinct count query supported by a good index

Ofcourse,thisindexneedstobemaintained.Buthavingitinplacespeedsuptheprocessingqueries.

3.7.3 Many-to-Many Dimensions

ManytomanydimensionsareapowerfulfeatureofAnalysisServicescubes.Theyenableeasysolutions forsomecomplex,yetcommonscenariosindimensionalmodeling.Whencubesresolvemanytomany queries,thejoinwiththeintermediatetableisdoneinAnalysisServicesmemoryandduringquerytime. Iftheintermediatetableislarge,especiallyifitlargerthanmemory,thesequeriescantakealongtime torespond.Werecommendthatyouusemanytomanydimensionsonlyiftheintermediatetablefitsin memory.ThelinksintheReferencessectiondescribesometechniquesthatenableyoutoreducethe memoryconsumptionoftheintermediatetable. References: AnalysisServicesManytoManyDimensions:QueryPerformanceOptimizationTechniques http://www.microsoft.com/downloads/en/details.aspx?FamilyID=3494E712C90B4A4EAD45 01009C15C665&displaylang=en BIDSHelperhastoolstoestimatebenefitsoftheManytomanycompressiondescribedinthe section: o http://bidshelper.codeplex.com/wikipage?title=Manyto Many%20Matrix%20Compression

199

Manytomanyprojecthttp://www.sqlbi.com/manytomany.aspx o Designpatternsformanytomanydimensions

4 Conclusion
This document provides guidance for creating and maintaining Analysis Services cubes that run in a production environment. In Part 1, you learned best practices for developing cubes and dimensions that are fast to process and query. Part 2 explored performance tuning from the standpoint of cubes that are already in production and not easily modified. For more information, see: http://sqlcat.com/:SQLCustomerAdvisoryTeam http://www.microsoft.com/sqlserver/:SQLServerWebsite http://technet.microsoft.com/enus/sqlserver/:SQLServerTechCenter http://msdn.microsoft.com/enus/sqlserver/:SQLServerDevCenter If you have any suggestions or comments, please do not hesitate to contact the authors. You can reach Thomas Kejser at tkejser@microsoft.com and Denny Lee at dennyl@microsoft.com. Did this paper help you? Please give us your feedback. Tell us on a scale of 1 (poor) to 5 (excellent), how would you rate this paper and why have you given it this rating? For example: Are you rating it high due to having good examples, excellent screen shots, clear writing, or another reason? Are you rating it low due to poor examples, fuzzy screen shots, or unclear writing?

This feedback will help us improve the quality of white papers we release. Sendfeedback.

200