OmniSciDBisanopensourceSQL-based,relational,columnardatabaseenginethatleveragesthefullperformanceandparallelismofmodernhardware(bothCPUsandGPUs)toenablequeryingofmulti-billionrowdatasetsinmilliseconds,withouttheneedforindexing,pre-aggregation,ordownsampling.OmniSciDBcanberunonhybridCPU/GPUsystems(NvidiaGPUsarecurrentlysupported),aswellasonCPU-onlysystemsfeaturingX86,Power,andARM(experimentalsupport)architectures.Toachievemaximumperformance,OmniSciDBfeaturesmulti-tieredcachingofdatabetweenstorage,CPUmemory,andGPUmemory,andaninnovativeJust-In-Time(JIT)querycompilationframework.
Forusageinfo,seetheproductdocumentation,andformoredetailsaboutthesystem'sinternalarchitecture,checkoutthedeveloperdocumentation.FurthertechnicaldiscussioncanbefoundontheOmniSciCommunityForum.
Therepositoryincludesanumberofthirdpartypackagesprovidedunderseparatelicenses.DetailsaboutthesepackagesandtheirrespectivelicensesisatThirdParty/licenses/index.md.
DownloadsandInstallationInstructionsOmniSciprovidespre-builtbinariesforLinuxforstablereleasesoftheproject:
DistroPackagetypeCPU/GPURepositoryDocsCentOSRPMCPUhttps://releases.omnisci.com/os/yum/stable/cpuhttps://docs.omnisci.com/latest/4_centos7-yum-cpu-os-recipe.htmlCentOSRPMGPUhttps://releases.omnisci.com/os/yum/stable/cudahttps://docs.omnisci.com/latest/4_centos7-yum-gpu-os-recipe.htmlUbuntuDEBCPUdebhttps://releases.omnisci.com/os/apt/stablecpuhttps://docs.omnisci.com/latest/4_ubuntu-apt-cpu-os-recipe.htmlUbuntuDEBGPUdebhttps://releases.omnisci.com/os/apt/stablecudahttps://docs.omnisci.com/latest/4_ubuntu-apt-gpu-os-recipe.html*tarballCPUhttps://releases.omnisci.com/os/tar/omnisci-os-latest-Linux-x86_64-cpu.tar.gz*tarballGPUhttps://releases.omnisci.com/os/tar/omnisci-os-latest-Linux-x86_64-cuda.tar.gzDevelopingOmniSciDB:TableofContentsLinksLicenseContributingBuildingTestingUsingCodeStyleDependenciesRoadmapLinksDeveloperDocumentationDoxygen-generatedDocumentationProductDocumentationReleaseNotesCommunityForumOmniSciHomepageOmniSciBlogOmniSciDownloadsLicenseThisprojectislicensedundertheApacheLicense,Version2.0.
Therepositoryincludesanumberofthirdpartypackagesprovidedunderseparatelicenses.DetailsaboutthesepackagesandtheirrespectivelicensesisatThirdParty/licenses/index.md.
ContributingInordertoclarifytheintellectualpropertylicensegrantedwithContributionsfromanypersonorentity,OmniScimusthaveaContributorLicenseAgreement("CLA")onfilethathasbeensignedbyeachContributor,indicatingagreementtotheContributorLicenseAgreement.Aftermakingapullrequest,abotwillnotifyyouifasignedCLAisrequiredandprovideinstructionsforhowtosignit.Pleasereadtheagreementcarefullybeforesigningandkeepacopyforyourrecords.
BuildingIfthisisyourfirsttimebuildingOmniSciDB,installthedependenciesmentionedintheDependenciessectionbelow.
OmniSciDBusesCMakeforitsbuildsystem.
mkdirbuildcdbuildcmake-DCMAKE_BUILD_TYPE=debug..make-j4Thefollowingcmake/ccmakeoptionscanenable/disabledifferentfeatures:
-DCMAKE_BUILD_TYPE=release-Buildtypeandcompileroptionstouse.OptionsareDebug,Release,RelWithDebInfo,MinSizeRel,andunset.-DENABLE_ASAN=off-Enableaddresssanitizer.Defaultisoff.-DENABLE_AWS_S3=on-EnableAWSS3support,ifavailable.Defaultison.-DENABLE_CUDA=off-DisableCUDA.Defaultison.-DENABLE_CUDA_KERNEL_DEBUG=off-EnabledebuggingsymbolsforCUDAkernels.Willdramaticallyreducekernelperformance.Defaultisoff.-DENABLE_DECODERS_BOUNDS_CHECKING=off-Enableboundscheckingforcolumndecoding.Defaultisoff.-DENABLE_FOLLY=on-UseFolly.Defaultison.-DENABLE_IWYU=off-Enableinclude-what-you-use.Defaultisoff.-DENABLE_JIT_DEBUG=off-EnabledebuggingsymbolsfortheJIT.Defaultisoff.-DENABLE_PROFILER=off-Enablegoogleperftools.Defaultisoff.-DENABLE_STANDALONE_CALCITE=off-RequirestandaloneCalciteserver.Defaultisoff.-DENABLE_TESTS=on-Buildunittests.Defaultison.-DENABLE_TSAN=off-Enablethreadsanitizer.Defaultisoff.-DENABLE_CODE_COVERAGE=off-Enablecodecoveragesymbols(clangonly).Defaultisoff.-DMAPD_DOCS_DOWNLOAD=on-Downloadthelatestmasterbuildofthedocumentation/docs.mapd.com.Defaultisoff.Note:thisisa>50MBdownload.-DPREFER_STATIC_LIBS=off-Staticlinkdependencies,ifavailable.Defaultisoff.TestingOmniSciDBusesGoogleTestasitsmaintestingframework.TestsresideundertheTestsdirectory.
Thesanity_teststargetrunsthemostcommontests.IfusingMakefilestobuild,thetestsmayberunusing:
makesanity_testsAddressSanitizerAddressSanitizercanbeactivatedbysettingtheENABLE_ASANCMakeflaginafreshbuilddirectory.AtthistimeCUDAmustalsobedisabled.InanemptybuilddirectoryrunCMakeandcompile:
mkdirbuild&&cdbuildcmake-DENABLE_ASAN=on-DENABLE_CUDA=off..make-j4Finallyrunthetests:
exportASAN_OPTIONS=alloc_dealloc_mismatch=0:handle_segv=0makesanity_testsThreadSanitizerThreadSanitizercanbeactivatedbysettingtheENABLE_TSANCMakeflaginafreshbuilddirectory.AtthistimeCUDAmustalsobedisabled.InanemptybuilddirectoryrunCMakeandcompile:
mkdirbuild&&cdbuildcmake-DENABLE_TSAN=on-DENABLE_CUDA=off..make-j4WeuseaTSANsuppressionsfiletoignorewarningsinthirdpartylibraries.SourcethesuppressionsfilebyaddingittoyourTSAN_OPTIONSenv:
exportTSAN_OPTIONS="suppressions=/path/to/mapd/config/tsan.suppressions"Finallyrunthetests:
makesanity_testsGeneratingPackagesOmniSciDBusesCPacktogeneratepackagesfordistribution.PackagesgeneratedonCentOSwithstaticlinkingenabledcanbeusedonmostotherrecentLinuxdistributions.
TogeneratepackagesonCentOS(assumingstartingfromtopleveloftheomniscidbrepository):
mkdirbuild-package&&cdbuild-packagecmake-DPREFER_STATIC_LIBS=on-DCMAKE_BUILD_TYPE=release..make-j4cpack-GTGZThefirstcommandcreatesafreshbuilddirectory,toensurethereisnothingleftoverfromapreviousbuild.
Thesecondcommandconfiguresthebuildtopreferlinkingtothedependencies'staticlibrariesinsteadofthe(default)sharedlibraries,andtobuildusingCMake'sreleaseconfiguration(enablescompileroptimizations).Linkingtothestaticversionsofthelibrarieslibrariesreducesthenumberofdependenciesthatmustbeinstalledontargetsystems.
Thelastcommandgeneratesa.tar.gzpackage.TheTGZcanbereplacedwith,forexample,RPMorDEBtogeneratea.rpmor.deb,respectively.
UsingThestartomnisciwrapperscriptmaybeusedtostartOmniSciDBinatestingenvironment.Thisscriptperformsthefollowingtasks:
initializesthedatastoragedirectoryviainitdb,ifrequiredstartsthemainOmniSciDBserver,omnisci_serverofferstodownloadandimportasampledataset,usingtheinsert_sample_datascriptAssumingyouareinthebuilddirectory,anditisasubdirectoryoftheomniscidbrepository,startomniscimayberunby:
../startomnisciStartingManuallyItisassumedthatthefollowingcommandsarerunfrominsidethebuilddirectory.
Initializethedatastoragedirectory.Thiscommandonlyneedstoberunonce.
mkdirdata&&./bin/initdbdataStarttheOmniSciDBserver:
./bin/omnisci_serverIfdesired,insertasampledatasetbyrunningtheinsert_sample_datascriptinanewterminal:
../insert_sample_dataYoucannowstartusingthedatabase.Theomnisqlutilitymaybeusedtointeractwiththedatabasefromthecommandline:
./bin/omnisql-pHyperInteractivewhereHyperInteractiveisthedefaultpassword.Thedefaultuseradminisassumedifnotprovided.
CodeStyleContributedcodeshouldcompilewithoutgeneratingwarningsbyrecentcompilersonmostLinuxdistributions.ChangestothecodeshouldfollowtheC++CoreGuidelines.
clang-formatA.clang-formatstyleconfiguration,basedontheChromiumstyleguide,isprovidedatthetopleveloftherepository.Pleaseformatyourcodeusingarecentversion(8.0+preferred)ofClangFormatbeforesubmitting.
Touse:
clang-format-iFile.cppclang-tidyA.clang-tidyconfigurationisprovidedatthetopleveloftherepository.Pleaselintyourcodeusingarecentversion(6.0+preferred)ofclang-tidybeforesubmitting.
clang-tidyrequiresallgeneratedfilestoexistbeforerunning.Theeasiestwaytoaccomplishthisistosimplyrunafullbuildbeforerunningclang-tidy.Abuildtargetwhichrunsclang-tidyisprovided.Touse:
makerun-clang-tidyNote:clang-tidymaymakeinvalidoroverlyverbosechangestothesourcecode.Itisrecommendedtofirstcommityourchanges,thenrunclang-tidyandreviewitsrecommendedchangesbeforeamendingthemtoyourcommit.
Note:theclang-tidytargetusestherun-clang-tidy.pyscriptprovidedwithLLVM,whichmaydependonPyYAML.Thetargetalsodependsonjq,whichisusedtofilterportionsofthecompile_commands.jsonfile.
DependenciesOmniSciDBhasthefollowingdependencies:
PackageMinVersionRequiredCMake3.16yesLLVM9.0yesGCC8.4.0no,ifbuildingwithclangGo1.12yesBoost1.72.0yesOpenJDK1.7yesCUDA11.0yes,ifcompilingwithGPUsupportgperftoolsyesgdal2.4.2yesArrow3.0.0yesCentOS7OmniSciDBrequiresanumberofdependencieswhicharenotprovidedinthecommonCentOS/RHELpackagerepositories.AprebuiltpackagecontainingallthesedependenciesisprovidedforCentOS7(x86_64).
Usethescripts/mapd-deps-prebuilt.shbuildscripttoinstallprebuiltdependencies.
Thesedependencieswillbeinstalledtoadirectoryunder/usr/local/mapd-deps.Themapd-deps-prebuilt.shscriptalsoinstallsEnvironmentModulesinordertosimplifymanagingtherequiredenvironmentvariables.Logoutandlogbackinafterrunningthemapd-deps-prebuilt.shscriptinordertoactiveEnvironmentModulescommand,module.
Themapd-depsenvironmentmoduleisdisabledbydefault.Toactivateforyourcurrentsession,run:
moduleloadmapd-depsTodisablethemapd-depsmodule:
moduleunloadmapd-depsWARNING:Themapd-depspackagecontainsnewerversionsofpackagessuchasGCCandncurseswhichmightnotbecompatiblewiththerestofyourenvironment.Makesuretodisablethemapd-depsmodulebeforecompilingotherpackages.
InstructionsforinstallingCUDAarebelow.
CUDAItispreferred,butnotnecessary,toinstallCUDAandtheNVIDIAdriversusingthe.rpmusingtheinstructionsprovidedbyNVIDIA.Therpm(network)method(preferred)willensureyoualwayshavethelateststabledrivers,whiletherpm(local)methodallowsyoutoinstalldoesnotrequireInternetaccess.
The.rpmmethodrequiresDKMStobeinstalled,whichisavailablefromtheExtraPackagesforEnterpriseLinuxrepository:
sudoyuminstallepel-releaseBesuretorebootafterinstallinginordertoactivatetheNVIDIAdrivers.
EnvironmentVariablesThemapd-deps-prebuilt.shscriptincludestwofileswiththeappropriateenvironmentvariables:mapd-deps-<date>.sh(forsourcingfromyourshellconfig)andmapd-deps-<date>.modulefile(forusewithEnvironmentModules,yumpackageenvironment-modules).Thesefilesareplacedinmapd-depsinstalldirectory,usually/usr/local/mapd-deps/<date>.Eitherofthesemaybeusedtoconfigureyourenvironment:the.shmaybesourcedinyourshellconfig;the.modulefileneedstobemovedtothemodulespath.
BuildingDependenciesThescripts/mapd-deps-centos.shscriptisusedtobuildthedependencies.ModifythisscriptandrunifyouwouldliketochangedependencyversionsortobuildonalternativeCPUarchitectures.
cdscriptsmoduleunloadmapd-deps./mapd-deps-centos.sh--compressmacOSscripts/mapd-deps-osx.shisprovidedthatwillautomaticallyinstalland/orupdateHomebrewandusethattoinstallalldependencies.PleasemakesuremacOSiscompletelyuptodateandXcodeisinstalledbeforerunning.XcodecanbeinstalledfromtheAppStore.
CUDAmapd-deps-osx.shwillautomaticallyinstallCUDAviaHomebrewandaddthecorrectenvironmentvariablesto~/.bash_profile.
Javamapd-deps-osx.shwillautomaticallyinstallJavaandMavenviaHomebrewandaddthecorrectenvironmentvariablesto~/.bash_profile.
UbuntuMostbuilddependenciesrequiredbyOmniSciDBareavailableviaAPT.CertaindependenciessuchasThrift,Blosc,andFollymustbebuiltastheyeitherdonotexistinthedefaultrepositoriesorhaveoutdatedversions.AprebuiltpackagecontainingallthesedependenciesisprovidedforUbuntu18.04(x86_64).Thedependencieswillbeinstalledto/usr/local/mapd-deps/bydefault;seetheEnvironmentVariablessectionbelowforhowtoaddthesedependenciestoyourenvironment.
Ubuntu16.04OmniSciDBrequiresanewerversionofBoostthantheversionwhichisprovidedbyUbuntu16.04.Thescripts/mapd-deps-ubuntu1604.shbuildscriptwillcompileandinstallanewerversionofBoostintothe/usr/local/mapd-deps/directory.
Ubuntu18.04Usethescripts/mapd-deps-prebuilt.shbuildscripttoinstallprebuiltdependencies.
Thesedependencieswillbeinstalledtoadirectoryunder/usr/local/mapd-deps.Themapd-deps-prebuilt.shscriptabovewillgenerateascriptnamedmapd-deps.shcontainingtheenvironmentvariableswhichneedtobeset.Simplysourcethisfileinyourcurrentsession(orsymlinkitto/etc/profile.d/mapd-deps.sh)inordertoactivateit:
source/usr/local/mapd-deps/mapd-deps.shEnvironmentVariablesTheCUDAandmapd-depslibdirectoriesneedtobeaddedtoLD_LIBRARY_PATH;theCUDAandmapd-depsbindirectoriesneedtobeaddedtoPATH.Themapd-deps-ubuntu.shandmapd-deps-prebuilt.shscriptswillgenerateascriptnamedmapd-deps.shcontainingtheenvironmentvariableswhichneedtobeset.Simplysourcethisfileinyourcurrentsession(orsymlinkitto/etc/profile.d/mapd-deps.sh)inordertoactivateit:
source/usr/local/mapd-deps/mapd-deps.shCUDARecentversionsofUbuntuprovidetheNVIDIACUDAToolkitanddriversinthestandardrepositories.Toinstall:
sudoaptinstall-y\nvidia-cuda-toolkitBesuretorebootafterinstallinginordertoactivatetheNVIDIAdrivers.
BuildingDependenciesThescripts/mapd-deps-ubuntu.shandscripts/mapd-deps-ubuntu1604.shscriptsareusedtobuildthedependenciesforUbuntu18.04and16.04,respectively.Thescriptswillinstallallrequireddependencies(exceptCUDA)andbuildthedependencieswhichrequireit.ModifythisscriptandrunifyouwouldliketochangedependencyversionsortobuildonalternativeCPUarchitectures.
cdscripts./mapd-deps-ubuntu.sh--compressArchscripts/mapd-deps-arch.shisprovidedthatwilluseyaytoinstallpackagesfromtheArchUserRepositoryandacustomPKGBUILDscriptforApacheArrow.Ifyoudon'thaveyayyet,installitfirst:https://github.com/Jguer/yay#installation
Note:ApacheArrow,whileavailableintheAUR,requiresafewcustombuildflagsinordertobeusedwithCore.AcustomPKGBUILDforitisincluded.
CUDACUDAandtheNVIDIAdriversmaybeinstalledusingthefollowing.
yay-S\linux-headers\cuda\nvidiaBesuretorebootafterinstallinginordertoactivatetheNVIDIAdrivers.
EnvironmentVariablesThecudapackageshouldsetuptheenvironmentvariablesrequiredtouseCUDA.Ifyoureceiveerrorssayingnvccisnotfound,thenCUDAbindirectoriesneedtobeaddedtoPATH:theeasiestwaytodosoisbycreatinganewfilenamed/etc/profile.d/mapd-deps.shcontainingthefollowing:
PATH=/opt/cuda/bin:$PATHexportPATH
评论