lambda-packs
PrecompiledpackagesforAWSlambda
Howtostarthttps://aws.amazo.com/lambda/adcreate/sigiitoaccoutLambda>Fuctios-CreatelambdafuctioBlakfuctioCofiguretriggers-NextCofigurefuctioRutime-Pytho2.7LambdafuctiohadleradroleHadler-service.hadlerRole-Createewrolefromtemplate(s)Roleame-testPolicytemplates-SimpleMicroservicePermissiosAdvacedsettigsMemory(MB)128Timeout1mi0secCodeetrytype-Uploada.ZIPfile-choosePack.zipfromrepTest->SaveadtestHowtomodifyModifyservice.pyfilefromsourcesfolderChooseallfilesisourcesfolderwhecompressig,do'tputitioefolderUploadzipfileofuctiopageCurretpacksSeleiumPhatomJSItroSeleiumoPhatomJS.Ifact-aready-madetoolforwebscrapig.Forexample,thedemoowopesaradompageiWikipediaadsedsitsheader.(PhatomJSatthesametimedisguisesitselfasaormalbrowser,kowshowtologi,clickadfilloutforms)Alsoaddedrequests,soyoucadoAPIrequestsfordifferetresourcestodiscard/takeawaytheiformatio.
Usefulforwebtestigadscrapig.
DemoCurretdemoopesradompagefromwiki(https://e.wikipedia.org/wiki/Special:Radom)adpritstitle.
Serverlessstartgitcloehttps://github.com/ryfeus/lambda-packs.gitcdlambda-packs/Seleium_PhatomJS/source/serverlessdeployserverlessivoke--fuctiomai--logYoucaalsoseetheresultsfromtheAPIGatewayedpoitiawebbrowser.
Documetatiohttps://seleium-pytho.readthedocs.io/
Pyresttest+WRKItroWhatdoesthelambdahavetodowithit?IautshelloAWSioeregioyoucasimultaeouslyru200lambdas(moreifyouwritetosupport).Lambdaworksi11regios.Soyoucaruiparallelmoretha2000lambdas,eachofwhichwillcoductloadtestigofyourservice.Fivemiutesofsuchtestigwillcostjustoedollar.
DemoDemoithispackagetriestosedrequeststogithub.comfor5secodswith1coectioadalsocoductpyresttestdummytest.
ToolsWRK(https://github.com/wg/wrk)-themaitoolforloadtestig.Itworkswithmultiplethreads,youcaspecifytheumberofcoectiosadlegthoftheload.Formorefie-tuig,youcauseLuaJITscripts(https://www.lua.org/).Pyrestest(https://github.com/svaoort/pyresttest)isahadytoolfortestigthefullpipelieoftheAPI.Forexample,theuserregisters,theusestheapikeytocreatetasks/makeotes/dowloadsfiles,thereadsthem,thedeletesthem.Documetatiohttps://github.com/wg/wrk
https://github.com/svaoort/pyresttest
Lxml+requestsItroPackageforparsigstaticHTMLpages.DifferecehereisthatitworksfasteradcosumeslessmemorythaPhatomJSbutislimiteditermswebsitesitcaparseadotherfeatures.
Serverlessstartserverlessistall-uhttps://github.com/ryfeus/lambda-packs/tree/master/Lxml_requests/source-lxml-requestscdlxml-requestsserverlessdeployserverlessivoke--fuctiomai--logBuildpackwgethttps://github.com/ryfeus/lambda-packs/blob/master/Lxml_requests/buildPack.shdockerpullamazoliux:latestdockerru-v$(pwd):/outputs--amelambdapackge-damazoliux:latesttail-f/dev/ulldockerexec-i-tlambdapackge/bi/bash/outputs/buildPack.shToolsLxml3.7.1
Documetatiohttps://lxml.de/
TesorflowItroOpesourcelibraryforMachieItelligece.BasicallyrevolutioizedAIadmadeitmoreaccessible.Usigtesorflowolambdaisotasbadasitmaysoud-forsomesimplemodelsitisthesimplestadthecheapestwaytodeploy.
DemoAshelloworldcodeIusedrecogitioofimagestraiedoimageet(https://www.tesorflow.org/tutorials/image_recogitio).Givethepricetaglambdaoeru(recogitioofoepicture)willcost$0.00005.Thereforeforadollaryoucarecogize20,000images.Itismuchcheaperthaalmostayalteratives,thoughcompletelyscalable(200fuctioscaberuiparallel),adcabeeasilyitegrateditocloudifrastructure.Curretdemodowloadsimagefromlik'imagelik'fromevetsource(ifempty-thedowloadshttps://s3.amazoaws.com/ryfeuslambda/tesorflow/imageet/cropped_pada.jpg)
ToolsTesorflow1.4.0
Documetatiohttps://www.tesorflow.org/tutorials/image_recogitio
NightlyversioNightlyversioarchiveismoretha50MBisizebutitisstilleligibleforusigwithAWSLambda(thoughyoueedtouploadpackthroughS3).Formorereadhere:
https://hackeroo.com/explorig-the-aws-lambda-deploymet-limits-9a8384b0bec3
Serverlessstartserverlessistall-uhttps://github.com/ryfeus/lambda-packs/tree/master/tesorflow/source-tesorflowcdtesorflowserverlessdeployserverlessivoke--fuctiomai--logBuildpackforPytho2:
wgethttps://raw.githubusercotet.com/ryfeus/lambda-packs/master/Tesorflow/buildPack.shwgethttps://raw.githubusercotet.com/ryfeus/lambda-packs/master/Tesorflow/idex.pydockerpullamazoliux:latestdockerru-v$(pwd):/outputs--amelambdapackge-damazoliux:latesttail-f/dev/ulldockerexec-i-tlambdapackge/bi/bash/outputs/buildPack.shforPytho3:
wgethttps://raw.githubusercotet.com/ryfeus/lambda-packs/master/Tesorflow/buildPack_py3.shwgethttps://raw.githubusercotet.com/ryfeus/lambda-packs/master/Tesorflow/idex_py3.pydockerpullamazoliux:latestdockerru-v$(pwd):/outputs--amelambdapackge-damazoliux:latesttail-f/dev/ulldockerexec-i-tlambdapackge/bi/bash/outputs/buildPack_py3.shNote:RememberYoushouldsetpytho3.6forAWSLambdafuctioeviromet.
LayerARNar:aws:lambda:us-east-1:339543757547:layer:tesorflow-pack
SklearItroPackageforfasofmachielearig,buildigmodelsadthelike.Idoubtthatthereisamorecoveietwaytodeploymodeltotherealworld.
ToolsScikit-lear0.17.1Scipy0.17.0Documetatiohttps://scikit-lear.org/
SkimageItroPackageofimageprocessigtools,adotolytostyleimage,butalsoalargesetofcomputervisioalgorithms.
Therearecurretlytwozippedpacksavailable,Pack.zipadPack_omatplotlib.zip,youprobablywattousePack_omatplotlib.zip.Seehttps://github.com/ryfeus/lambda-packs/issues/5formoreiformatio.
ToolsScikit-image0.12.3
Documetatiohttps://scikit-image.org/
OpeCV+PILItroAotherpackageofimageprocessigtools,adotolytostyleimage,butalsoalargesetofComputervisioalgorithms.
ToolsOpeCV3.1.0PIL4.0.0Documetatiohttps://pillow.readthedocs.io/
https://docs.opecv.org/3.0-beta/doc/py_tutorials/py_tutorials.html
PadasItroPackageforfasofstatistics,datascietistsaddataegieers.RAMatlambdais1.5gigabytes,adthemaximumoperatigtime-5miutes.Iamsurethatwillbeeoughformosttasks.
ToolsPadas0.19.0
Documetatiohttps://padas.pydata.org/
SpacyItroOpesourcelibraryforNaturalLaguageProcessigipytho.
ToolsSpacy2.0.11Documetatiohttps://spacy.io/
ExampleExamplecodeloadslaguagemodelfromS3adusesittoaalyzesetece.
TesseractItroOCR(opticalcharacterrecogitio)libraryfortextrecogitiofromtheimage.
Documetatiohttps://github.com/tesseract-ocr/tesseract
PDFgeerator+Microsoftofficefilegeerator(docx,xlsx,pptx)+imagegeerator(jpg,pg)+bookgeerator(epub)Itro"Helloworld"codeipackagecreatesexampleofeverydocumet.Basicallytheselibsarelowmemory(lesstha128MB)adhighspeed(lesstha0.5secods)soit'ssomethiglike~1mdocumetsgeeratedper1$itermsofAWSLambdapricig.
Toolsdocx(pytho-docx-https://pypi.pytho.org/pypi/pytho-docx)xlsx(XlsxWriter-https://pypi.pytho.org/pypi/XlsxWriter)pptx(pytho-pptx-https://pypi.pytho.org/pypi/pytho-pptx)pdf(Reportlab-https://pypi.pytho.org/pypi/reportlab)epub(EbookLib-https://pypi.pytho.org/pypi/EbookLib)pg/jpg/...(Pillow-https://pypi.pytho.org/pypi/Pillow)Satelliteimageryprocessig(rasterio+OSGEO+pyproj+shapely+PIL)ItroAWSLambdapackiPythoforprocessigsatelliteimagery.Basicallyiteablestodeploypythocodeiaeasyadcheapwayforprocessigsatelliteimageryorpolygos.I“helloworld”codeofthepackIdowloadred,gree,blueLadsat8badsfromAWS,makeTrueColorimageoutofitaduploadittoS3.Ittakes35secodsad824MBofRAMforitso~2500sceescabeprocessedfor1$.
ToolsRasterio(https://github.com/mapbox/rasterio0.36)OSGEO(https://trac.osgeo.org/gdal/wiki/GdalOgrIPytho)Pyproj(https://github.com/jswhit/pyproj)Shapely(https://github.com/Toblerity/Shapely)PIL(https://pillow.readthedocs.io/)PyTorchPytho3.6basedPyTorch
ToolsPyTorch1.0.1(CPU)torchvisio0.2.1IstalledPackages(deps)umpy-1.16.1pillow-5.4.1six-1.12.0torchvisio-0.2.1BuildPack#Youeed`docker`beforeru./build-with-docker.sh
评论