PrecompiledpackagesforAWSlambda
Howtostarthttps://aws.amazon.com/lambda/andcreate/signinintoaccountLambda>Functions-CreatelambdafunctionBlankfunctionConfiguretriggers-NextConfigurefunctionRuntime-Python2.7LambdafunctionhandlerandroleHandler-service.handlerRole-Createnewrolefromtemplate(s)Rolename-testPolicytemplates-SimpleMicroservicePermissionsAdvancedsettingsMemory(MB)128Timeout1min0secCodeentrytype-Uploada.ZIPfile-choosePack.zipfromrepTest->SaveandtestHowtomodifyModifyservice.pyfilefromsourcesfolderChooseallfilesinsourcesfolderwhencompressing,don'tputitinonefolderUploadzipfileonfunctionpageCurrentpacksSeleniumPhantomJSIntroSeleniumonPhantomJS.Infact-aready-madetoolforwebscraping.Forexample,thedemonowopensarandompageinWikipediaandsendsitsheader.(PhantomJSatthesametimedisguisesitselfasanormalbrowser,knowshowtologin,clickandfilloutforms)Alsoaddedrequests,soyoucandoAPIrequestsfordifferentresourcestodiscard/takeawaytheinformation.
Usefulforwebtestingandscraping.
DemoCurrentdemoopensrandompagefromwiki(https://en.wikipedia.org/wiki/Special:Random)andprintstitle.
Serverlessstartgitclonehttps://github.com/ryfeus/lambda-packs.gitcdlambda-packs/Selenium_PhantomJS/source/serverlessdeployserverlessinvoke--functionmain--logYoucanalsoseetheresultsfromtheAPIGatewayendpointinawebbrowser.
Documentationhttps://selenium-python.readthedocs.io/
Pyresttest+WRKIntroWhatdoesthelambdahavetodowithit?InanutshellonAWSinoneregionyoucansimultaneouslyrun200lambdas(moreifyouwritetosupport).Lambdaworksin11regions.Soyoucanruninparallelmorethan2000lambdas,eachofwhichwillconductloadtestingofyourservice.Fiveminutesofsuchtestingwillcostjustonedollar.
DemoDemointhispackagetriestosendrequeststogithub.comfor5secondswith1connectionandalsoconductpyresttestdummytest.
ToolsWRK(https://github.com/wg/wrk)-themaintoolforloadtesting.Itworkswithmultiplethreads,youcanspecifythenumberofconnectionsandlengthoftheload.Formorefine-tuning,youcanuseLuaJITscripts(https://www.lua.org/).Pyrestest(https://github.com/svanoort/pyresttest)isahandytoolfortestingthefullpipelineoftheAPI.Forexample,theuserregisters,thenusestheapikeytocreatetasks/makenotes/downloadsfiles,thenreadsthem,thendeletesthem.Documentationhttps://github.com/wg/wrk
https://github.com/svanoort/pyresttest
Lxml+requestsIntroPackageforparsingstaticHTMLpages.DifferencehereisthatitworksfasterandconsumeslessmemorythanPhantomJSbutislimitedintermswebsitesitcanparseandotherfeatures.
Serverlessstartserverlessinstall-uhttps://github.com/ryfeus/lambda-packs/tree/master/Lxml_requests/source-nlxml-requestscdlxml-requestsserverlessdeployserverlessinvoke--functionmain--logBuildpackwgethttps://github.com/ryfeus/lambda-packs/blob/master/Lxml_requests/buildPack.shdockerpullamazonlinux:latestdockerrun-v$(pwd):/outputs--namelambdapackgen-damazonlinux:latesttail-f/dev/nulldockerexec-i-tlambdapackgen/bin/bash/outputs/buildPack.shToolsLxml3.7.1
Documentationhttps://lxml.de/
TensorflowIntroOpensourcelibraryforMachineIntelligence.BasicallyrevolutionizedAIandmadeitmoreaccessible.Usingtensorflowonlambdaisnotasbadasitmaysound-forsomesimplemodelsitisthesimplestandthecheapestwaytodeploy.
DemoAshelloworldcodeIusedrecognitionofimagestrainedonimagenet(https://www.tensorflow.org/tutorials/image_recognition).Giventhepricetaglambdaonerun(recognitionofonepicture)willcost$0.00005.Thereforeforadollaryoucanrecognize20,000images.Itismuchcheaperthanalmostanyalternatives,thoughcompletelyscalable(200functionscanberuninparallel),andcanbeeasilyintegratedintocloudinfrastructure.Currentdemodownloadsimagefromlink'imagelink'fromeventsource(ifempty-thendownloadshttps://s3.amazonaws.com/ryfeuslambda/tensorflow/imagenet/cropped_panda.jpg)
ToolsTensorflow1.4.0
Documentationhttps://www.tensorflow.org/tutorials/image_recognition
NightlyversionNightlyversionarchiveismorethan50MBinsizebutitisstilleligibleforusingwithAWSLambda(thoughyouneedtouploadpackthroughS3).Formorereadhere:
https://hackernoon.com/exploring-the-aws-lambda-deployment-limits-9a8384b0bec3
Serverlessstartserverlessinstall-uhttps://github.com/ryfeus/lambda-packs/tree/master/tensorflow/source-ntensorflowcdtensorflowserverlessdeployserverlessinvoke--functionmain--logBuildpackforPython2:
wgethttps://raw.githubusercontent.com/ryfeus/lambda-packs/master/Tensorflow/buildPack.shwgethttps://raw.githubusercontent.com/ryfeus/lambda-packs/master/Tensorflow/index.pydockerpullamazonlinux:latestdockerrun-v$(pwd):/outputs--namelambdapackgen-damazonlinux:latesttail-f/dev/nulldockerexec-i-tlambdapackgen/bin/bash/outputs/buildPack.shforPython3:
wgethttps://raw.githubusercontent.com/ryfeus/lambda-packs/master/Tensorflow/buildPack_py3.shwgethttps://raw.githubusercontent.com/ryfeus/lambda-packs/master/Tensorflow/index_py3.pydockerpullamazonlinux:latestdockerrun-v$(pwd):/outputs--namelambdapackgen-damazonlinux:latesttail-f/dev/nulldockerexec-i-tlambdapackgen/bin/bash/outputs/buildPack_py3.shNote:RememberYoushouldsetpython3.6forAWSLambdafunctionenvironment.
LayerARNarn:aws:lambda:us-east-1:339543757547:layer:tensorflow-pack
SklearnIntroPackageforfansofmachinelearning,buildingmodelsandthelike.Idoubtthatthereisamoreconvenientwaytodeploymodeltotherealworld.
ToolsScikit-learn0.17.1Scipy0.17.0Documentationhttps://scikit-learn.org/
SkimageIntroPackageofimageprocessingtools,andnotonlytostyleimage,butalsoalargesetofcomputervisionalgorithms.
Therearecurrentlytwozippedpacksavailable,Pack.zipandPack_nomatplotlib.zip,youprobablywanttousePack_nomatplotlib.zip.Seehttps://github.com/ryfeus/lambda-packs/issues/5formoreinformation.
ToolsScikit-image0.12.3
Documentationhttps://scikit-image.org/
OpenCV+PILIntroAnotherpackageofimageprocessingtools,andnotonlytostyleimage,butalsoalargesetofComputervisionalgorithms.
ToolsOpenCV3.1.0PIL4.0.0Documentationhttps://pillow.readthedocs.io/
https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_tutorials.html
PandasIntroPackageforfansofstatistics,datascientistsanddataengineers.RAMatlambdais1.5gigabytes,andthemaximumoperatingtime-5minutes.Iamsurethatwillbeenoughformosttasks.
ToolsPandas0.19.0
Documentationhttps://pandas.pydata.org/
SpacyIntroOpensourcelibraryforNaturalLanguageProcessinginpython.
ToolsSpacy2.0.11Documentationhttps://spacy.io/
ExampleExamplecodeloadslanguagemodelfromS3andusesittoanalyzesentence.
TesseractIntroOCR(opticalcharacterrecognition)libraryfortextrecognitionfromtheimage.
Documentationhttps://github.com/tesseract-ocr/tesseract
PDFgenerator+Microsoftofficefilegenerator(docx,xlsx,pptx)+imagegenerator(jpg,png)+bookgenerator(epub)Intro"Helloworld"codeinpackagecreatesexampleofeverydocument.Basicallytheselibsarelowmemory(lessthan128MB)andhighspeed(lessthan0.5seconds)soit'ssomethinglike~1mdocumentsgeneratedper1$intermsofAWSLambdapricing.
Toolsdocx(python-docx-https://pypi.python.org/pypi/python-docx)xlsx(XlsxWriter-https://pypi.python.org/pypi/XlsxWriter)pptx(python-pptx-https://pypi.python.org/pypi/python-pptx)pdf(Reportlab-https://pypi.python.org/pypi/reportlab)epub(EbookLib-https://pypi.python.org/pypi/EbookLib)png/jpg/...(Pillow-https://pypi.python.org/pypi/Pillow)Satelliteimageryprocessing(rasterio+OSGEO+pyproj+shapely+PIL)IntroAWSLambdapackinPythonforprocessingsatelliteimagery.Basicallyitenablestodeploypythoncodeinaneasyandcheapwayforprocessingsatelliteimageryorpolygons.In“helloworld”codeofthepackIdownloadred,green,blueLandsat8bandsfromAWS,makeTrueColorimageoutofitanduploadittoS3.Ittakes35secondsand824MBofRAMforitso~2500scenescanbeprocessedfor1$.
ToolsRasterio(https://github.com/mapbox/rasterio0.36)OSGEO(https://trac.osgeo.org/gdal/wiki/GdalOgrInPython)Pyproj(https://github.com/jswhit/pyproj)Shapely(https://github.com/Toblerity/Shapely)PIL(https://pillow.readthedocs.io/)PyTorchPython3.6basedPyTorch
ToolsPyTorch1.0.1(CPU)torchvision0.2.1InstalledPackages(deps)numpy-1.16.1pillow-5.4.1six-1.12.0torchvision-0.2.1BuildPack#Youneed`docker`beforerun./build-with-docker.sh
评论