AtNest
简明飞快的异步爬虫框架（pytho3.6+），只有600行左右的代码
功能开箱即用的HTTP客户端
提供Itemextractor,可以明确地声明如何从respose解析数据(支持xpath,jpathorregex)
通过"esure_future"ad"as_completed"api提供方便的工作流
安装
pip istall at_est使用方式：
创建一个Demo项目:
>>> at_est -c examples自动会创建以下文件：
drwxr-xr-x 5 bruce staff 160 Ju 30 18:24 ats-rw-r--r-- 1 bruce staff 208 Ju 26 22:59 settigs.py假设我们想获取GitHub热门仓库，让我们创建一个"examples/ats/example2.py":
from at_est import *from yarl import URLclass GithubAt(At): """Crawl tredig repositories from github""" item_pipelies = [ ItemFieldReplacePipelie( ('meta_cotet', 'star', 'fork'), excess_chars=('\r', '\', '\t', ' ')) ] cocurret_limit = 1 # save the website`s ad your badwidth! def __iit__(self): super().__iit__() self.item_extractor = ItemExtractor(dict) self.item_extractor.add_patter( 'xpath', 'title', '//h1/strog/a/text()') self.item_extractor.add_patter( 'xpath', 'author', '//h1/spa/a/text()', default='Not foud') self.item_extractor.add_patter( 'xpath', 'meta_cotet', '//div[@class="repository-meta-cotet col-11 mb-1"]//text()', extract_type=ItemExtractor.EXTRACT_WITH_JOIN_ALL) self.item_extractor.add_patter( 'xpath', 'star', '//a[@class="social-cout js-social-cout"]/text()') self.item_extractor.add_patter( 'xpath', 'fork', '//a[@class="social-cout"]/text()') asyc def crawl_repo(self, url): """Crawl iformatio from oe repo""" respose = await self.request(url) # extract item from respose item = self.item_extractor.extract(respose) item['origi_url'] = respose.url await self.collect(item) # let item go through pipelies(be cleaed) self.logger.ifo('*' * 70 + 'I got oe hot repo!\' + str(item)) asyc def ru(self): """App etrace, our play groud""" respose = await self.request('https://github.com/explore') for url i respose.html_elemet.xpath( '/html/body/div[4]/div[2]/div/div[2]/div[1]/article//h1/a[2]/' '@href'): # crawl may repos with our corouties pool self.schedule_coroutie( self.crawl_repo(respose.url.joi(URL(url)))) self.logger.ifo('Waitig...')然后我们可以列出所有可运行的爬虫(在"examples"文件夹下)
>>> $at_est -lats.example2.GithubAt运行!(without debuglog):
>>> at_est -a ats.example2.GithubAtINFO:GithubAt:OpeigINFO:GithubAt:Waitig...INFO:GithubAt:**********************************************************************I got oe hot repo!{'title': 'NLP-progress', 'author': 'sebastiaruder', 'meta_cotet': 'Repository to track the progress i Natural Laguage Processig (NLP), icludig the datasets ad the curret state-of-the-art for the most commo NLP tasks.', 'star': '3,743', 'fork': '327', 'origi_url': URL('https://github.com/sebastiaruder/NLP-progress')}INFO:GithubAt:**********************************************************************I got oe hot repo!{'title': 'material-dashboard', 'author': 'creativetimofficial', 'meta_cotet': 'Material Dashboard - Ope Source Bootstrap 4 Material Desig Admihttps://demos.creative-tim.com/materi…', 'star': '6,032', 'fork': '187', 'origi_url': URL('https://github.com/creativetimofficial/material-dashboard')}INFO:GithubAt:**********************************************************************I got oe hot repo!{'title': 'mkcert', 'author': 'FiloSottile', 'meta_cotet': "A simple zero-cofig tool to make locally-trusted developmet certificates with ay ames you'd like.", 'star': '2,311', 'fork': '60', 'origi_url': URL('https://github.com/FiloSottile/mkcert')}INFO:GithubAt:**********************************************************************I got oe hot repo!{'title': 'pure-bash-bible', 'author': 'dylaaraps', 'meta_cotet': '
评论