HtmlExtractor 网页信息抽取组件_系统开发案例-程序员客栈

HtmlExtractor是一个Java实现的基于模板的网页结构化信息精准抽取组件，本身并不包含爬虫功能，但可被爬虫或其他程序调用以便更精准地对网页结构化信息进行抽取。HtmlExtractor是为大规模分布式环境设计的，采用主从架构，主节点负责维护抽取规则，从节点向主节点请求抽取规则，当抽取规则发生变化，主节点主动通知从节点，从而能实现抽取规则变化之后的实时动态生效。如何使用？HtmlExtractor由2个子项目构成，html-extractor和html-extractor-web。html-extractor实现了数据抽取逻辑，是从节点，html-extractor-web提供web界面来维护抽取规则，是主节点。html-extractor是一个jar包，可通过mave引用：<depedecy><groupId>org.apdplat</groupId><artifactId>html-extractor</artifactId><versio>1.1</versio></depedecy>html-extractor-web是一个war包，需要部署到Servlet/Jsp容器上。在html-extractor-web目录下运行mvjetty:ru就可以启动Servlet/Jsp容器jetty，之后打开浏览器访问：https://localhost:8080/html-extractor-web/api/查看自己定义的规则。注意：页面模板中定义的所有CSS路径和抽取表达式全部抽取成功，才算抽取成功，只要有一个CSS路径或抽取表达式失败，就是抽取失败。单机集中式使用方法：//1、构造抽取规则List<UrlPatter>urlPatters=ewArrayList<>();//1.1、构造URL模式UrlPatterurlPatter=ewUrlPatter();urlPatter.setUrlPatter("https://moey.163.com/\\d{2}/\\d{4}/\\d{2}/[0-9A-Z]{16}.html");//1.2、构造HTML模板HtmlTemplatehtmlTemplate=ewHtmlTemplate();htmlTemplate.setTemplateName("网易财经频道");htmlTemplate.setTableName("fiace");//1.3、将URL模式和HTML模板建立关联urlPatter.addHtmlTemplate(htmlTemplate);//1.4、构造CSS路径CssPathcssPath=ewCssPath();cssPath.setCssPath("h1");cssPath.setFieldName("title");cssPath.setFieldDescriptio("标题");//1.5、将CSS路径和模板建立关联htmlTemplate.addCssPath(cssPath);//1.6、构造CSS路径cssPath=ewCssPath();cssPath.setCssPath("div#edText");cssPath.setFieldName("cotet");cssPath.setFieldDescriptio("正文");//1.7、将CSS路径和模板建立关联htmlTemplate.addCssPath(cssPath);//可象上面那样构造多个URLURL模式urlPatters.add(urlPatter);//2、获取抽取规则对象ExtractRegularextractRegular=ExtractRegular.getIstace(urlPatters);//注意：可通过如下3个方法动态地改变抽取规则//extractRegular.addUrlPatters(urlPatters);//extractRegular.addUrlPatter(urlPatter);//extractRegular.removeUrlPatter(urlPatter.getUrlPatter());//3、获取HTML抽取工具HtmlExtractorhtmlExtractor=ewDefaultHtmlExtractor(extractRegular);//4、抽取网页Strigurl="https://moey.163.com/08/1219/16/4THR2TMP002533QK.html";HtmlFetcherhtmlFetcher=ewJSoupHtmlFetcher();Strightml=htmlFetcher.fetch(url);List<ExtractResult>extractResults=htmlExtractor.extract(url,html);//5、输出结果iti=1;for(ExtractResultextractResult:extractResults){System.out.pritl((i++)+"、网页"+extractResult.getUrl()+"的抽取结果");if(!extractResult.isSuccess()){System.out.pritl("抽取失败：");for(ExtractFailLogextractFailLog:extractResult.getExtractFailLogs()){System.out.pritl("\turl:"+extractFailLog.getUrl());System.out.pritl("\turlPatter:"+extractFailLog.getUrlPatter());System.out.pritl("\ttemplateName:"+extractFailLog.getTemplateName());System.out.pritl("\tfieldName:"+extractFailLog.getFieldName());System.out.pritl("\tfieldDescriptio:"+extractFailLog.getFieldDescriptio());System.out.pritl("\tcssPath:"+extractFailLog.getCssPath());if(extractFailLog.getExtractExpressio()!=ull){System.out.pritl("\textractExpressio:"+extractFailLog.getExtractExpressio());}}cotiue;}Map<Strig,List<ExtractResultItem>>extractResultItems=extractResult.getExtractResultItems();for(Strigfield:extractResultItems.keySet()){List<ExtractResultItem>values=extractResultItems.get(field);if(values.size()>1){itj=1;System.out.pritl("\t多值字段:"+field);for(ExtractResultItemitem:values){System.out.pritl("\t\t"+(j++)+"、"+field+"="+item.getValue());}}else{System.out.pritl("\t"+field+"="+values.get(0).getValue());}}System.out.pritl("\tdescriptio="+extractResult.getDescriptio());System.out.pritl("\tkeywords="+extractResult.getKeywords());}多机分布式使用方法：1、运行主节点，负责维护抽取规则：方法一：在html-extractor-web目录下运行mvjetty:ru。方法二：在html-extractor-web目录下运行mvistall，然后将target/html-extractor-web-1.0.war部署到Tomcat。2、获取一个HtmlExtractor的实例（从节点），示例代码如下：StrigallExtractRegularUrl="https://localhost:8080/HtmlExtractorServer/api/all_extract_regular.jsp";StrigredisHost="localhost";itredisPort=6379;ExtractRegularextractRegular=ExtractRegular.getIstace(allExtractRegularUrl,redisHost,redisPort);HtmlExtractorhtmlExtractor=ewDefaultHtmlExtractor(extractRegular);3、抽取信息，示例代码如下：Strigurl="https://moey.163.com/08/1219/16/4THR2TMP002533QK.html";HtmlFetcherhtmlFetcher=ewJSoupHtmlFetcher();Strightml=htmlFetcher.fetch(url);List<ExtractResult>extractResults=htmlExtractor.extract(url,html);iti=1;for(ExtractResultextractResult:extractResults){System.out.pritl((i++)+"、网页"+extractResult.getUrl()+"的抽取结果");if(!extractResult.isSuccess()){System.out.pritl("抽取失败：");for(ExtractFailLogextractFailLog:extractResult.getExtractFailLogs()){System.out.pritl("\turl:"+extractFailLog.getUrl());System.out.pritl("\turlPatter:"+extractFailLog.getUrlPatter());System.out.pritl("\ttemplateName:"+extractFailLog.getTemplateName());System.out.pritl("\tfieldName:"+extractFailLog.getFieldName());System.out.pritl("\tfieldDescriptio:"+extractFailLog.getFieldDescriptio());System.out.pritl("\tcssPath:"+extractFailLog.getCssPath());if(extractFailLog.getExtractExpressio()!=ull){System.out.pritl("\textractExpressio:"+extractFailLog.getExtractExpressio());}}cotiue;}for(ExtractResultItemextractResultItem:extractResult.getExtractResultItems()){System.out.prit("\t"+extractResultItem.getField()+"="+extractResultItem.getValue());}System.out.pritl("\tdescriptio="+extractResult.getDescriptio());System.out.pritl("\tkeywords="+extractResult.getKeywords());}

HtmlExtractor 网页信息抽取组件

技术信息

作品详情

功能介绍

重点城市程序员兼职推荐

重点岗位程序员兼职推荐