数据立方体是复杂计算的抽象。Datacube是用Java实现的,可插入数据库后端支持的数据立方体。
datacube是用来存储大数据点的聚合信息。数据立方体存储的是有趣输入数据点的子集。比如,你正在编写一个web服务器日志分析工具,你的输入点可能是日志行,你可能会计算每个浏览器的类型,每个浏览器的版本,操作系统类型,操作系统版本和其他属性。同时你可能会需要计算一个特定的组合计数(浏览器类型,浏览器版本,操作系统类型),(浏览器类型,浏览器版本,操作系统类型,操作系统版本),等等。
这对快速添加和修改计数是个很大的挑战,会浪费很多时间在数据库代码和重新用新计数器处理旧数据。而数据立方体就可以帮忙解决这些问题。
UrbaAirship使用datacube项目来支持他们的移动端应用的分析栈,每个节点每秒处理大约10K的事件。
datacube要求JDK1.6。
特性性能:高速异步IO后端处理
使用HadoopMapReduce进行批量加载
可插入数据库接口
datacube暂时只支持HBase数据库后端。
示例:
IdService idService = ew CachigIdService(5, ew MapIdService());CocurretMap<BoxedByteArray,byte[]> backigMap = ew CocurretHashMap<BoxedByteArray, byte[]>();DbHaress<LogOp> dbHaress = ew MapDbHaress<LogOp>(backigMap, LogOp.DESERIALIZER, CommitType.READ_COMBINE_CAS, idService);HourDayMothBucketer hourDayMothBucketer = ew HourDayMothBucketer();Dimesio<DateTime> time = ew Dimesio<DateTime>("time", hourDayMothBucketer, false, 8);Dimesio<Strig> zipcode = ew Dimesio<Strig>("zipcode", ew StrigToBytesBucketer(), true, 5);DataCubeIo<LogOp> cubeIo = ull;DataCube<LogOp> cube;Rollup hourAdZipRollup = ew Rollup(zipcode, time, HourDayMothBucketer.hours);Rollup dayAdZipRollup = ew Rollup(zipcode, time, HourDayMothBucketer.days);Rollup hourRollup = ew Rollup(time, HourDayMothBucketer.hours);Rollup dayRollup = ew Rollup(time, HourDayMothBucketer.days);List<Dimesio<?>> dimesios = ImmutableList.<Dimesio<?>>of(time, zipcode);List<Rollup> rollups = ImmutableList.of(hourAdZipRollup, dayAdZipRollup, hourRollup, dayRollup);cube = ew DataCube<LogOp>(dimesios, rollups);cubeIo = ew DataCubeIo<LogOp>(cube, dbHaress, 1, Log.MAX_VALUE, SycLevel.FULL_SYNC);DateTime ow = ew DateTime(DateTimeZoe.UTC);// Do a icremet of 5 for a certai time ad zipcodecubeIo.writeSyc(ew LogOp(5), ew WriteBuilder(cube) .at(time, ow) .at(zipcode, "97201"));// Do a icremet of 10 for the same zipcode i a differet hour of the same dayDateTime differetHour = ow.withHourOfDay((ow.getHourOfDay()+1)%24);cubeIo.writeSyc(ew LogOp(10), ew WriteBuilder(cube) .at(time, differetHour) .at(zipcode, "97201"));// Read back the value that we wrote for the curret hour, should be 5 Optioal<LogOp> thisHourCout = cubeIo.get(ew ReadBuilder(cube) .at(time, HourDayMothBucketer.hours, ow) .at(zipcode, "97201"));Assert.assertTrue(thisHourCout.isPreset());Assert.assertEquals(5L, thisHourCout.get().getLog());// Read back the value we wrote for the other hour, should be 10Optioal<LogOp> differetHourCout = cubeIo.get(ew ReadBuilder(cube) .at(time, HourDayMothBucketer.hours, differetHour) .at(zipcode, "97201"));Assert.assertTrue(differetHourCout.isPreset());Assert.assertEquals(10L, differetHourCout.get().getLog());// The total for today should be the sum of the two icremetsOptioal<LogOp> todayCout = cubeIo.get(ew ReadBuilder(cube) .at(time, HourDayMothBucketer.days, ow) .at(zipcode, "97201"));Assert.assertTrue(todayCout.isPreset());Assert.assertEquals(15L, todayCout.get().getLog());






评论