![基于分布式离线技术的电量数据批量高速处理方法及系统](/CN/2016/1/176/images/201610880143.jpg)
基本信息:
- 专利标题: 基于分布式离线技术的电量数据批量高速处理方法及系统
- 专利标题(英):Electricity consumption data batch high speed processing method and system based on distributed offline technology
- 申请号:CN201610880143.5 申请日:2016-10-09
- 公开(公告)号:CN106502772A 公开(公告)日:2017-03-15
- 发明人: 蒋锦霞 , 朱青山 , 裴旭斌 , 戴波 , 王红凯 , 何东 , 闫娇娇
- 申请人: 国网浙江省电力公司信息通信分公司 , 国家电网公司 , 浙江华云信息科技有限公司
- 申请人地址: 浙江省杭州市西湖区黄龙路8号
- 专利权人: 国网浙江省电力公司信息通信分公司,国家电网公司,浙江华云信息科技有限公司
- 当前专利权人: 国网浙江省电力公司信息通信分公司,国家电网公司,浙江华云信息科技有限公司
- 当前专利权人地址: 浙江省杭州市西湖区黄龙路8号
- 代理机构: 浙江翔隆专利事务所
- 代理人: 王晓燕
- 主分类号: G06F9/48
- IPC分类号: G06F9/48 ; G06F17/30
The invention provides an electricity consumption data batch high speed processing method and system based on distributed offline technology, and relates to a data batch processing method system. At present, there is a lacking of the high speed mass data storage and calculation model, and not all the types and forms of mass data storage and calculation can be fulfilled. The method comprises the steps of (1) an electricity consumption front sampler sending the sampled electricity consumption data to a Kafka data queue for buffering in a real-time mode, (2) Storm cluster reading the electricity consumption information in the Kafka queue, and storing the information in a real-time mode to a Hbase, (3) Spark extracting the electricity consumption information to be processed from the Hbase, and uploading the information to a Hive data sheet, (4) through Spark conducting the offline operation of related Hive data sheet to acquire the electricity consumption of the current period, and processing related abnormalities. The method combines the advantages of both the Storm and the Spark, enhances the overall processing capacity. From the dynamic migration technology of transaction security node, a transaction security protocol of complete task node migration is proposed. On the foundation of the guarantee of no packet loss, no repetition in the migration process, at the same time the execution efficiency of the migration itself is enhanced, and the system stability is elevated.
IPC结构图谱:
G | 物理 |
--G06 | 计算;推算;计数 |
----G06F | 电数字数据处理 |
------G06F9/00 | 电数字数据处理的控制单元 |
--------G06F9/06 | .应用存入的程序的,即应用处理设备的内部存储来接收程序并保持程序的 |
----------G06F9/22 | ..微控制或微程序装置 |
------------G06F9/48 | ...程序启动;程序切换,例如通过中断 |