基于hadoop的网络数据分析.pptx
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- 基于 hadoop 网络 数据 分析
- 资源描述:
-
,2012-03-12,#,Hadoop-based Internet Traffic Analysis,2012-03-14,Youngseok Lee,leecnu.ac.kr,Chungnam National University,1,Gartner:10 Key IT Trends for 2012,2,Data Explosion,Digital universe,2005:130 ExaByte,2 time per two years,90,%of the data in the world today was created within the past two years,Motivation,Cheap storage,Mobile computing,3,Big Data,Big,data,data sets whose size is beyond the ability of commonly used software tools to capture,manage,and process the data within a tolerable elapsed,time,Example,Walmart:1 million customer transactions every hour,2.5 PB of data,Facebook:40 billion photos from users,2.5,PB of user data+15 TB/day(4/2009),eBay has 6.5 PB of user data+50 TB/day(5/2009),4,Predicting Flu Trends with Queries,Ginsberg,J.,et al.(2009).Detecting,influenza epidemics using search engine query,data.Nature,457,1012-1014.,5,Cutting Edge of Distributed/Parallel Computing,Distributed:MapReduce,1 PB sorting by Google,2008:6 hours and 2 minutes on 4,000 computers,2011:33 minutes on 8000 computers,Parallel:multi-core CPU,GPU,FPGA,40Gbps IP forwarding capability,S.Han et al.,“PacketShader:A GPU-Accelerated Software Router“ACM SIGCOMM,2010,6,Open-source framework for running applications on large clusters built of commodity hardware,Implementation of MapReduce and HDFS,MapReduce:computational paradigm,HDFS:distributed file system,7,MapReduce,MapReduce is a programming model for large data set,First suggested by Google,J.Dean and S.Ghemawat,“MapReduce:Simplified Data Processing on Large Cluster,”OSDI,2004 8,User only specify a map and a reduce function,Distributed grep,Count of URL access frequency,Reverse web-link graph,Term-vector per host,Inverted index,Distributed sort,8,Data from Internet Measurement,Web,web doc,blog,twitter,IP packets,raw,header,dns,Flow,Cisco NetFlow v5/v9,Log,Server,switch,router,sensor,machine,appliances,Management data,SNMP MIB,Routing table,BGP update,table,Active probing,ping,traceroute,skitter,9,How Large?,Type Packets,4.6TB for 1 hr in 10GE,NetFlow,1.2GB for 1 day for a router in CNU,Routing table,Routeviews in bzip2,RIBS:45MB every 2 hrs-540MB for 1 day-197GB for 1 yr,Updates:1MB for every 15 minutes,10 Gbps,10 Gbps,10 Gbps,4.5 TB,10,CAIDA Data,Data sources,Macroscopic topology data(IPv4/IPv6)by Archipelago,Traffic traces at Internet core backbone links,Traffic traces from UCSD Network Telescope,M.Fomenkov and k.claffy,“Internet measurement data management challenges,in Workshop on Research Data Lifecycle Management.Jul 2011,Workshop on Research Data Lifecycle Management.,11,Tools or Platform for Measurement and Analysis,Open source,CAIDA,Active probing tools,Ping,traceroute,skitter,mper,scamper,Archipelago,Passive tools,CoralReef,flow-tools,tcpdump/wireshark,Bro,snort,12,Our Research,Problem statement,How to handle a large set of packet/flow data?,Input:packet/flow files/streaming,Output:statistics/analytics results,Ex)top 10 heavy users,popular applications,favorite websites,anomaly detection,Proposal,Leverage the distributed parallel computing platform,Googles programming model,MapReduce,Open-source system,Hadoop,13,1 An Internet Traffic Analysis Method with MapReduce,Cloudman workshop,April 2010,2,“,A Hadoop-based Packet Trace Processing Tool,TMA 2011,April,2011,Internet Traffic Measurement&Analysis,Packet,HTTP request packet,DNS packet,Abstract of a set of packets:,flow,(168.188.1.10,61.10.1.1,3000,80)1MB,1000 packets,30 secs,Flow,Data,Routers,High Performance Server,Storage,14,Packet,Data,Fiber splitter,Architecture of Traffic Measurement and Analysis System,Institution C,Institution A,Hadoop-based,Traffic Measurement and Analysis Cloud,Hadoop-based Traffic Measurement and Analysis Cloud,(Public),Institution B,Institution D,flow streaming data,packet streaming data,collected packet file,packet streaming data,Administrator A,Administrator B,Administrator C,collected flow file,Slave,Master,Traffic,Collector,Web Visualizer,Pcap,InputFormat,Text Input/Output Format,Traffic Analysis,Mapper&Reducer,NetFlow,Traffic Analyzer,HDFS,Hadoop,Packets,Binary Input/Output Format,Hadoop I/O Formats,15,Our Tools,Flow processing tool,MapReduce algorithm to summarize flow statistics,Packet processing tool,16,Analysis in Hadoop,17,1 TB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,Reduce Task,Map Task,Map Task,Map Task,Map Task,Reduce Task,Map Task,Map Task,Map Task,Map Task,MapReduce,HDFS,Packet Analysis in Hadoop,18,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,TextInputFormat,1 TB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,Text,MapReduce,HDFS,Packet Analysis in Hadoop,19,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,1 TB,Separator?,Which InputFormat?,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,InputFormat in Hadoop,20,Detect a packet in the HDFS block,00 00 68 2B AD 4C 38 A4 04 00 5C 00 00 00 5C 00 00 00 FF FF 00 21 B5 01 68 2B AD 4C 2B 1C 07 00 3C 00 00 00 3C 00 00 00 01 80,Read Packets from HDFS,21,Block2,Block3,New InputFormat,00 00 68 2B AD 4C 38 A4 04 00 5C 00 00 00 5C 00 00 00 FF FF 01 68 2B AD 4C 2B 1C 07 00 3C 00 00 00 3C 00 00 00 01 80,Detect the first Packet,22,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-01 2010-10-10,block,boundary,92 bytes,Start time:2010-10-01,End time:2010-10-10,time :60 sec,16 Byte,16 Byte,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,Start time:2010-10-01,End time:2010-10-10,time :60 sec,Start time:2010-10-01,End time:2010-10-10,time :60 sec,60 sec,OK!,OK!,OK!,Packet Analysis Architecture,23,P,3,I/O Formats,Hadoop,HDFS,Packet,Collector,Packet Analyzer,(Mapper&Reducer),packet,packet trace file,BinaryInput/,OutputFormat,Pcap,InputFormat,TextInput/,OutputFormat,Our Packet Analysis Tool,24,Traffic,Analysis,P,3,Command,MapReduce,Jobs,CoralReef,Total traffic and host/port count statistics,PcapTotalStats,count up bytes/packets,emit unique IP/port and flow,crl_stats,Csource=pcap file,summarize#of unique,IP/Port and flows,Periodic flow statistics,PcapTotalFlowStats,generate flows and count up,bytes/packets,crl_flow I b Ci=interval,Csource=pcap file,/t2_rate-s,summarize,periodical flow statistics,Periodic simple traffic statistics,PcapRate,compute periodic bytes/packets regarding IPv4/v6/non-IP per interval,crl_rate Ci=interval,Csource=pcap file,Top N,PcapTopN,sort records and emit top N record,command,line/t2_top Sb nn,Total,c,ount grouping by key,PcapCountUp,count,up bytes/packets per key,-,Performance Evaluation,25,Testbed,Packet Trace files,Result,26,Conclusion,Hadoop fits well into big Internet data analysis,Future Internet design and modeling,Hadoop,One size does not fit all!,Hadoop+DB,The,data center,is the computer,!,More data usually beats better algorithm,27,展开阅读全文

咨信网声明:本文档由用户上传并分享,仅供学习交流、研究之用,未经授权,严禁复制、发行、汇编、翻译或网络传播等,侵权必究。
咨信网小贴示:
1、浏览或下载可查看了解,请点击【咨信网告知】。
2、下载文件中如有侵权或不适当内容,将立即纠正。
3、如果因为网速或其他原因下载失败请重新下载,重复下载[60天内]不扣币。
4、开具发票登录电脑端申请,请点击【索取发票】。
咨信网客服组:【微信客服】
1、浏览或下载可查看了解,请点击【咨信网告知】。
2、下载文件中如有侵权或不适当内容,将立即纠正。
3、如果因为网速或其他原因下载失败请重新下载,重复下载[60天内]不扣币。
4、开具发票登录电脑端申请,请点击【索取发票】。
咨信网客服组:【微信客服】


基于hadoop的网络数据分析.pptx
















自信AI助手














微信客服
客服QQ
发送邮件
意见反馈



链接地址:https://www.zixin.com.cn/doc/14770661.html