web数据挖掘.ppt

,单击此处编辑母版标题样式,单击此处编辑母版文本样式,第二级,第三级,第四级,第五级,*,Web Usage Mining,*,Web Usage Mining,Introduction and Trends,演讲人：阮备军,2026/1/27 周二,1,Web Usage Mining,提纲,1.简介,1.1,WUM,的定义和应用,1.2,WUM,基本的过程,1.3 原型系统和商用系统,2.趋势,2026/1/27 周二,2,Web Usage Mining,Web Mining,的分类12：,1.,Content Mining,2.Structure Mining,3.Usage Mining,Web Mining,分类,2026/1/27 周二,3,Web Usage Mining,1.1.1 定义,(,Jaideep,Srivastava1),“the process of applying data mining techniques,to the discovery of usage patterns from Web data”,2026/1/27 周二,4,Web Usage Mining,1.1.2功能,System Improvement,1).,Site Improvement,2).,Web Caching and Network Transmission,(E.Cohen5),2026/1/27 周二,5,Web Usage Mining,1.1.2功能,1).,Site Improvement,根据实际用户的浏览情况，调整网站的网页的连接结构和内容，更好的服务用户,A=B=C=D,A=D,极端：,Adaptive web sites(Mike Perkowitz78),2026/1/27 周二,6,Web Usage Mining,1.1.2功能,2).,Caching&Network Transmission,(E.Cohen5),例如：从,proxy,的访问信息中可以分析用户的访问模式，,从而可以预测用户的,Page,访问，提高,Web Caching,的性能,A=B=C,A=B=D,A=B,Cached:C、D,2026/1/27 周二,7,Web Usage Mining,1.1.2功能,*与传统的销售途径不同的是,：,网上零售系统可捕捉到大量的采购过程的细节，提供了更加深入分析的可能,2026/1/27 周二,8,Web Usage Mining,1.1.2功能,Personalization,定义：,根据发现的用户喜好，动态地为用户定制观看的内容,或提供浏览建议。,直接实现形式,：,Recommender,系统,(10,J.Ben Schafer),。,作用：,1)方便用户查询和浏览,2)增强广告的作用,3)促进网上销售,4)提高用户忠诚度,2026/1/27 周二,9,Web Usage Mining,1.2 基本的过程,*1).,Data Gathering,2).Preprocessing,3).Pattern discovery,4).Patterns analysis,2026/1/27 周二,10,Web Usage Mining,1.2 基本的过程,Site Files,Raw Usage Data,User Profile,Site Content&Structure,Server/Session/Episode Data,Usage Statistics,User Clusters,Association Rules,Sequential Patterns,Knowledge,1).,Data Gathering,2).Preprocessing,4).Patterns analysis,3).Pattern discovery,2026/1/27 周二,11,Web Usage Mining,1.2.1,Data Gathering,可以利用的,Web Data,包括:,1).,Content:,页面的实际内容,2).,Structure:,包括,intra-structure,和,inter-structure,3).,Usage:,例如：,Click Stream,4).User Profile:,例如：,registration data,和,customer profiles,5.Business Data？,2026/1/27 周二,12,Web Usage Mining,1.2.1,Data Gathering,数据来源:,1),Server Level Collection,2),Client Level Collection,3),Proxy Level Collection,2026/1/27 周二,13,Web Usage Mining,1.2.1,Data Gathering,Server Level Collection,:,Web Log File,2)Packet Sniffing,技术 缺点是扩展性差（加密通道和用户跟踪）,3)Web Page Content&structure,4)Application Server,2026/1/27 周二,14,Web Usage Mining,1.2.1,Data Gathering,Client Level Collection:,优点是可以比较全面和准确收集到用户数据,Applet&Script,缺点:不支持代理，功能会被用户关闭。,Modified browser,优点是全面和准确收集到用户数据，缺点是需要吸引或强制用户使用,2026/1/27 周二,15,Web Usage Mining,1.2.1,Data Gathering,Proxy Level Collection:,这种数据收集方式适合有大量静态页面的网站,2026/1/27 周二,16,Web Usage Mining,1.2.2,Preprocessing,1),.Content&Structure,2),.,Usage Data,2026/1/27 周二,17,Web Usage Mining,1.2.2,Preprocessing,Content&Structure:,解决两个问题,第一、,page,的独特性的表示方法？,第二、,page,的内容和结构的量化形式（,quantifiable form）,2026/1/27 周二,18,Web Usage Mining,1.2.2,Preprocessing,structure,举例：,ABC Site Map,E,index,1,A,B,1,C,F,Navigation,Mixed,Media,Page Type,2026/1/27 周二,19,Web Usage Mining,1.2.2,Preprocessing,ABC Site Map,的结构可表示为：,M=,F1=index,(frame,1,left|frame,A,main),F2=1,(get,A,main),(get,C,main),F3=A,(get,B,top),F4=C,(get,E,top),(get,F,top),F5=B,F6=E,F7=F,2026/1/27 周二,20,Web Usage Mining,1.2.2,Preprocessing,Index=1-A=1-C=E,Index=1-A=B,Map,的结构表示了用户可能的,click,访问的路径,2026/1/27 周二,21,Web Usage Mining,1.2.2,Preprocessing,Content Classification:,1),Usage Type,2),Content Features,*具体种类的划分跟应用的环境相关,2026/1/27 周二,22,Web Usage Mining,1.2.2,Preprocessing,Usage Type,1)Head Page,例如：,Index,2)Media Page,例如：,B,E,F,3)Navigation Page,例如：,1,C,4)Mixed Page,例如：,A,2026/1/27 周二,23,Web Usage Mining,1.2.2,Preprocessing,Content Features,(22,Soumen,Chakrabarti,),:,1。可以使用,vector space model,来表示,page,的内容：,把,page,的示成一个在此高维空间中的一个向量,page,出现的,term,作为向量的维,维对应的值是通过某种方式计算出来的权重(例如：,TFIDF),它反映了,term,在此,page,区别其他,page,的时候的重要程度。,2。,Clustering&Classification,2026/1/27 周二,24,Web Usage Mining,1.2.2,Preprocessing,Usage Data:,数据的抽象4,1.,User,2.,Page View(Click),用户在某个时刻看到的浏览器中的内容,3.,Click Stream,一组,连续,的,page view,请求,4.,User Session,(transaction),用户的阶段性的,click stream,可以跨多个服务器,5.,Server Session,用户的,在一个服务器上的阶段性的,click stream,6.,Episode,用户,session,中的一段有意义的,click stream,2026/1/27 周二,25,Web Usage Mining,1.2.2,Preprocessing,Usage Data:,数据的抽象4,Raw Data,Page View,Click Stream,User Session,Server Session,Episodes,2026/1/27 周二,26,Web Usage Mining,1.2.2,Preprocessing,Usage Preprocessing,的流程,1.数据清理,2.识别,User/Session,3.,识别,Page View,4.识别,Episode,2026/1/27 周二,27,Web Usage Mining,1.,Data cleaning:,*,把,log,条目分割成对应的数据项目,*剔除图片或其它非,page,类的,log,条目,*剔除,spider/agent,的访问条目,*规格化,URI,*,提取,POST,数据项,1.2.2,Preprocessing,2026/1/27 周二,28,Web Usage Mining,1.2.2,Preprocessing,清理后的,Sample Log,IP Address,Time/Date,Method/URI,Referrer,Agent,202.120.224.4,15:30:01/2-,Jan-01,GET,Index.htm,,ok.edu/link.htm,Mozilla/4.0(IE5.0W98),202.120.224.4,15:30:01/2-,Jan-01,GET 1.htm,,ex.edu/index.htm,Mozilla/4.0(IE5.0W98),202.120.224.4,15:30:01/2-,Jan-01,GET,A.htm,,ex.edu/index.htm,Mozilla/4.0(IE5.0W98),202.120.224.4,15:37:09/2-,Jan-01,GET,E.htm,,ex.edu/C.htm,Mozilla/4.0(IE5.0W98),202.120.224.4,15:33:04/2-,Jan-01,GET,Index.htm,,ok.edu/res.php,Mozilla/4.0(IE4.0NT),202.120.224.4,15:33:04/2-,Jan-01,GET 1.htm,,ex.edu/index.htm,Mozilla/4.0(IE4.0NT),202.120.224.4,15:33:04/2-,Jan-01,GET,A.htm,,ex.edu/index.htm,Mozilla/4.0(IE4.0NT),202.120.224.4,15:35:11/2-,Jan-01,GET,B.htm,,ex.edu/A.htm,Mozilla/4.0(IE4.0NT),202.120.224.4,15:35:11/2-,Jan-01,GET,C.htm,,ok.edu/A.htm,Mozilla/4.0(IE5.0W98),2026/1/27 周二,29,Web Usage Mining,2.,User and Session identification:,1.2.2,Preprocessing,1.,IP Address&Agent,2.Embedded Session ID,3.Registration(User Profile),4.Cookie,5.Software Agent(,Applet&Scrtipt,),6.Modified Browser,2026/1/27 周二,30,Web Usage Mining,在只有,IP/Agent,的情况下，我们假设,User,和,Session,是等同*的:,1.2.2,Preprocessing,15:33:04/2-,Jan-01,GET,Index.htm,,ok.edu/res.php,15:33:04/2-,Jan-01,GET 1.htm,,ex.edu/index.htm,15:33:04/2-,Jan-01,GET,A.htm,,ex.edu/index.htm,15:35:11/2-,Jan-01,GET,B.htm,,ex.edu/A.htm,15:30:01/2-,Jan-01,GET,Index.htm,,ok.edu/link.htm,15:30:01/2-,Jan-01,GET 1.htm,,ex.edu/index.htm,15:30:01/2-,Jan-01,GET,A.htm,,ex.edu/index.htm,15:37:09/2-,Jan-01,GET,E.htm,,ex.edu/C.htm,15:35:11/2-,Jan-01,GET,C.htm,,ok.edu/A.htm,Mozilla/4.0(IE5.0W98),202.120.224.4,User1:,202.120.224.4,Mozilla/4.0(IE4.0NT),User2:,2026/1/27 周二,31,Web Usage Mining,3.Page View Identification,:,1.2.2,Preprocessing,1-,A,,ok.edu/res.php,B,A.htm,1-,A,,ok.edu/link.htm,E,C.htm,1-,C,A.htm,Mozilla/4.0(IE5.0W98),202.120.224.4,User1:,202.120.224.4,Mozilla/4.0(IE4.0NT),User2:,2026/1/27 周二,32,Web Usage Mining,Path Completion,解决由于,Cache,带来的问题路径不全的问题,1.2.2,Preprocessing,time,1-,A,1-,C,1-,E,1-,C,1-,F,Session_1,Session_2,2026/1/27 周二,33,Web Usage Mining,4.Episode Identification,:,1.2.2,Preprocessing,定义:,meaningful subset of user/server session,Auxiliary-Media,Episodes:,time,A,A,M,M,A,A,A,M,A,Episode1,Episode2,Episode3,2026/1/27 周二,34,Web Usage Mining,Auxiliary-Media,Episodes:,1.2.2,Preprocessing,可以使用的方法包括：,1,）Page type,2）Reference length,3）Maximal Forward Reference,2026/1/27 周二,35,Web Usage Mining,1).,Page type Episode,:,根据,page view,的类型来确定,1.2.2,Preprocessing,time,A,A,M,M,A,A,A,M,A,Episode1,Episode2,Episode3,Session X,缺点是，,必须要求每个,page view,有,固定,的,用途,。,2026/1/27 周二,36,Web Usage Mining,2).,Reference Length Episode,:,page view,的类型是根据用户在此处停留的时间来确定。,停留时间相对长的，就认为是,Media Page,停留时间短的则是一个,Auxiliary Page,1.2.2,Preprocessing,2026/1/27 周二,37,Web Usage Mining,2).,Maximal Forward Reference Episode,23,chen,它的假设是：一个,episode,是访问某个,Media page,的一个最大深度的,page view,引用路径，即从用户,session,的第一个,page view,开始，一直向前访问，直到有一个,backward page view,出现,1.2.2,Preprocessing,Index 1-A 1-C E,1-C B,Time,Episode1,Episode2,2026/1/27 周二,38,Web Usage Mining,1.2.3,Pattern Discovery,其中有：,1)Statistical Analysis,2)Frequent,Itemsets,and Association Rules,3)Clustering&Classification,4)Sequential Patterns,5)Dependency Modeling,2026/1/27 周二,39,Web Usage Mining,1.2.3,Pattern Discovery,1)Statistical Analysis,主要用于改进系统的,Performance,design,等,包括：1),most frequently accessed pages,2)average view time of a page,3)average length of a path through a site,2026/1/27 周二,40,Web Usage Mining,1.2.3,Pattern Discovery,2)Frequent Item-sets and Association Rules,可以寻找出经常频繁访问的,page,组，,可用于修改,Web Site,的设计或提前缓冲页面，改进系统的性能。,2026/1/27 周二,41,Web Usage Mining,1.2.3,Pattern Discovery,3)Clustering&Classification,包括两方面的应用：,*user,用于,Market segmentation(,市场分割)和个人内容定制,*page(content),后者主要用于,IR,和,Surf Assistance,2026/1/27 周二,42,Web Usage Mining,1.2.3,Pattern Discovery,5)Sequential Patterns,可用于用户的,visit pattern.,包括：,1.,trend analysis,2.change point detection,3.similarity Analysis.,2026/1/27 周二,43,Web Usage Mining,1.2.3,Pattern Discovery,6)Dependency Modeling,不仅提供了理论化分析用户行为的框架，同时也可以用来，提高网上产品销量，提高用户的访问提供方便,可以使用的方法有：,1）Markov Model,2）Bayesian Brief Network,2026/1/27 周二,44,Web Usage Mining,1.2.4,Pattern Analysis,目的是根据实际应用，通过用户的选择和观察，把发现的,rules,patterns,和,statistics,转换为知识,Knowledge。,1）Information filtering(,例如：,SQL),2）Visualization,3)OLAP,4),Knowledge Query,2026/1/27 周二,45,Web Usage Mining,1.3,原型系统和商用系统,2026/1/27 周二,46,Web Usage Mining,IBM:,SpeedTracer,17,从日志文件中重建,user traversal path,然后识别出,user session;,在此基础上，利用数据挖掘算法发现 最常见的路径 和 经常访问页面组,;,系统提供三种统计类型的报告：,1),user-based,例如：频繁访问本站的前,N,名的用户地址,2),path-based,例如：被频繁访问的前,N,名的路径,3),group-based,例如：被频繁访问的前,N,名的,page group,2026/1/27 周二,47,Web Usage Mining,WebLogMiner,14,15,经过清理的,log,数据以,DataCube,的形式存储在,Database,中,提供,OLAP,提供包括序列模式,和,关联规则在内的多种数据挖掘的方法,Simon Fraser University,2026/1/27 周二,48,Web Usage Mining,Web Utilization Miner,完整的,log preparation,querying,和,visualization,主要提供,sequence pattern,的发现,提供,MINT,查询语言,3)提供树结构的,sequence pattern,显示方式,wum.wiwi.hu-berlin.de,/,德国柏林,Humboldt,大学商学院,2026/1/27 周二,49,Web Usage Mining,2.,Trends,2026/1/27 周二,50,Web Usage Mining,PAKDD,2001,1.,Server Log,提供的可用信息太少,2.动态页面的大量使用使得分析,log,更为困难,3.,Session,的分析一直是个难点19,4.其他一些数据没有记录：例如：,Search keywords,5.Crawlers,的过滤,6.,巨大的数据量及其自动转换,7.Market Level insight,Ron,Kohavi,:,Mining E-Commerce Data:The Good,the Bad,and the Ugly.,*,WUM,在,E-Commerce,领域内的应用还刚起步,2026/1/27 周二,51,Web Usage Mining,WEBKDD2000,主题：,Web Mining for E-Commerce-Challenges and Opportunities,2026/1/27 周二,52,Web Usage Mining,16 提出了,Recommender,系统的在,E-Commerce,领域内的发展机遇和挑战：,1）,结合多种用户数据的,建立,subtle recommender,系统,*,Demographic Information,例如：,register data,*Purchase Data,*,Explict,Rating,例如：用户对产品的评价,*,Ownership Data,2),Recommender,系统和市场分析决策结合(*,),3）,用户数据的共享,4),道德问题,5）,扩展性*,Recommender/Personalization,系统,可能的切入点,2026/1/27 周二,53,Web Usage Mining,WUM,技术与现有的,E-Commerce,系统集成,可能的切入点,2026/1/27 周二,54,Web Usage Mining,改造和构造新的算法，把原有旧的数据源（例如,Web Log）,和新的数据(,Business Data),结合起来，用以发现新的,Usage Pattern,可能的切入点,2026/1/27 周二,55,Web Usage Mining,在文2021中把把用户的,Tranversal,Pattern,和,purchasing Pattern,结合起来考虑，并构建了一个算法用以挖掘用户的,large transaction pattern(,使用规则来表示).,可能的切入点,2026/1/27 周二,56,Web Usage Mining,可能的切入点:提供完整的应用功能,1.,数据收集，清理和转换,2.数据存储,3.数据挖掘,4.个人定制,5.市场分析和决策,2026/1/27 周二,57,Web Usage Mining,可能的切入点:提供完整的应用功能,2026/1/27 周二,58,Web Usage Mining,Conclusion,1,Jaideep,Srivastava,Robert,Cooley,Mukund,Deshpande,Pang-Ning,Tan,Web Usage Mining:,Discovery and Applications of Usage Patterns from Web Data(2000).,SIGKDD Explorations,Vol.1,Issue 2,2000.,11Robert Cooley,Bamshad,Mobasher,and,Jaideep,Srivastava,Data Preparation for Mining World Wide Web Browsing Patterns(1999),Knowledge and Information Systems V1(1).,2026/1/27 周二,59,Web Usage Mining,1,Jaideep,Srivastava,Robert,Cooley,Mukund,Deshpande,Pang-Ning,Tan,Web Usage Mining:Discovery and Applications of Usage Patterns from Web Data(2000).SIGKDD Explorations,Vol.1,Issue 2,2000.,2Robert Cooley,Bamshad,Mobasher,and,Jaideep,Srivastava,Web Mining:Information and Pattern Discovery on the World Wide Web(A Survey Paper)(1997),in Proceedings of the 9th IEEE International Conference on Tools with Artificial Intelligence(ICTAI97),November 1997.,3WWW.W3C.ORG,W3C Working Draft WD-logfile-960323,4WWW.W3C.ORG,Web Characterization Terminology&Definitions Sheet,W3C Working Draft 24-May-1999,5E.Cohen,B.Krishnamurthy,and J.Rexford.Improving end-to-end performance of the web using server volumes and proxy filters.In Proc.ACM SIGCOMM,pages 241-253,1998.,6T.Fawcett and F.Provost.Activity monitoring:Noticing interesting changes in behavior.In Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pages 53-62,San Diego,CA,1999.ACM.,7Mike,Perkowitz,Oren,Etzioni,:Towards adaptive Web sites:Conceptual framework and case study.Artificial Intelligence 118(1-2):245-275(2000),8Mike,Perkowitz,Oren,Etzioni,:Adaptive Web Sites:an AI Challenge.IJCAI(1)1997:16-23,9 Alex Buchner and Maurice D,Mulvenna,.Discovering internet marketing intelligence through online analytical web usage mining.SIGMOD Record,27(4):54-61,1998.,10J.Ben Schafer,Joseph A.,Konstan,John,Riedi,Recommender,systems in e-commerce,Proceedings of the First ACM Conference on Electronic Commerce(EC-99),November 3-5,1999,Denver,CO,USA.ACM,1999 158-166,11Robert Cooley,Bamshad,Mobasher,and,Jaideep,Srivastava,Data Preparation for Mining World Wide Web Browsing Patterns(1999),Knowledge and Information Systems V1(1).,2026/1/27 周二,60,Web Usage Mining,12,J.Han,O.R.,Za?ane,and Y.Fu,Resource and Knowledge Discovery in Global Information Systems:A Scalable Multiple Layered Database Approach,Proc.of a Forum on Research and Technology Advances in Digital Libraries(ADL95),McLean,Virginia,May 1995.,13O.,Zaiane,and J.Han,WebML,:Querying the World-Wide Web for Resources and Knowledge,Proc.(CIKM98)Intl Workshop on Web Information and Data Management(WIDM98),Bethesda,Maryland,Nov.1998,pp.9-12.,14J.Pei,J.Han,B.,Mortazavi-Asl,and H.Zhu Mining Access Patterns Efficiently from Web Logs,Proc.2000 Pacific-Asia Conf.on Knowledge Discovery and Data Mining(PAKDD00),Kyoto,Japan,April 2000.,15O.R.,Zaiane,M.,Xin,J.Han,Discovering Web Access Patterns and Trends by Applying OLAP and Data Mining Technology on Web Logs,Proc.Advances in Digital Libraries Conf.(ADL98),Santa Barbara,CA,April 1998,pp.19-29.,16J.Ben Schafer,Joseph A.,Konstan,John,Riedi,Recommender,systems in e-commerce,Proceedings of the First ACM Conference on Electronic Commerce(EC-99),November 3-5,1999,Denver,CO,USA.ACM,1999 158-166,17Kun-lung Wu,Philip S Yu,and Allen,Ballman,.,Speedtracer,:A web usage mining and analysis tool.IBM Systems Journal,37(1),1998.,18,Bamshad,Mobasher,R,.Cooley,J.,Srivastava,Automatic,Personalization Based On Web Usage Mining,Communication of ACM,August,2000(Volume 43,Issue 8,19 Measuring the Accuracy of,Sessionizers,for Web Usage Analysis(with B.,Berent,M.,Spiliopoulou,and J.Wiltshire),in Proceedings of the Web Mining Workshop at the First SIAM International Conference on Data Mining,April 2001,Chicago,20C.-H,Yun,and M.-S.Chen,Mining Web Transaction Patterns in an Electronic Commerce Environment,Proc.of the 4th Pacific-Asia Conf.on Knowledge Discovery and Data Mining,pp.216-219,April 18-20,2000,21C.-H.,Yun,and M.-S.Chen,Using Pattern-Join and Purchase-Combination for Mining Web Transaction Patterns in an Electronic Commerce Environment,Proc.of the 24th annual Internl Computer Software and Application Conference(COMPSAC-2000),pp.99-104,October 25-27,2000.,22Soumen,Chakrabarti,Data,mining for hypertext:A tutorial survey.ACM SIGKDD Explorations,1(2),pages 1-1