EE 380L: Advanced Topics in DATA MINING (Unique 17060) Fall 2011.

Class Schedule and list of Papers

Link to topic assignments, presentation dates, and guest speaker schedules: Google Doc

Link to project presentation dates: Google Doc

Link to presentation videos (subject to removal): Google Doc

1.         Introduction and course overview (1 lecture)

2.         Graphical models and approximate inference (2 lectures)
Read/re-visit Bishop, chapters 8-11.

A.        Variational inference

Tutorial: M.I. Jordan, Z. Ghahramani, T.S. Jaakkola, L.K. Saul. "An Introduction to Variational Methods for Graphical Models." Machine Learning 37(2), 1999: pp 183-233. [PDF]

Video lecture: http://videolectures.net/mlss04_bishop_gmvm/

B.        Sampling based approaches

Tutorial: C. Andrieu, N. de Freitas, A. Doucet, M.I. Jordan. "An Introduction to MCMC for Machine Learning." Machine Learning 50(1), 2003: pp 5-43. [PDF]

     Video lecture: http://videolectures.net/mlss09uk_murray_mcmc/
     Radford Neal’s book
http://www.cs.utoronto.ca/~radford/ftp/review.pdf

Software: Infer.NET is a framework for running Bayesian inference in graphical models.



Topic 3 onwards are covered through student led presentations. * denotes a lead paper.

3.         Topic models (3 lectures)

Bibliography and pointers to code at http://www.cs.princeton.edu/~mimno/topics.html

David Blei's resource page : http://www.cs.princeton.edu/~blei/topicmodeling.html

David Blei's KDD 2011 tutorial : [PDF]

Data sets:

               http://www.cs.umass.edu/~mccallum/data.html

               http://cs.nyu.edu/~roweis/data.html

               http://www.zjucadcg.cn/dengcai/Data/TextData.html

A.        Survey/overview: LDA and extensions

Slides: http://www.cs.umd.edu/class/spring2008/cmsc828g/Slides/topic-models.pdf

Video lecture: http://videolectures.net/mlss09uk_blei_tm/

D.M. Blei. "Introduction to Probabilistic Topic Models." Communications of the ACM (to appear 2011). [PDF]

* D.M. Blei, A. Ng, M. Jordan. "Latent Dirichlet allocation." Journal of Machine Learning Research 3, 2003: pp 993-1022. [PDF]

* D.M. Blei, J.D. Lafferty. " A Correlated Topic Model of Science." Annals of Applied Probability, 2007. [PDF]. C Code at www.cs.princeton.edu/~blei/ctm-c/

A. Agovic, A. Banerjee. "Gaussian Process Topic Models." UAI 2010. [PDF]

B.        Hierarchical approaches

* D.M. Blei, T.L. Griffiths, M.I. Jordan. "The Nested Chinese Restaurant Process and Bayesian Nonparametric Inference of Topic Hierarchies." Journal of the ACM 57(2), 2010: 1-30. [PDF]

John Paisley, Chong Wang, David Blei. "The Discrete Infinite Logistic Normal Distribution for Mixed-Membership Modeling." AISTAT 2011. [PDF]

Y. Teh, M. Jordan, M. Beal, and D. Blei, Hierarchical Dirichlet Processes.  Journal of American Statistical Association, 2006. [PDF].
Tutorial by Teh at
mlg.eng.cam.ac.uk/tutorials/07/ywt.pdf

C.        Evaluation, inferencing, and online/parallel

* H.M. Wallach, I. Murray, R. Salakhutdinov, D. Mimno. "Evaluation Methods for Topic Models." ICML 2009. [PDF]

* A. Asuncion, M. Welling, P. Smyth, Y-W. Teh. "On Smoothing and Inference for Topic Models." UAI 2009. [PDF]

M. Hoffman, D.M. Blei, F. Bach. "Online Learning for Latent Dirichlet Allocation." NIPS 2010. [PDF]

* D. Newman, A. Asuncion, P. Smyth, M. Welling. "Distributed Algorithms for Topic Models." Journal of Machine Learning Research 10, 2009: pp 1801-1828. [PDF]

4.         CRFs and information extraction (1 lecture, guest speaker)

* C. Sutton, A. McCallum. "An Introduction to Conditional Random Fields." http://arxiv.org/abs/1011.4088 [PDF]
CRF Software: http://www.inference.phy.cam.ac.uk/hmw26/crf/#software

CRF resources http://iic.arizona.edu/search/by_tag?tag=crf

 

J. Zhu, N. Lao, N. Chen, E.P. Xing. "Conditional Topical Coding: An Effcient Topic Model Conditioned on Rich Features." KDD 2011. [PDF]

 

5.         Mining Multi-relational Data for Affinity Estimation (6 lectures)

 

Data sets:

               HetRec Datasets

              Several Affinity Datasets
            $1 Million Prize to Speed Innovation in Retail Personalization (Overstock) http://overstockreclabprize.com/

ECML/PKDD 2011 Discovery Challenge: VideoLectures.Net recommender system.

 

A.        Matrix factorization (probabilistic or otherwise)

Tutorial for PMF and RBMs for CF: http://www.cs.toronto.edu/~hinton/csc2515/notes/pmf_tutorial.pdf

* R. Salakhutdinov, A. Mnih. "Probabilistic Matrix Factorization." NIPS 2008. [PDF]

R. Salakhutdinov, A. Mnih. "Bayesian Probabilistic Matrix Factorization using MCMC." ICML 2008. [PDF] [Code: http://www.mit.edu/~rsalakhu/BPMF.html]

*R. Gemulla, P.J. Haas, E. Nijkamp, Y. Sismanis. "Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent." KDD 2011 [PDF]

*A.P. Singh, G.J. Gordon. "A Unified View of Matrix Factorization Models." ECML/PKDD 2008. [PDF]

G. Takács, I. Pilászy, B. Németh, D. Tikk. "Scalable collaborative filtering approaches for large recommender systems." Journal of Machine Learning Research 10, 2009: pp 623-656. [PDF]

B.        Block models

* E.M. Airoldi, D.M. Blei, S.E. Fienberg, E.P. Xing. "Mixed Membership Stochastic Blockmodels." Journal of Machine Learning Research 9, 2008: pp 1981-2014. [PDF]

I. Sutskever, R. Salakhutdinov, and J. Tenenbaum. "Modelling Relational Data using Bayesian Clustered Tensor Factorization." NIPS 2010. [PDF]

* L. Mackey, D. Weiss, M.I. Jordan. "Mixed Membership Matrix Factorization." ICML 2010 [PDF slides] [MATLAB code at http://code.google.com/p/m3f/]

C.         Multi-relational data analysis adding "side information"

* H. Shan, A. Banerjee. "Generalized Probabilistic Matrix Factorizations for Collaborative Filtering." ICDM 2010. [PDF]

* R.P. Adams, G.E. Dahl, I. Murray. "Incorporating side information into probabilistic matrix factorization using Gaussian Processes." UAI 2010. [PDF]

*A.K. Menon, K-P Chitrapura, S. Garg, D. Agarwal, N. Kota. "Response Prediction Using Collaborative Filtering with Hierarchies and Side-information." KDD 2011 [PDF]

I. Porteous, A. Asuncion, M. Welling. "Bayesian Matrix Factorization with Side Information and Dirichlet Process Mixtures." AAAI 2010. [PDF]

M.S. Handcock, A.E. Raftery, J.M. Tantrum. "Model-based clustering for social networks." J. of the Royal Statistical Society: Series A (Statistics in Society) 170, 2007: pp 301-354. [PDF]

D.        Multi-level Bayesian modeling

*D. Agarwal, B-C Chen, B. Long. "Localized Factor Models for Multi-Context Recommendation." KDD 2011 [PDF]

* D. Agarwal, B-C Chen. "Regression-based latent factor models." KDD 2009. [PDF]

P. Hoff. "Hierarchical multilinear models for multiway data." 2009. [PDF]

E.         Supervised learning in multi-relational data; extracting structure/meaning

* D. Agarwal, B-C Chen. "fLDA: Matrix Factorization through Latent Dirichlet Allocation." WSDM 2010. [PDF]

* D.M. Roy, C. Kemp, V.K. Mansinghka, J.B. Tenenbaum. "Learning annotated hierarchies from relational data." NIPS 2006. [PDF]

I. Porteous, E. Bart, M. Welling. "Multi-LDA/HDP A Non Parametric Bayesian Model for Tensor Factorization." AAAI 2008 [PDF]

A.K. Menon, C. Elkan. "A log-linear model with latent features for dyadic prediction." ICDM 2010. [PDF] [Slides] [Code]

F.         Dynamics

*D. Stern, R. Herbrich, and T. Graepel. "Matchbox: Large Scale Bayesian Recommendations." WWW 2009. [PDF]

*Q. Ho, L. Song, E. Xing. "Evolving Cluster Mixed-Membership Blockmodel for Time-Evolving Networks." AISTAT 2011. [PDF]

L Tang, H Liu, J Zhang, H. Liu. "Community evolution in dynamic multi-mode networks." KDD 2008. [PDF]

L. Xiong, X. Chen, T. Huang, J. Schneider, J. Carbonell, “Temporal Collaborative Filtering with Bayesian Probabilistic Tensor Factorization”, SIAM Data Mining (SDM), 2010. [PDF]

6.         Transfer and multi-task learning (3 lectures)

Data sets and resources:

 Transfer Learning Resources: Code, Data, Surveys [Homepage]

 Unsupervised and Transfer Learning Challenge (http://www.causality.inf.ethz.ch/unsupervised-learning.php#cont)

A.        Overview

*S.J. Pan, Q. Yang. "A Survey on Transfer Learning." IEEE Transactions on Knowledge and Data Engineering. 22(10), 2010: pp 1345-1359. [PDF].   Also see slides by S.J. Pan. [PPT]

E.H. Zhong, W. Fan, Q. Yang, O. Verscheure, J. Ren. "Cross Validation Framework to Choose amongst Models and Datasets for Transfer Learning." ECML/PKDD 2010. [PDF]

B.        Transfer Learning Challenge

* "Unsupervised and Transfer Learning Challenge." (http://www.causality.inf.ethz.ch/unsupervised-learning.php#cont)

I. Guyon, G. Dror, V. Lemaire, G. Taylor, D.W. Aha. "Unsupervised and Transfer Learning Challenge." IJCNN 2011 [PDF]

C.         Multitask learning

B. Cao, N. Liu, Q. Yang. "Transfer Learning for Collective Link Prediction in Multiple Heterogeneous Domains." ICML 2010 [PDF]

*Laurent Jacob, Francis Bach, Jean-Philippe Vert "Clustered Multi-Task Learning: a Convex Formulation." NIPS 2008 [PDF]

7.         Distributed and Privacy Preserving Data Mining (2 lectures)

A.        Privacy preserving learning

*S. Merugu, J. Ghosh. "Privacy-perserving distributed clustering using generative models." ICDM 2003. [PDF] (also see paper at KDD 2005)

*M. Kearns, J. Tan, J. Wortman. "Privacy-Preserving Belief Propagation and Sampling." NIPS 2007. [PDF]

B.        Measures and limitations

*S. Chawla, C. Dwork, F. McSherry, A. Smith, H. Wee. "Toward Privacy in Public Databases." Lecture Notes in Computer Science, 2005: pp 363-385. [PDF] [Extended PDF]

C. Dwork. "Differential Privacy: A Survey of Results." Lecture Notes in Computer Science, 2008: pp 1-19. [PDF]

* E. Zheleva, L. Getoor. "To join or not to join: the illusion of privacy in social networks with mixed public and private user profiles." WWW 2009. [PDF] [Slides]

N. Mohammed, R. Chen, B.C.M. Fung, P.S. Yu. "Differentially Private Data Release for Data Mining." KDD 2011 [PDF]

8.         Scaling to large data sets and parallel/cloud computing (2 lectures)

A.        ADDM

* S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein. "Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers." To appear in Foundations and Trends in Machine Learning. [Webpage] [PDF] [Slides]

MATLAB scripts: http://www.stanford.edu/~boyd/papers/admm/

B.        Articles

Choose chapters from R. Bekkerman, M. Bilenko, and J. Langford (Eds) "Scaling Up Machine Learning", Cambridge University Press, 2011. [http://www.cs.umass.edu/~ronb/scaling_up_machine_learning.htm]

Recent workshop: http://lccc.eecs.berkeley.edu/

9.         Additional resources

General:

A Halevy, P. Norvig, F. Periera, "The Unreasonable Effectiveness of Data." IEEE Computer 24, 2009: pp 8-12. [Abstract]

Topic Models:

C. Wang, D.M. Blei. Lafferty. "Collaborative Topic Modeling for Recommending Scientific Articles." KDD 2011 [PDF]

D.M. Blei, J.D. Lafferty. "Dynamic Topic Models." ICML 2006. [PDF]

G. Doyle, C. Elkan. "Accounting for Burstiness in Topic Models." ICML 2009. [PDF]

Volker Tresp. "Multivariate Models for Relational Learning." Tutorial, ILP 2010. [PDF]

Multirelational and Network Models:

A. Goldenberg, A.X. Zheng, S.E. Fienberg, E.M. Airoldi. "A Survey of Statistical Network Models." [PDF]

X. Su, T.M. Khoshgoftaar. "A survey of collaborative filtering techniques." Advances in AI 2009, 2009: pp 1-19. [Abstract] [PDF]

E. Zheleva, G. Namata. "Stochastic Blockmodels: A Survey." [PDF slides]

J. Tenenbaum. "How to Grow a Mind: Statistics, Structure and Abstraction". NIPS 2010 Posner Lecture. [Video]

J.B. Tenenbaum, C. Kemp, T.L. Griffiths, N.D. Goodman. "How to Grow a Mind: Statistics, Structure, and Abstraction." Science 331, 2001: pp 1279-1285. [PDF] [Supplementary material]

Kai Yu's Homepage: http://www.dbs.informatik.uni-muenchen.de/~yu_k/

Other Challenges

Active Learning Challenge [Homepage]

I. Guyon, G. Cawley, G. Dror, V.Lemaire. "Results of the Active Learning Challenge." JMLR 2011. [PDF]

Heritage Health Prize Challenge [Homepage]


Check out http://www.kaggle.com/