Research and Teaching Page of William Webber

Hi! Up until July 2013, I was a postdoctoral research associate in the e-discovery lab of the iSchool at the University of Maryland, College Park. I was working on NSF IIS Program #1065250, Development and Evaluation of Search Technology for Discovery of Evidence in Civil Litigation, under the direction of Doug Oard and colleagues.

I completed by PhD thesis at the University of Melbourne in 2010, under the supervision of Alistair Moffat and Justin Zobel. You may like to look at my resume. My personal page may also be of interest to you.

I am now working as an private consultant in e-discovery, information retrieval, and text analysis.

My research interests include Information Retrieval for Civil Litigation (E-Discovery); Experimental and Evaluation Methodology for Information Retrieval; and Distributed Information Retrieval.

I keep a research blog on evaluation, information retrieval, computer science, and information technology.

PhD Thesis

Publications

• "Evaluating Non-Deterministic Retrieval Systems", Gaya K. Jayasinghe, William Webber, Mark Sanderson, Lasitha S. Dharmasena, and J. Shane Culpepper, Proc. ACM-SIGIR, Gold Coast, Australia, July 2014 (to appear) (author's version.)
• "Extending Test Collection Pools Without Manual Runs", Gaya K. Jayasinghe, William Webber, Mark Sanderson, and J. Shane Culpepper, Proc. ACM-SIGIR, Gold Coast, Australia, July 2014 (to appear) (author's version.)
• Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-Maximization, Ning Gao, William Webber, and Douglas W. Oard, Proc. ECIR, Amsterdam, Netherlands, April 2014, pages 1--12 (author's version.)
• Automation and Sampling in E-Discovery, William Webber and Jason R. Baron, Internet Law Bulletin, Volume 16, Number 7 (November, 2013), pages 158--161.
• Towards Minimizing the Annotation Cost of Certified Text Classification, Mossaab Bagdouri, William Webber, David D. Lewis, Douglas W. Oard, Proc. ACM-CIKM, San Franciso, CA, USA, October 2013, pages 989--998 (author's version).
• Information Retrieval for E-Discovery, Douglas W. Oard and William Webber, Foundations and Trends in Information Retrieval, Volume 7, Issues 2--3 (June, 2013), pages 99--237 (author's version).
• The Effect of Threshold Priming and Need for Cognition on Relevance Assessment, Falk Scholer, Diane Kelly, Hanseul S. Lee, Wan-ching Wu, and William Webber, Proc. ACM-SIGIR, Dublin, Ireland, July 2013, pages 623--632 (published version).
• Sequential Testing in Classifier Evaluation Yields Biased Estimates of Effectiveness, William Webber, Mossaab Bagdouri, David D. Lewis, and Douglas W. Oard, Proc. ACM-SIGIR, Dublin, Ireland, July 2013, pages 933--936 (published version).
• Document Features Predicting Assessor Disagreement, Praveen Chandar, William Webber, and Ben Carterette, Proc. ACM-SIGIR, Dublin, Ireland, July 2013, pages 745--748 (published version).
• Assessor Disagreement and Text Classifier Accuracy, William Webber and Jeremy Pickens, Proc. ACM-SIGIR, Dublin, Ireland, July 2013, pages 929--932 (published version).
• Approximate Recall Confidence Intervals, William Webber, ACM Transactions on Information Systems, Volume 31, Issue 1 (January, 2013), pages 2:1--33 (code and data) (published version).
• Alternative Assessor Disagreement and Retrieval Depth, William Webber, Praveen Chandar, and Ben Carterette, Proc. CIKM, Maui, Hawaii, USA, October 2012, pages 125--134.
• Effect of Written Instructions on Assessor Agreement, William Webber, Bryan Toth, and Marjorie Desamito, Proc. SIGIR, Portland, August 2012, pages 1053--1054.
• Principles for Robust Evaluation Infrastructure, Justin Zobel, William Webber, Mark Sanderson, and Alistair Moffat, Proc. CIKM Data Infrastructures for Supporting Information Retrieval Evaluation, Glasgow, UK, October 2011, 4 pages.
• Re-examining the Effectiveness of Manual Review, William Webber, Proc. SIGIR Information Retrieval for E-Discovery Workshop, Beijing, China, July 2011, 8 pages.
• A Similarity Measure for Indefinite Rankings, William Webber, Alistair Moffat, and Justin Zobel, ACM Transactions on Information Systems, Volume 28, Issue 4 (November, 2010), pages 20:1--38 (implementation in C).
• Assessor Error in Stratified Evaluation, William Webber, Douglas W. Oard, Falk Scholer, and Bruce Hedin, Proc. ACM-CIKM, Toronto, October 2010, page 539--548.
• The Effect of Pooling and Evaluation Depth on Metric Stability, William Webber, Alistair Moffat, and Justin Zobel, Proc. EVIA, Tokyo, June 2010, pages 7--15.
• Improvements That Don't Add Up: Ad-Hoc Retrieval Results Since 1998, Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel, Proc. ACM-CIKM, Hong Kong, November 2009, pages 601--610.
• Score Adjustment for Correction of Pooling Bias, William Webber and Laurence Park, Proc. ACM-SIGIR, Boston, July 2009, pages 444--451.
• Relative Significance is Insufficient: Baselines Matter Too, Timothy G. Armstrong, Justin Zobel, William Webber, and Alistair Moffat, Proc. ACM-SIGIR Workshop on The Future of IR Evaluation, Boston, July 2009, pages 25--26.
• Has Adhoc Retrieval Improved Since 1994?, Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. Proc. ACM-SIGIR, Boston, July 2009, pages 692--693.
• EvaluatIR: An Online Tool for Evaluating and Comparing IR Systems, Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel, Proc. ACM-SIGIR, Boston, July 2009, page 834 (demo), EvaluatIR.org website).
• Statistical Power in Retrieval Experimentation, W. Webber, A. Moffat, and J. Zobel, Proc. ACM-CIKM, Napa, USA, October 2008, pages 571--580.
• Score Standardization for Inter-Collection Comparison of Retrieval Systems, W. Webber, A. Moffat, and J. Zobel, Proc. ACM-SIGIR, Singapore, July 2008, pages 51--58.
• Precision-At-Ten Considered Redundant, W. Webber, A. Moffat, J. Zobel, and T. Sakai, Proc. ACM-SIGIR, Singapore, July 2008, pages 695-696.
• Score Standardization for Robust Comparison of Retrieval Systems, W. Webber, A. Moffat, J. Zobel, Proc. 12th Australasian Document Computing Symposium, Melbourne, December 2007, pages 1-8. Winner, Best Student Paper Prize. (Please see addendum concerning resampling method used in Figure 5.6.)
• Strategic System Comparisons via Targeted Relevance Judgments, A. Moffat, W. Webber, J. Zobel, Proceedings of the International ACM-SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, July 2007.
• Design and Evaluation of a Pipelined Distributed Information Retrieval Architecture, W. Webber, Thesis, Master of Engineering, University of Melbourne, June 2007.
• A pipelined architecture for distributed text query evaluation, A. Moffat, W. Webber, J. Zobel, R. Baeza-Yates, Information Retrieval, Springer Netherlands, October 2006.
• Load balancing for term-distributed parallel retrieval, A. Moffat, W. Webber, and J. Zobel, Proceedings of the International ACM-SIGIR Conference on Research and Development in Information Retrieval, Washington, August 2006.
• Space-limited ranked query evaluation using adaptive pruning, N. Lester, A. Moffat, W. Webber, and J. Zobel, Proceedings of the WISE Workshop on Information Systems Engineering, M. Kitsuregawa, E. Neuhold, and A.H.H. Ngu (eds), New York, November 2005, pp. 470-477.
• In Search of Reliable Retrieval Experiments, W. Webber and A. Moffat, Proc. 10th Australasian Document Computing Symposium, Sydney, December 2005, pages 26-33.
• "Melbourne University 2005: Enterprise and Terabyte Tracks", Anh, Webber, Moffat. Proc. 2005 TREC Notebook Proceedings, Gaithersburg, MD, November 2005.
• "RMIT University at TREC 2005: Terabyte and Robust Track", Y. Bernstein, B. Billerbeck, S. Garcia, N. Lester, F. Scholer, J. Zobel, W. Webber.
• "RMIT University at TREC 2004", B. Billerbeck, A. Cannane, A. Chattaraj, N. Lester, W. Webber, H.E. Williams, J. Yiannis, J. Zobel.

Other writings

• A tutorial on interval estimation for a proportion, with particular reference to e-discovery (v 0.1, August 2nd, 2012).

Professional activities

Editorships

• Information Retrieval (member of editorial board, 2011--present)

Invited talks

• Principles for Robust Evaluation Infrastructure, NTCIR-9, December 2011 (with Mark Sanderson).

Conference chairs and committees

• EVIA Workshop 2010 (co-chair), 2011 (co-chair)
• ACM CIKM Conference, 2009 (Program Committee), 2010 (Program Committee), 2011 (Program Committee)
• ACM SIGIR Conference, 2009 (Program Committee), 2010 (Program Committee), 2011 (Program Committee)
• Australian Document Computing Symposium, 2007 (Program Committee), 2009 (Program Committee)
• String Processing and Information Retrieval Symposium, 2008 (Organizing Committee)
• Asian Information Retrieval Symposium, 2009 (Publicity Chair, Australia), 2010 (Area Co-Chair: Evaluation)
• International Joint Conference on Artificial Intelligence, 2011 (senior program committee)

Reviewing:

• ACM Transactions on Information Systems
• Journal of the American Society for Information Science and Technology
• Information Retrieval (journal)
• The Computer Journal
• Australian Document Computing Symposium, 2005, 2007, 2008 (shared Best Reviewer award), and 2009
• ACM Conference on Information and Knowledge Management, 2006, 2007, 2008, and 2009
• Symposium on Combinatorial Pattern Matching, 2008
• European Conference on Information Retrieval, 2008
• ACM SIGIR Conference, 2008 and 2009
• Conference on Very Large Databases, 2007 and 2008
• ACM International Conference on Web Search and Data Mining, 2008
• IEEE Transactions on Knowledge and Data Engineering, 2011

In First Semester, 2014, I taught COMP90042, Web Search and Text Analysis at the University of Melbourne.

In Spring 2012, I taught LBSC 690, Information Technology at the University of Maryland, College Park.

In 2009, I co-taught 433467, Text and Document Management, in first semester, at the University of Melbourne.

• An R package, recci , for calculating confidence intervals on recall and F1 score, as described (for recall) in my paper, Approximate Recall Confidence Intervals.
• An implementation of the rank-biased overlap (RBO) indefinite rank similarity measure.

William Webber
09/Dec/2014