Yang Sun, Ziming Zhuang, C. Lee Giles. A large-scale study of robots.txt. In Carey L. Williamson, Mary Ellen Zurko, Peter F. Patel-Schneider, Prashant J. Shenoy, editors, Proceedings of the 16th International Conference on World Wide Web, WWW 2007, Banff, Alberta, Canada, May 8-12, 2007. pages 1123-1124, ACM, 2007. [doi]
Abstract is missing.