| 5 | -- | 6 | Thomas G. Dietterich. Editorial |
| 7 | -- | 9 | Leslie Pack Kaelbling. Introduction |
| 11 | -- | 32 | David E. Moriarty, Risto Miikkulainen. Efficient Reinforcement Learning through Symbiotic Evolution |
| 33 | -- | 57 | Steven J. Bradtke, Andrew G. Barto. Linear Least-Squares Algorithms for Temporal Difference Learning |
| 59 | -- | 94 | John N. Tsitsiklis, Benjamin Van Roy. Feature-Based Methods for Large Scale Dynamic Programming |
| 95 | -- | 121 | Robert E. Schapire, Manfred K. Warmuth. On the Worst-Case Analysis of Temporal-Difference Learning Algorithms |
| 123 | -- | 158 | Satinder P. Singh, Richard S. Sutton. Reinforcement Learning with Replacing Eligibility Traces |
| 159 | -- | 195 | Sridhar Mahadevan. Average Reward Reinforcement Learning: Foundations, Algorithms, and Empirical Results |
| 197 | -- | 225 | Matthias Heger. The Loss from Imperfect Value Functions in Expectation-Based and Minimax-Based Tasks |
| 227 | -- | 250 | Sven Koenig, Reid G. Simmons. The Effect of Representation and Knowledge on Goal-Directed Exploration with Reinforcement-Learning Algorithms |
| 251 | -- | 281 | Richard Maclin, Jude W. Shavlik. Creating Advice-Taking Reinforcement Learners |
| 283 | -- | 290 | Jing Peng, Ronald J. Williams. Incremental Multi-Step Q-Learning |