WebSim is a (freeware) simulator for learning algorithms, function approximators, and gradient descent methods. It is implemented as a set of Java classes by Leemon Baird, Mance Harmon, Scott Weaver, and Ansgar Laubsch.
The RL-Framework is a project to build a standard software protocol for benchmarking and interconnecting reinforcement learning agents and environments at the Department of Computing Science, University of Alberta.
Many of the popular testbeds in RL can be designed using one of the standard simulation packages. The ones listed below are widely used in the engineering literature to test scenarios (policies).