In this paper we present a novel simulation environment called SwarmSimX with the ability to simulate dozens of robots in a realistic 3D environment. The software architecture of SwarmSimX allows new robots, sensors, and other libraries to be loaded at runtime, extending the functionality of the simulation environment significantly. In addition, SwarmSimX allows an easy exchange of the underlying libraries used for the visual and physical simulation to incorporate different libraries (e.g., improved or future versions). A major feature is also the possibility to perform the whole simulation in real-time allowing for human-in-the-loop or hardware-in-the-loop scenarios. SwarmSimX has been already employed in several works presenting haptic shared control of multiple mobile robots (e.g., quadrotor UAVs). Additionally, we present here two validation tests showing the physical delity and the real-time performance of SwarmSimX. For the tests we used NVIDIA PhysX and Ogre3D as physics and rendering libraries, respectively