This dissertation makes several contributions to the training of reinforcement learning agents directly on real robots. It introduces a reliable software suite and a new exploration strategy to replace the standard step-based one. The thesis also explores additional types of expert knowledge to guide RL, focusing on an elastic neck and quadruped locomotion. The presented approaches are extensively validated through experiments on four different robots.