Image generation with spatial conditioning has been an active research topic. Its application to soccer images enables the creation of synthetic data for training, content generation, entertainment, and advertising. By incorporating additional control modalities beyond text, spatially conditioned image generation models allow fine-grained control over the positions of elements of interest in the generated images. However, when applied to soccer images, these models struggle to precisely control player jersey colors and fail to accurately position the soccer pitch within the image. In this work, we propose a framework called SoccerGen that addresses these limitations. It enables precise control over camera calibration, the positions of the ball and the players, and the colors of player jerseys. To the best of our knowledge, this work is the first to tackle the challenges of finely controlled soccer image generation.