Automating orchard tasks, such as pruning tree branches, requires tree-structure understanding -- a significant challenge for computer vision. This paper introduces the first large-scale dataset for semantic and instance segmentation of modern fruit orchards. It consists of videos showing Cherry and Apple trees in modern-orchard scenes, and includes both labeled synthetic and real data, along with synthetic tree meshes. To address prohibitive costs of annotating numerous tree branches, we study unsupervised domain adaptation from synthetic to real data. For this setting, we propose a new Semantically-Guided Depth Refinement (SGDR) that leverages zero-shot depth estimation and semantic-aware smoothing. SGDR outperforms strong baselines and state of the art. Furthermore, we also benchmark the dataset in the supervised setting, where the initial annotations from the first frame are automatically propagated throughout the video using the foundation Segment Anything Model (SAM). The resulting pseudo labels are then manually corrected to generate the ground truth. For the supervised setting, we introduce SAM-Mask2Former (SAM-M2F) aimed at instance segmentation. By providing this dataset and benchmarking for both settings, we aim to enable new research for precision agriculture.