Over the last decade, advances in computing technology, image processing software and graphical processing power have extended the role of medical imaging far beyond traditional 2D visualization. The spatial information embedded in (cone-beam) CT and MRI scans is being increasingly used to personalize treatments by means of computer-assisted surgery. This field encompasses cutting-edge technologies such as virtual surgical planning, 3D printing of personalized constructs such as anatomical models or implants, virtual and augmented reality, and even robot-guided surgery. However, to date, several challenges are impeding the widespread use of these technologies in clinical settings. This lecture will discuss some of these challenges, particularly regarding image reconstruction and segmentation, and will give recommendations for future research.