Segmentation of the prostate and organs at risk in male pelvic CT images using deep learning

Inter- and intra-observer variation in delineating regions of interest (ROIs) occurs because of differences in expertise level and preferences of the radiation oncologists. We evaluated the accuracy of a segmentation model using the U-Net structure to delineate the prostate, bladder, and rectum in male pelvic CT images. The dataset used for training and testing the model consisted of raw CT scan images of 85 prostate cancer patients. We designed a 2 DU-Net model to directly learn a mapping function that converts a 2D CT grayscale image to its corresponding 2D OAR segmented image. Our network contains blocks of convolution 2D layers with variable kernel sizes, channel number, and activation functions. On the left side of the U-Net model, we used three 3 x 3 convolutions, each followed by a rectified linear unit (ReLu) (activation function), and one max pooling operation. On the right side of the U-Net model, we used a 2 x 2 transposed convolution and two 3 x 3 convolution networks followed by a ReLu activation function. The automatic segmentation using the U-Net generated an average dice similarity coefficient (DC) and standard deviation (SD) of the following: DC+/-SD (0.88+/-0.12), (0.95+/-0.04), and (0.92+/-0.06) for the prostate, bladder, and rectum, respectively. Furthermore, the mean of average surface Hausdorff distance (ASHD) and SD were 1.2+/-0.9 mm, 1.08+/-0.8 mm, and 0.8+/-0.6mm for the prostate, bladder, and rectum, respectively. Our proposed method, which employs the U-Net structure, is highly accurate and reproducible for automated ROI segmentation. This provides a foundation to improve automatic delineation of the boundaries between the target and surrounding normal soft tissues on a standard radiation therapy planning CT scan.