Embodiments of the present invention provide systems, methods, and computer storage media for 3D segmentation. A 3D segmentation network can perform a voxel-wise classification of a 3D volume such as a brain tumor. The 3D segmentation network can accept a plurality of 3D representations of the 3D volume (e.g., MRI modalities) into corresponding 3D input channels. Generally, 3D convolutions can be applied by convolutional layers of the 3D segmentation network. Convolutional blocks can generate successive resolutions of feature maps from the plurality of 3D representations by downsampling outputs from prior convolutional blocks. The feature maps can be upsampled and combined to aggregate local and global detail from multiple resolutions. A multi-class classifier can be applied to each voxel at the output layer to generate a voxel-wise prediction map with the same spatial size as the inputs.
3D Refinement Module for Combining 3D Feature Maps