Crack detection is a critical task in monitoring and inspection of civil engineering structures. Image classification and bounding box approaches have been proposed in existing vision-based automated concrete crack detection methods using deep convolutional neural networks. The current study proposes a crack detection method based on deep fully convolutional network (FCN) for semantic segmentation on concrete crack images. Performance of three different pre-trained network architectures, which serves as the FCN encoder's backbone, is evaluated for image classification on a public concrete crack dataset of 40,000 227 × 227 pixel images. Subsequently, the whole encoder-decoder FCN network with the VGG16-based encoder is trained end-to-end on a subset of 500 annotated 227 × 227-pixel crack-labeled images for semantic segmentation. The FCN network achieves about 90% in average precision. Images extracted from a video of a cyclic loading test on a concrete specimen are used to validate the proposed method for concrete crack detection. It was found that cracks are reasonably detected and crack density is also accurately evaluated.