Portrait matting is an image processing technology that takes the portrait in the image as the foreground and accurately extracts it, and it is widely used in portrait photography and other fields. Given the problem that previous portrait matting methods often rely on prior information (such as trimaps) and the accuracy of predicted alpha matte is insufficient, we propose a portrait matting model based on an end-to-end manner, it can accurately predict the high-quality alpha matte by only feeding the original RGB image. This model is mainly composed of three sub-networks: the coarse segmentation network, which can obtain the coarse trimap by a deep network; the fine segmentation network, which can combine the prediction results of the coarse segmentation network and obtain the fine eleven-value image by a shallow network; the refined matting network, which is used to integrate the double-grained segmentation results of the first two networks and finally outputs a high-quality alpha matte. To solve the problem that the existing datasets of portrait matting are insufficient, we also create a large-scale portrait matting dataset including 40,870 training images and 3300 testing images based on several public datasets and apply data augmentation to this dataset. The experimental results show our model has achieved comparable results with state-of-the-art matting methods in both compositional images and real images. In addition, we also implement the ablation experiments to prove the effectiveness of each part of our model.