With the development of deep learning, remote sensing image (RSI) semantic segmentation has produced significant advances. The majority of existing methods use fully convolutional network (FCN) that lacks fine-grained multiscale representation and fails to extract global context information. Thus, we improve FCN by adding two modules—multiscale attention (MSA) and nonlocal filter (NLF). The MSA module enhances the network’s fine-grained multiscale representation capability and allows modeling the interdependencies of feature maps among different channels. The NLF module can capture global context information by sequential using fast Fourier transform (FFT), parameter learnable filters, and inverse FFT. By using MSA module for encoder and NLF module for decoder in the FCN framework, MSA and NLF network (MsanlfNet) can obtain both fine-grained multiscale spatial feature and global context information, thus achieving a balance between performance and computational effort. Experimental results on the remote sensing semantic segmentation public datasets demonstrate that our method can achieve better performance. The code is available at https://github.com/xyuanLin/MsanlfNet .