Image steganalysis aims to detect whether secret information is hidden in an image. This technique has critical applications in the field of information security. Most existing methods combine popular computer vision components for design without profoundly exploring the key factors applicable to image steganalysis. This letter reveals the limitations of existing feature passing and downsampling methods for image steganalysis tasks. We found that existing methods that pass shallow features through residual connections cannot cope with the problem of feature disappearance during network forward. In addition, the information-reducing downsampling methods used by these methods suppress the expression of steganographic features. To address these issues, we propose a feature enhancement passing module (FEPM) to help pass shallow features to deep layers and an attention downsampling module (ADM) to perform attention learning on full-resolution features. Combining these two structures, we designed an ultra-lightweight and high-precision image steganalysis network, FPNet, which contains only 0.16 M parameters. The results of several experiments in the same environment show that our method outperforms current state-of-the-art methods in several aspects, including detection accuracy and computational effort.