Establishing structured reconstruction models and efficient reconstruction algorithms according to practical engineering needs is of great concern in the applied research of Compressed Sensing (CS) theory. Targeting problems during high-speed video capture, the paper proposes a set of video CS scheme based on intra-frame and inter-frame constraints and Genetic Algorithm (GA). Firstly, it employs the intra-frame and inter-frame correlation of the video signals as the priori information, creating a video CS reconstruction model on the basis of temporal and spatial similarity constraints. Then it utilizes overcomplete dictionary of Ridgelet to divide the video frames into three structures, smooth, single-oriented, or multijointed. Video frames cluster according to the structure using Affinity Propagation (AP) algorithm, and finally clusters are reconstructed using evolutionary algorithm. It is proved efficient in terms of reconstruction result in the experiment.