Coherent-mode representation provides physical insight and computational simplification into the analysis of random optical signals. In this work, we present the coherent-mode decomposition for pulsed electromagnetic beam fields. We show that the mode decomposition can be done for any valid space-frequency or space-time coherence matrix representing nonstationary pulsed electric field, and moreover, the spectral and temporal modes are connected via a Fourier transform relation. The analysis also yields the coherent modes of electromagnetic time-domain signals in temporal optics. We present the overall degree of coherence as a measure of the average temporal or spectral and spatial coherence of the beam. Several illustrative examples are discussed analytically and numerically.