Steady-State Visual Evoked Potential (SSVEP) is one of the popular methods of brain-computer interfacing (BCI). It is used to translate the Electroencephalogram (EEG) signals into actions or choices. The main challenge in processing the SSVEP signal recognition is finding an appropriate intermediate representation to facilitate the classification task afterwards. In the literature, frequency domain analysis was extensively adopted as an intermediate representation for SSVEP classification. In this presented paper, we propose a deep learning model that uses a hybrid architecture based on Convolutional and Recurrent Neural Networks to classify SSVEP signals in the time domain directly. We achieved accuracy 93.59% compared to 87.40% for the state-of-the-art method: canonical correlation analysis in the frequency domain. The proposed architecture facilitates the real-time classification of SSVEP signals in the time domain for real-time applications such as robot cars and exoskeletons.