Short-term load forecasting is typically used by electricity market participants to optimize their trading decisions and by system operators to ensure reliable grid operation. In particular, it allows the latter to foresee potential power imbalances and other critical grid states and thereafter, to enforce appropriate mitigation actions. Especially, forecasting critical grid states such as congestions, plays an essential role in this context. This paper proposes a recurrent neural network that is trained to forecast day-ahead time-series and prediction intervals for residual loads. Moreover, a comprehensive overview on probabilistic evaluation metrics is given. The ignorance score and the quantile score are used during the training whereas other metrics are for evaluation as they facilitate comparability between the different forecasting approaches with the naive baselines. The proposed deep learning model can be both specified as a parametric or as a non-parametric model and delivers reliable forecasts for day-ahead purposes.