The proliferation of smartphones has led to an increase in mobile health (mHealth) apps over the years. Thus, it is imperative to evaluate these apps by identifying shortcomings or barriers hampering effective delivery of intended services. In this paper, we evaluate 104 mental health apps on Google Play and App Store by performing sentiment analysis of 88125 user reviews using machine learning (ML), and then conducting thematic analysis on the reviews. We implement and compare the performance of five classifiers using supervised ML algorithms that are widely used for solving classification problems. The best performing classifier, with F1-score of 89.42%, was then used in predicting the sentiment polarity of reviews. Next, we conduct a thematic analysis of positive and negative reviews to identify themes representing various factors affecting the effectiveness of mental health apps positively and negatively. Our results uncover 21 negative themes and 29 positive themes. The negative themes fall under the following categories: usability issues, content issues, ethical issues, customer support issues, and billing issues. Some of the positive themes include aesthetically pleasing interface, app stability, customizability, high-quality content, content variation/diversity, personalized content, privacy and security, and low-subscription cost. Finally, we offer design recommendations on how the identified negative factors can be tackled to improve the effectiveness of mental health apps.