Evaluating and comparing multi-objective optimizers is an important issue. But, when doing a comparison, it has to be noted that the results can be influenced highly by the selected Quality Indicator. Therefore, the impact of individual Quality Indicators on the ranking of Multi-objective Optimizers in the proposed method must be analyzed beforehand. In this paper the comparison of several different Quality Indicators with a method called Chess Rating System for Evolutionary Algorithms (CRS4EAs) was conducted in order to get a better insight on their characteristics and how they affect the ranking of Multi-objective Evolutionary Algorithms (MOEAs). Although it is expected that Quality Indicators with the same optimization goals would yield a similar ranking of MOEAs, it has been shown that results can be contradictory and significantly different. Consequently, revealing that claims about the superiority of one MOEA over another can be misleading.