We present BVRI colors of 13 jovian and 8 saturnian irregular satellites obtained with the 2.56 m Nordic Optical Telescope on La Palma, the 6.5 m Magellan Baade Telescope on La Campanas, and the 6.5 m MMT on Mt. Hopkins. The observations were performed from December 2001 to March 2002. The colors of the irregular satellites vary from grey to light red. We have arbitrarily divided the known irregular satellites into two classes based on their colors. One, the grey color class, has similar colors to the C-type asteroids, and the other, the light red color class, has colors similar to P/D-type asteroids. We also find at least one object, the jovian irregular J XXIII Kalyke, that has colors similar to the red colored Centaurs/TNOs, although its classification is insecure. We find that there is a correlation between the physical properties and dynamical properties of the irregular satellites. Most of the dynamical clusters have homogeneous colors, which points to single homogeneous progenitors being cratered or fragmented as the source of each individual cluster. The heterogeneously colored clusters are most easily explained by assuming that there are several dynamical clusters in the area, rather than just one, or that the parent body was a differentiated, heterogeneous body. By analyzing simple cratering/fragmentation scenarios, we show that the heterogeneous colored S IX Phoebe cluster, is most likely two different clusters, a grey colored cluster centered on S IX Phoebe and a light red colored cluster centered on S/2000 S 1. To which of these two clusters the remaining saturnian irregulars with inclinations close to 174° belong is not clear from our analysis, but determination of their colors should help constrain this. We also show through analysis of possible fragmentation and dispersion of the six known uranian irregulars that they most likely make up two clusters, one centered on U XVI Caliban and another centered on U XVII Sycorax. We further show that, although the two objects have similar colors, a catastrophic fragmentation event creating one cluster containing both U XVI Caliban and U XVII Sycorax would have involved a progenitor with a diameter of ∼395 km. While such an event is not impossible it seems rather improbable, and we further show that such an event would leave 5–6 fragments with sizes comparable to or larger than U XVI Caliban. The stable region around Uranus has been extensively searched to limiting magnitudes far beyond that of U XVI Caliban. The fact that only U XVI Caliban, the larger U XVII Sycorax and four much smaller objects have been found leaves us with a distribution not compatible with a catastrophic event with such a large progenitor. The most likely solution is therefore two separate events creating two uranian dynamical clusters.