Im more specifically referring to the Eastern Cape
Differs from area to area as said.
In the eastern cape for example:
Tsitsikamma
SE brings in cold, nutrient rich water. In other words, that proper Pro-Nutro brown water.
S and SW flattens the swell and the water turns a milky colour with lots of dead plant particles - low oxygen. This only happens if it blows for a couple of days. If it just turns south, it is still fine.
In Cape St Francis on the other hand, I don't mind a S wind blowing. Churns up the water. St Francis Bay (only 10km away) goes dead quiet and icy cold if a southerly wind blows for a couple of days.
Generally, it depends on the direction the coast is facing.