That will just put newer user further under the impression that the star ratings mean anything which in turn will lead to more fights and more waves of random down voting all over in an attempt for vengeance.
That is one bloody future you're proposing.