Ther’s nothing wrong with colorful clothing. It was good enough for Yaakov Avinu to give his son. His only “mistake” was in showing favoritism to ONE child, not in giving him a colorful cloak. Had it been a white cloak made especially for Yosef, it would still have made his brothers jealous.
As long as a shirt and pants are neat and clean, people should be less concerned with their color than with the middos and integrity of the person wearing them.