The tragedy is our minds are being switched to think that fashion is right all the time. Even if we are careless and think fashion is nonsense and you like to be in your comfortable clothes, still you are in trouble everytime picking up clothes. This is because you don't know your personality and thereby you can't carry it well.
First of all know what you wear, is it worth or is it good for your body?
Do you often go to discounted section and make up your mind on some item just because you have only those options available?
Stop doing this and explore yourself, your character and your dignity.
You might get negative judgments while on this path but it's worth because in the end it will not effect you at all.