I look at it from the other side. I think it's morally wrong for people to act in ways that are good for themselves while harming society as a whole.

And who is going to decide which actions are harmful to society? Some are obvious. It might be good for me to rob a bank so that I have more money to do what I want. However, this is very harmful to all the people who have saved money all their lives so that they can afford to support themselves in their old age.

What about the government robbing me of the money I have worked to save all my life so that I can affors to retire sone day? That is good for society (at least the part that has not worked hard and has not saved for their old age), but not so good for me. I consider that to be immoral.