I’ve both spoken those words and had them spoken to me. Sometimes (rarely, I’m sure), it is because I might have been behaving in a slightly juvenile manner. One of my friends used to say, “Growing old is mandatory; growing up is optional.”
I’m pretty sure that, “Grow up!” whether aimed at me or from me, says more about the speaker than it does about the target. It likely means that the speaker is irritated, inconvenienced, or maybe embarrassed by the actions of another. And it rarely results in growth.