You didn’t write what exactly happens between you when you are all together (what kind of aggression does the boy have, how does it manifest itself, who is he directed at, how do you and your spouse react to your son’s behavior, etc.). Therefore, it is difficult to name a specific reason.
When you, your spouse or the nanny with the son alone, for certain, practically all attention of the adult is concentrated on him. When you and your family gather together, you communicate not only with your son, but also with each other.
Therefore, it is possible that the child in the manner described by you simply wants to attract attention.
There may be other reasons. Perhaps the son does not know how to build communication with both of you in a different way, or is trying to figure out for himself whether his parents have the same requirements for him or he just needs contact bodily games.
And perhaps you and your spouse do not always show mutual agreement with your son (for example, you involuntarily compete with each other for the love of your son, find out the relationship between you, contradict yourself towards the child), and the boy, showing aggression towards one of the parents, thus signifies the other parent that he is on his side. In any case, it is important to understand what kind of information the child wants to convey to you, what goal he achieves with his behavior.
Of course, the parents are required to establish restrictions for the child. But the rules of conduct should not change depending on the circle with which the son spends time. It is very important that all members of the family enjoy their time together.
Do not forget to have fun with your child. More immediacy and creativity. Fool around, improvise.
The experience of such experiences will strengthen the relationship between you.