I like the implication and the rules around the baby language in ASOUE
Across most of the series only Violet and Klaus can understand Sunny and often translate her babbles for others. Apart from Fernald who has a little sister Fiona. So based off of this I feel the implication is that Baby is just a language that only Babies speak and it seems Babies or at least Sunny understand English but are unable to respond in it.
I also feel there's a implication that you can't just learn the language you have to have been a older sibling. Maybe it's just you have to have taken care of a baby but I feel like Mr. Poe disproves that as I believe he was unable to understand Sunny as well despite having Children. Ofc there's a chance Mr. Poe is the expectation or that he hasn't had to translate it in a while and forgot how to since his children are 10ish I believe.