There would have had to be gossip amongst the servants, the family and even the village about Edith and farmers. She did have a short affair with Mr. Drake, and I would think most people people would have thought she was having an affair with Mr. Drew, just seems like someone would have mentioned that pattern.
I basically have it on an endless loop in the evenings, but every time I have to take a break after season 3. I usually watch season 1 of Ted Lasso, or seasons 1-2 of Victoria, then come back for season 4, I feel like season 3 is so intense, I need a break!
On my millionth rewatch, season 1 episode 2 Robert says Patrick was "in and out of this house since the day he was born", makes me wonder where they lived, did they live in Crawley House, where Matthew and Isobel moved too? Or did they live somewhere else?