PB tracked in the parasite
If you think about it, it makes the most sense. Rick hadn't been impacted by the parasites at all when he walked in, but everyone else was already brainwashed.
Only Rick, Morty, and PB ever go off-world but anything that would impact Morty would probably impact Rick on basically the same schedule/cadence.
But here's the silver bullet, IMHO.
PB kicked off the first flashback after "Uncle Steve" was killed. The last remaining pieces of the parasitic infection were within PB, where they had originated. Once Rick locks everybody in, they all start getting affected by the parasite equally.