310 hours in this game and I only just figured out
To shake off a hermit if you accidentally pick them up. I just waited until night and crept outside to force them to jump off. But I’ve discovered if you mash left/right/left/right you throw them off. I feel like an idiot!