r/trustandsafetypros

I'm currently researching probabilistic moderation agents for catching fake e-commerce reviews for a project.

One challenge I'm trying to model is how bad actors adapt when they realize how automated detection works. In my setup, the agent uses fixed probability thresholds and likelihood ratios for signals like account age, review frequency, and text similarity. If a review drops below a 20% probability of being genuine, it gets flagged for high-priority human ban review.

The issue is that bad actors quickly figure out where these boundaries are. Once they realize that posting 5 reviews a day triggers a high-frequency signal, they throttle back to 1 review every few days. Or they buy cheap digital items just to get a verified purchase badge and artificially boost their Level 1 score.

For anyone who has worked on content moderation or fraud systems, How do you handle threshold gaming and signal decay in practice? Do you dynamically adjust your likelihood estimates as fraud patterns shift, or do you rely on periodic manual recalibration of your base rates?

Working in T&S on the Ops side seems to be constant fight. Fight with Product for prioritization, fight with business for feature enablement, while trying to rally all other stakeholders. How do you handle this? Or is it just my org?

I am modeling a 4-action utility-based agent for product review moderation with the following action space: permit, warn, hide, and report.

In typical Trust & Safety architectures:

1. How is the "warn" action usually operationalized for user-submitted content? Does it mean appending a public uncertainty tag to the post, or routing it to a human review queue?
2. What criteria or probability threshold triggers an immediate hard action (hide/report) versus an escalation to manual review?

Hi! I’ve been in trust for several years mostly in threat investigations and scaled abuse roles. I’m considering a trust related PM role. Has anyone transitioned from analyst/investigator to Product Manager role? I’d love to hear your experience about the transition.

Does anyone work in Google for the above role. I want to understand what level of SQL and Python is required for this role so that I can prepare for. Thank you in advance!

Do any roles like this exist? Anyone care to add an anecdote from personal experience?

Was a security engineer investigator at Meta and while the actual work itself was very interesting and seemingly impactful, it was an extremely fast paced, high stress, performance culture driven place. Found that it was simply not sustainable for me, then the May layoffs hit and decimated the org.

Considering a pivot back to tech down the road if roles like this exist - do they? If so, what companies?

TIA

Edit: not a true cyber guy so threat hunting and a lot of CTI roles are a bit outside my lane.

Hi everyone,

Sorry to make another „market is bad“ post. But seeing some good folks lately getting laid off and not being able to find a job quickly, I am open for advices for European market,

How do you look for a new job nowadays? And if you ever think of pivoting to another field, which fields do you think are best suitable for TnS veterans?