u/mxroute

Most people these days would probably say "You can't." I get the instinct. Everyone is using AI wrong. EVERYONE. I cursed endlessly at the Disney Plus AI just the other day, worthless junk. Everyone is trying to use AI to replace humans on support, and force customers to talk to an AI. AI can't handle customer support, it's simply too complex.

For me though, I've been waiting for AI my whole life. This isn't something I'm trying to use to save money. I'm not trying to weight the loss of revenue vs the savings from firing the support team. So while everyone else is misusing AI to completely bomb customer support, I have a question to answer: How do I make everyone else look bad and use it the right way? 

I actually have an answer. Most of you that received a support ticket response in the last two weeks unknowingly saw the answer. It's actually pretty simple. That's why it makes everyone else look stupid, and I like that. I needed a confidence boost.

The answer is that it doesn't replace support at all. It's a tool that helps support run faster, smoother, and with more accuracy. Here's how it works:

Our obsessively secured, completely private and locally hosted LLM, read every support ticket that we've answered this year. It can access logs and basic account information by internal API, and no it has absolutely no way of exfiltrating that data. Nor does it have a way of accessing the functions that limit it's access. Simply, it guesses what we were most likely to say in reply to a support ticket, and it writes that in a draft. The human walks in, checks the ticket and draft, and makes a decision. Is this what I would have sent? If yes, send it. If no, reject it (and send feedback to the LLM). If almost, edit and then send. If the reply requires action from us, it determines what we were most likely to say and leaves us a note with the required actions to send the draft (if we agree, of course).

The customer never speaks to an AI. The AI simply helps the human reply faster. Less browser tabs, less log greps, less "I forgot how that feature works, let me check before I reply."

Just because everyone else is using it wrong doesn't mean we're not going to use it. We're just going to use it as a middle finger to all of them.

Our support ticket backlog is down to the smallest levels in years. We're almost completely caught up. A human has been involved in every single ticket. And best of all: We didn't have to increase prices to do it.

The real large one is here:

https://docs.mxroute.com/docs/troubleshooting/smtp-rejections.html

This is the massive result of years of customers demanding we filter this or that better, and me wanting to return specific errors for different scenarios. It means we have quite a large number of potential SMTP responses, and keeping up with them can be a bit messy.

The next, the management.mxroute.com portal is now fully documented:

https://docs.mxroute.com/docs/management/

And finally, the panel.mxroute.com control panel is now fully documented as well:

https://docs.mxroute.com/docs/panel/

My position has been "The button is right there. You don't need to read an article to see it." Obviously, that attitude has been a failure. We're constantly asked questions about buttons that are right there. So there we are, complete product documentation.

A lot of you have seen this SMTP response. It means you tripped a filter designed to catch the moment that a bad actor successfully tests your SMTP credentials. It blocks all outbound mail until I personally review the behavior and either clear the block or open a support ticket with you indicating the problem, the evidence, and the path to resolution. It also catches you every time you send "SMTP test" because I'd rather catch an unimportant email than let a compromised account run free for a while.

There was a time where this would catch less than 5 email accounts per week, and this manual review consumed very little of my time. For a while now, it has consumed a great deal of time because you people can't keep your shit secure. It's fine I get it, things happen, but the real point is we've outgrown manual review.

The problem with automation has been that the decision to clear the block or suspend the email account requires strong reasoning well beyond the capability of scripting. You need an eye for it.

As of tomorrow, this job is being taken over by my GPU (in an environment so well secured it could only be explained by extreme paranoia). It has been trained on the data, and it has greatly exceeded the logic of both my filters and my reasoning. For the past week it has been running for observation and detection only. It had 1 false positive, while my outbound filters designed to catch these have had as many as 5 false positives per day during the same time frame. Each one of my false positives were unable to send mail for several hours while awaiting human review.

What this means for you:

1. Faster resolution for accounts suspected of being compromised.

2. No outbound block until compromise has been confirmed.

3. Less false positives.

So basically, it's a win for everyone and a negative for no one. Except maybe for people who hate GPU workloads, but they're creeps anyway.

Spent a few hours today redoing the user impersonation system on the panel. Mitigating third-party bugs caused unnecessary complications in that code path, which held back this change. The result: resellers can now manage customer accounts via API. This enables the easier creation of custom control panels for resellers who want to completely own the UX.

Alright, back to support tickets and migrating Dovecot indexes to SSD on the Safari server.

Let's face it, you're using AI to manage your MXroute service. I know it. You know it. The people you're sending email to know it. However, some of your prompts are causing some mild annoyances for us. Those annoyances are ranging from obscene policy violations to minor cleanup jobs. It's best that wa get on the same page here. Please give this link to your AI agent:

https://mxroute.com/llms.txt

Your system administrator, posing as a "CEO," thanks you.

From the look of things today, all spammers everywhere must have found a new supplier of cocaine. I've never seen this many spammers so motivated to attack in one day. The entire day was lost to new trends:

1. All Mastodon instances were hit with registration spam so hard that they all collectively reached bounce rates with Google that are usually reserved for 1 customer every 2-3 days, many of them exceeded it. We've temporarily blocked outbound mail from Mastodon instances purely out of self preservation.

2. All new compromised email account trends. Typically we see a new trend that needs to be identified here once every 6+ months, and identifying the new trend helps us catch every compromised email account right away. This resulted in a flood of outbound mail to German ISPs, but fortunately hasn't yet resulted in any new blocking activity. (Note: Compromised email accounts typically come from credential stuffing, not server security issues)

3. Brute force attacks across the fleet have stepped up dramatically. But so has our detection and mitigation.

4. All new inbound spam trends are taking over, including severe attacks from Microsoft Azure that are difficult to filter without penalty to good senders. I'm trying, but I have to make slow and careful moves.

Some days suck, and you lose time better spent on product improvement, to unforeseen security concerns forced on you by third parties. But you take each day as it comes, and always rise to the challenge.

Today we were able to identify a massive spam/scam operation using our service as ground zero for their activities. They weren't sending spam email. Rather, they were running a gigantic forum and blog spam operation that ultimately terminated on our platform. The end goal appears to be a massive scam to steal money from unsuspecting strangers via public facing marketplaces by spamming forums and blog comments with links to scam operations like fake tours (attracting travelers/tourists), all ranked high on said marketplaces due to massive amounts of fake reviews. In the last 24 hours, I estimate that they spammed over 10,000 websites. Unfortunately, they've been with us since November of last year.

As we grow, it becomes more difficult to surface all of the unique ways that people try to spam or scam others while using our service as the base of their operations. But the goal is to keep our reputation spotless so that anytime an email comes from an MXroute customer, it means something, it's inbox worthy. But every time we find one of these operations, we get better at catching them early on.

The native iOS app for MXroute has been submitted to Apple for review. Fingers crossed. I know they can be kind of... you know, sometimes.

It's a simple app. It supports multiple accounts. Contacts and calendars sync with our Cal/CardDAV server, keeping them in sync with webmail.mxroute.com. Virtual folders with folder mapping functionality. Threading. Basic sieve filtering. Push notifications by folder (inbox enabled by default, can be enabled for other folders as desired). Bulk actions on email. Basic attachment support.

It's nothing fancy, but that's kind of what I was going for with it.

Today I pushed a new update to webmail.mxroute.com. Now you can login to multiple email accounts. First you login to what you'll treat as the "master," and then you'll connect multiple email accounts to it so that you can easily switch between them. Once you connect additional email accounts, you will only need to login to the first one each time to gain access to each of the email accounts that you had connected. To provide this in a way that I was comfortable with, we added 2FA to the webmail as well. To connect additional accounts, 2FA is required.

As usual, and as would be true with any amount of developers (yeah I caught what one of you were about to say), 1-3 of you are inevitably going to find some combination of steps that my brain failed to consider and you'll trigger what you would call a bug. I even pointed GPT 6 at the staging environment to try to find those scenarios, but you'll still find them I'm sure. So please do use the "Feedback" button at the top right of the webmail to report any bugs you experience.

Security notes:

I didn't want you to have to re-login to multiple email accounts every time your webmail session expired. To resolve that, it means that we need to store the password in a vault for each account that you connect. While 2FA doesn't protect IMAP/POP3/SMTP, if a webmail login gives you access to multiple email accounts then the value of 2FA on the webmail suddenly exists. It now becomes a necessity as one login obtains more data than could be obtained by the same login over IMAP/POP3.

Storing reusable passwords bothers me. I don't like to do it. However, I've done it in a way that satisfies my paranoia to a reasonable degree. The stored passwords are encrypted based on two values: a private server key and the password for the first email account that you logged in with (the "master" if you will). That first password isn't stored, you have to provide it at all times where it becomes necessary to encrypt or decrypt the passwords for the connected email accounts. So if you change the password to the first email account outside of webmail, you'll temporarily lose access to the connected accounts until you resolve the conflict in the UI (which you'll be prompted to do). Of course, changing the password to a connected account outside of webmail requires reconciliation as well. While I'm obviously not planning for webmail to be compromised, this would survive several different compromise events.

I started with a simple project, and as usual my paranoia overcomplicated it until it became this whole big thing. But the end user shouldn't need to know how complicated it is, it just might possibly cause them to ask "why" a few times. It should go without saying that I haven't covered every single security mechanism for the entirety of the webmail here, these are high level notes for this change.

Hey guys,

These are uncomfortable moments for me, because there are two values that I consider essential to running MXroute: honesty and transparency. It means that I'll be honest with you at all times, regardless of how bad it makes me look. I know that everyone makes mistakes, and I know that a lot of people prefer to do business with people who keep their mistakes to themselves. I know that causes some friction for some people. But as profit has never been a primary goal for me, I'm never going to let go of these values. If the truth sinks me, then that is the outcome I deserve.

As we've grown this year, IMAP performance and reliability has continually degraded. I was never looking for old, long standing problems. I was looking for new problems to address what I perceived as new behavior. I chased symptoms, not the root problem. It never occurred to me that the behavior could come from a fundamental misunderstanding over a decade in age.

What this is, it's one of those moments where you "know" something for so long that you never go back to question it. Whether that origin came from something I was taught earlier in my career, caused by vague documentation or simply misread documentation, it's far too old of a problem for me to actually trace it back. The problem came from this: https://doc.dovecot.org/2.4.5/core/summaries/settings.html#mail_max_userip_connections

In my head, for as far back as I can remember, this variable meant this:

> The maximum number of IMAP connections allowed from each IP address.

When quite clearly, the documentation says this:

> The maximum number of IMAP connections allowed for a user from each IP address.

So in my head, a limit of 150 meant one IP address could make 150 IMAP connections at a time. Instead, it meant that one IP could make 150 connections at a time per email account. And when this became a problem, my focus remained downstream of this one configuration variable.

The reason growth caused this to become a problem was simply a matter of statistical probability. The more customers we have, the more likely it is that someone is going to make 4800 simultaneous connections to the IMAP server.

The good news is, I've corrected this misunderstanding across the fleet, and I'm declaring the 1600th victory on IMAP performance. And I've learned the value of two things:

1. Relying on "the truth I've always known" is dangerous. Because you can be wrong for a decade.

2. There should be nothing considered so basic that it should be above scrutiny. Every single value is worth being reconsidered at all times, and it should always begin with the documentation no matter how much I may think that I already know it.

I apologize for my failure, and I promise to do better.

Today is a sad day. I've been ignoring customer complaints to avoid doing this for months now. But here are the unfortunate facts for a company that I once respected greatly:

Their #1 export is spam. It's pissing our customers off, and they're letting me know, routinely.

As of today, we're proactively whitelisting all good senders that we can locate on the Rackspace network, as well as whitelisting their actual email service. But their networks as a whole? They're going on our Expert Spam Filtering blacklist (https://docs.mxroute.com/docs/expert-spam-filtering.html#expert-spam-filtering).

The goal here is to make sure we don't interrupt communication with the people you're already talking to on their network. But the rest of their network, we can't pretend anymore. We can't pretend that they're not a spam network ("networks" technically, multiple ASNs). Simply spinning up a server in their network and sending mail from it is now a negative signal. An email coming from their network is infinitely more likely to be spam or phishing than anything else.

For me, this is a "flag at half mast" day.

Today we had 5 unrelated customers popped in a row via Wordpress. I wouldn't be surprised if Comcast gets a bit upset about it, because the result was a brand new phishing campaign targeted specifically toward their customers. I've got a bit of a handle on it now, but times are getting harder when it comes to catching compromised email accounts before they send spam/phishing emails. So this is a quick reminder to anyone reading this that uses Wordpress: Update your shit.

Sometimes this is a new exploit. Sometimes this is an old exploit that someone just decided to mass scan for. Either way, I need your help as I can't control the other side.

I'm hiring someone on the "dark web" to physically assault an inanimate object in your field of vision.

You've been warned.

Edit: Back online.

ProtonMail is experiencing an outage. We are unable to deliver mail to them at the moment. It's possible that the emails in queue may bounce back to the sender. If you get a bounce for an email sent to Proton that says "This message was classified as rSPAM and may not be delivered" that does not mean that we wouldn't have delivered it if they were online, it simply means our backup relay provider filtered it, and they filter more heavily than we do.

For years our mail queues were held together by duct tape and threat of bodily harm. Yet another thing that bit us as we grew so quickly this year. The end result that customers would have noticed was roughly 1% (I'm rounding up) of email seeing unnecessary delivery delays in the morning hours. Most people didn't notice, as most of this wasn't time sensitive and frankly they didn't care if it reached their recipient (I could block most of them and they wouldn't open a ticket for 3 years).

Last night it reached critical mass, and it was death by a thousand cuts. No single noteworthy cause, but several small ones. It's possible that a few customers actually noticed. So that's what I did today. I sat in one place, had Panda Express delivered (can eat on that for like 2 days), and managed the queues by hand until I finished addressing each little problem that caused it to get out of hand.

I tracked down every tiny problem and replaced old bandaid solutions with actual solutions that stand the test of time.

It's super boring. You don't care about this. This is basically a boomer Facebook post.

Name and shame. Another ASN controlled by spammers. Looks like they're putting some effort into it too. Too bad their test round was their last round.