Hey r/redditeng! I’m Lisa. You probably don't know me, but I've been here the whole time. 

For the last six years, I've been Chief of Staff to Chris Slowe, Reddit's founding engineer and CTO (now our first Technical Fellow). I'm also the person who founded r/redditeng over five years ago with Jerry, Punit, Braden, and a couple others who joined me in getting it off the ground. We wanted to give Reddit engineers a place to talk about what we build, how we build it, and what we learned along the way. We also wanted to show you what it’s like to work here. Oh, and we wanted to show you our cats, too!

My last day at Reddit is coming up, so before I go, I want to say thank you.

Thank you first to all of you for reading. It's still a little amazing to me that thousands of people chose to follow a subreddit about how we build Reddit. Some of you got excited to work here because of posts you saw! And some of you got hired. Thank you for the comments, the questions, the upvotes. It means a lot to our writers. 

Thank you to everyone who has written something for us over the years. You are amazing! We've published something every week since the sub was launched on 4/15/2021, and that only happened because people were willing to take time away from their actual jobs to explain their work, share what they'd learned, and put their ideas out into the world, which can feel a little scary, no matter how much you know. One of my favorite parts of doing this has been helping make that a little easier. You’re my people. You’ve always been. 

Thank you to our Legal and Comms teams, who have supported us when we've needed them and, just as importantly, trusted us to do this well. That trust gave our little volunteer team and our engineers room to talk honestly about our work while helping us navigate the moments when we needed a little extra guidance.

And thank you to Chris for everything and for trusting me to start this in the first place and supporting it for all these years. Like me, he believes our engineers should have a place to tell their own stories, in their own voices. He gave us the space to make that happen.

Most of all, thank you to the engineering branding team, past and present. What started as a handful of us trying something has become a real team of people who care about telling these stories and making this community good. One of the nicest things about leaving is knowing that r/redditeng (or Snoosweek) doesn't need me to keep going. But, damn it, I’m going to miss it all. 

I'm very proud of what we made here. I’m happy it’s going to carry on and I can be one of your readers. I may be finishing my time at Reddit but I haven’t finished loving the people, the platform, or this sub.  

Thanks for reading all these years. And please keep reading. There's a lot more good stuff to come.

P.S. Paying the cat tax with the cats who help(ed) me with my work over the years here

Daniel and Sam (RIP Sam)

Oliver

Noah (RIP)

Eli (RIP)

Theodore \"Teddy\" Meowsevelt

Written by Alexey Bykov, Staff Software Engineer at Reddit & Google Developer Expert for Android technology

Reddit serves approximately hundred of millions playbacks a day on Android.
In our last two posts: Improving video playback with ExoPlayer and Taking ExoPlayer Further: Reddit's performance techniques we covered ExoPlayer performance and what you can do to improve startup latency, rebuffering, video quality & stability. (Check them out if you haven't yet, they also show how your production metrics may improve after every optimisation)

But bundling those performance practices into a reusable component with a safe and clear API turned out to be a different challenge. Over time, every new video integration became harder, and we kept finding edge cases that were difficult to support without breaking existing behaviour.

In this article, we'll share how we rewrote our abstraction on top of ExoPlayer from scratch: a better API for teams building video features, and even better performance and stability.

Goal: One obvious way to play video

Our main requirement for the API was simple: engineers integrating video shouldn't need to be video or performance experts. They shouldn't have to think about prefetching, player creation, prewarming, or lifecycle. Video should be fast by default and take a few lines of declarative code to integrate.

ExoPlayer is considered one of the best open-source video players among all platforms. It gives you a dozen valid ways to get to the same playable video, and at our scale, that freedom stops being a feature and starts being a challenge.

Tim Peters put it best in the Zen of Python: "There should be one, and preferably only one, obvious way to do it."

Architecture

Playback engine: ExoKit

To bundle all of the optimisations together, we built an agnostic playback engine called ExoKit.

https://preview.redd.it/7zeheph0a3ch1.png?width=1999&format=png&auto=webp&s=72be5448ce219a028e75822a603b348987e7ad64

It abstracts away ExoPlayer, makes the API stricter, owns all player communication, and keeps every playback in a central place with a single app-wide video state and effects handling.

It supports all of the performance features we covered earlier, such as:

• Player pooling and prewarming the pool on the app start, since creating a player can still take up to ~200ms according to our production traces
• Decoder reuse for identical videos, which saves up to ~80ms more
• Warming up videos during first composition (not to be confused with prefetching), so the first keyframes are decoded and rendered before the user scrolls to the video.

You can read more about these optimisations and their impact on our metrics here.

On top of that, it offers two APIs: a declarative one for rendering and an imperative one for managing playback state.

u/Composable
fun PostVideo(
    mediaId: String,
    url: String,
    modifier: Modifier = Modifier,
) {
    val key = remember { PlaybackKey(mediaId, "feed") }

    // Imperative API: express playback intent.
    val actions = rememberPlaybackActions()
    PlayButton {
        actions.action(key, PlaybackAction.Play)
    }

    // Declarative API: render from playback truth.
    val state = rememberVideoPlaybackState(mediaId)
    if (state.isBuffering()) {
        LoadingIndicator()
    }

    // Declarative API: describe the video surface.
    Video(
        modifier = modifier,
        props = VideoProps(
            url = url,
            playbackKey = key,
        ),
        surfaceLifecycle = rememberLifecycle(),
    )
}

But even though it looks minimalistic enough, it still leaves a lot of responsibility on engineers. They have to decide where to play the video (which gets tricky with autoplay on), manage the player lifecycle, fire telemetry and coordinate between the declarative and imperative APIs.

To make it easier, we landed on a much more opinionated API. 

Self-hosted UI Components

Our first step was to abstract the imperative API from the feature layer: one less decision means one less way to get it wrong. But that logic had to live somewhere.

Rather than building reusable abstractions and inject them into every screen's ViewModel, we went the opposite way: every UI component became fully independent and self-hosted.

https://preview.redd.it/0j7sidx6a3ch1.png?width=1999&format=png&auto=webp&s=750eeccabc33f9602262ccac1eb32b7b9b3f2878

To ensure every composable can access product-related context, such as analytics, every block implements the following contract:

interface Component<Props : Any> {
    u/Composable
    fun Content(props: Props, modifier: Modifier)

    fun key(props: Props): Any = props
}

Implementations live in an impl module and can also access product-related context.
Here is a simplified example of what a lower-level composable might look like inside a component implementation:

u/Composable
// Very simplified version of the code for mute button
fun MuteButton(
    playbackKey: PlaybackKey,
    modifier: Modifier = Modifier,
) {
    // Imperative API
    val playbackActions = rememberPlaybackActions() // manage playback
    val globalActions = rememberGlobalActions() // global state, e.g. settings

    // Declarative API
    val audioSettings by rememberAudioSettings()
    val playbackState by rememberPlaybackState(playbackKey)

    // no sound -> hide, loading -> settings fallback, has sound -> real state
    val muted = when (playbackState.audio) {
        AudioTrackState.HAS_NO_SOUND -> return
        AudioTrackState.UNKNOWN -> !audioSettings.isEnabled(playbackKey.surfaceId)
        AudioTrackState.HAS_SOUND -> playbackState.isMuted
    }

    IconButton(
        modifier = modifier,
        onClick = {
            globalActions.action(
                GlobalAction.SetSurfaceAudioSetting(
                    playbackKey.surfaceId,
                    audioEnabled = muted,
                ),
            )
            playbackActions.action(
                playbackKey,
                PlaybackAction.Mute(!muted),
            )
            // More things, like handling product-related telemetry
        },
    ) {
        Icon(if (muted) Icons.VolumeOff else Icons.VolumeUp)
    }
}

The same pattern applies to every clickable or interactive media component. Every UI component also usually has its own ViewModel.

At the screen level, media becomes a set of declarative components that can be arranged like any other UI:

u/Composable
fun SimpleVideoScreen(
    screenState: SimpleVideoScreenState,
    videoComponent: Component<VideoProps>,
    playComponent: Component<PlayProps>,
    muteComponent: Component<MuteProps>,
    seekbarComponent: Component<SeekbarProps>,
    modifier: Modifier = Modifier,
) {
    val key = screenState.playbackKey

    Box(modifier.fillMaxSize()) {
        // 1. Video
        videoComponent.Content(props = screenState.toMediaProps())

        // 2. Play
        playComponent.Content(props = key.toPlayProps())

        // 3. Mute
        muteComponent.Content(props = key.toMuteProps())

        // 4. Seekbar
        seekbarComponent.Content(props = key.toSeekbarProps())
    }
}

Usually, the Media Foundation team, which focuses full time on media performance and developer experience, implements the components whose internals use both declarative and imperative APIs, whereas feature teams interact with the player through the declarative API.

This new setup gave us three big benefits:

• Product screens stay simple. They only need to arrange the media components and pass in a few props. Nothing more.
• Playback works the same way across the whole app. Every component shares the same playback state wired by playback key, actions, settings and telemetry pipeline under the hood, so users get one consistent experience no matter where they are.
• Performance. Screen-level state does not need to update for every media-related event. Only the affected media component recomposes.

Since introducing a component-based model in our app, AndroidX has made a new addition to their Media3 UI. This new model might serve as a good starting point if you’re thinking of solving similar problems today.  One key difference remains: our media components are tied to global playback state, while Media3 UI’s are tied to one Player. So the challenges still remain the same: player creation and the rest of the lifecycle ownership.

Experimentation Setbacks

Choreographer-based Seekbar

I personally found the seekbar a challenging component to implement smoothly, and there are a few interesting decisions we made.

So what does “smooth” mean in practice?
Basically, don't do more work than the screen can render. A 60Hz display takes roughly 16.7ms per frame; at 120Hz (which is not a rare thing anymore), that reduces to  ~8.3ms to render a frame. And if it goes beyond this limit, the user sees a jank.

This is where Choreographer can help. It runs right before Android draws the next frame, using the display’s real refresh rate. If the previous frame takes too long to draw, it waits for the next frame instead of firing again. That means we do not stack seekbar updates on top of an already-janky UI.

Estimated position or player position?
In our custom seekbar we predict the position by extrapolating from the last update, and only update it if the video is playing.
If your application's use cases are limited to video, ExoPlayer.getCurrentPosition() is a cheap operation, and it already uses an estimated position under the hood.

long elapsedTimeMs = SystemClock.elapsedRealtime() - positionUpdateTimeMs;
long estimatedPositionMs =
    Util.usToMs(positionUs) + (long) (elapsedTimeMs * playbackParameters.speed);

Polling this function will give you a smooth, speed-aware position for free. My recommendation is to still drive the updates using Choreographer and not use an arbitrary delay within a Coroutine or an Handler. 

Reddit's production experience
Moving seekbar updates out of screen-level state and driving them with Choreographer made the full-screen video experience measurably smoother.

On our full video screen, the slow-frame rate dropped by 7.3%.

https://reddit.com/link/1urq7f6/video/ifte8fv9g7ch1/player

Playback error 1004 & MediaSource reuse

While experimenting with our newly written ExoKit in production, there was a noticeable jump in playback error 1004 with "Unknown error" message. It affected a small share of devices and was challenging to reproduce locally. Our initial hypothesis was that it was a device-specific playback issue.

The root cause was our MediaSource cache. In some cases, the same cached MediaSource was reused across different player instances. This is something that I wouldn't recommend doing, as every media source is bound to the playback handler of the player it was originally attached to. Otherwise, you risk hitting an error which is thrown here.

Reddit's production experience
We fixed this by keying cached media sources by player id, so a MediaSource can only be reused by the same player instance.

Another possible fix was to move all playbacks onto a single playback thread. That also avoids the handler mismatch, but our production data showed that it made overall startup performance worse:

• % Video started in less than 250 ms: −0.199%
• % Video started in less than 500 ms: −0.132%
• % Video started in more than 1 sec: +1.165%
• % Video started in more than 2 sec: +0.634%

Rebuffering problems

If your app saves and restores playback position, be careful. In our old player we found that at least 36% of plays had a very short stall, under one second.

The main reason came from old choices made years ago. The old player restored the saved position as a check on every play and every move between screens. But the saved position did not match the position the player held at run time. Here is why: if you call player.pause() and then save player.position, and later call player.seek(savedPosition), the two numbers may differ by a few milliseconds. player.pause() does not finish right away. If you want the true position, wait for a playback state change first.

Reddit's production experience
We stopped saving positions this way. Instead we lean on the run time cache kept in the media source, and we reuse the player for videos the user watched before. This removed all of these tiny stalls.

Saving position on your own is still useful when the system kills the process of your app and you need to restore it for a long video.
If you cover this case, watch the order of your calls. Always call seek() first and prepare() after. If you do it the other way, the player loads key frames and chunks you don't plan to play.

Trade-off: Only one playable video at a time

Simultaneous playback is possible on Android, but hardware decoders are finite and shared across the whole device. A high-end phone might decode two or three videos at once, a low-end one just a single video, and an app sitting in Picture-in-Picture can quietly hold a decoder you were counting on. When you run out, you either fall back to software decoding (higher CPU usage, dropped frames) or fail playback with errors like 4001 or 4003.

Reddit's production experience
We could have partly mitigated this by checking the device performance class (a challenge in itself, since not all devices support it), but the development and testing cost wasn't worth the benefit. Instead, ExoKit runs a state machine that picks one active playback by priority (at Reddit it's based on how much of the video unit is visible, but it could also be based on how much of the screen's playable zone it fills).

We expected every surface to start faster after the rewrite. But surfaces that went from playing several videos to playing one improved their startup latency even more than the ones that already played a single video in the control group.

Conclusion

Besides an improved developer experience, the rewrite reduced perceived start latency by 65% at P50 and by 20% at P90. (An additional factor which helped here was the removal of a lot of unnecessary IO work from the startup path. In production traces, a single Main → IO dispatch before playback could add up to 11ms at the P99.)

Kudos to Merve Karaman, Ahmed Nawara, Irene Yeh and Vikram Aravamudhan for making this rewrite possible. We used to talk about this as a dream 2-3 years ago, and now it's our new reality.

Thanks to the following folks for helping me review this article: Iaroslav Khramov, Nicholas Ngorok

Hi friends - The job market is rough, and a lot of good people are looking for work. Good news is, Reddit is hiring across a wide range of functions, not just Engineering.

If you're looking, or know someone who is, take a look: redditinc.com/careers

Feel free to share with your network, group chats, Discord servers, fantasy leagues, family text threads, and other highly sophisticated professional recruiting channels. And not just with your engineering friends. Let's look out for one another!

Good luck out there. We're rooting for you.

https://preview.redd.it/2jcwvm1qlg7h1.jpg?width=828&format=pjpg&auto=webp&s=2f492f00a9e78dd5916fe9320ce933581ca82786

Written by Shadi Atarsha, Transport team

When I started my career in infrastructure engineering, I wasn't sure how my work connected to the people actually using the product. I imagined myself deep in low-level systems that nobody knew existed, the kind of work that only surfaces when something breaks at 3 AM. 

Four years later, I've come to understand what I believe is one of the most important responsibilities of an infrastructure engineer: Keeping infrastructure concerns off application and platform engineers’ plates, so they can focus on their domain instead of paying the tax of context-switching into ours.

The way you deliver that experience is by doing the hard work of managing complexity, and building simple interfaces and abstractions so other teams don't have to think about it.

This post is about one such effort: how we removed an entire Envoy proxy from the serving path of Reddit's Home feed, Search rankings, and Notifications, and in doing so, improved availability, cut hundreds of CPU cores, and made onboarding new services trivial. The technical story involves gRPC service mesh, cross-namespace routing, CPU-aware load balancing, and a migration pattern for safely moving thousands of requests. But the underlying theme is simpler: sometimes the best thing infrastructure can do is disappear.

The Problem

When you open Reddit, your personalized experiences are powered by Ranking Platform, which is a set of gRPC services handling tens of thousands of requests per second across multiple Kubernetes namespaces and clusters. Traffic comes from our primary API gateway (GraphQL), Notification Platform, Answers, and other clients.

All of this traffic used to flow through an Envoy Gateway, a reverse proxy deployed and maintained by the Ranking Platform team, routing requests to the right service based on gRPC method and custom headers.

https://preview.redd.it/3f6kh3qoko4h1.png?width=1999&format=png&auto=webp&s=8801e7fee8cb9d756d0761c628fbe7221ff8c644

Here is the thing with great tools like Envoy: adopting one without deep experience is like giving a team a Formula 1 car when they have only ever driven regular sedans. A sedan is forgiving. You can change your own oil, and when something breaks it stalls on the side of the road. An F1 car has hidden sharp edges. It takes a pit crew to keep it on the track, and when it does fail, the failure is loud.
At first it feels like a huge upgrade. Then the complexity sneaks in. Configuration explodes, operational overhead grows, debugging becomes non-trivial, and before you know it, the team is not just the driver anymore. They need to be the mechanic and the pit crew too, on-call for a big dependency they did not originally sign up to own.

Why doesn't the Transport Infrastructure team just own these gateways? 
For one, we are a small Transport team supporting a large fleet of services across the company. More critically, the value of this kind of infrastructure comes from consistency across the fleet, which is hard to achieve when each team manages it independently.
We love Envoy and our Ingress deployment serves us well, but for gRPC service-to-service communication, we chose simplicity. So about a year ago, we adopted a proxyless service mesh instead.

Portal Transporter

https://preview.redd.it/sw9n0ggtko4h1.png?width=1999&format=png&auto=webp&s=f07e4a16f0d51d34a7dfdf676b82ecef437cc4d7

Portal Transporter is Reddit's xDS-based control plane. It is a Kubernetes controller that watches GRPCRoute resources, EndpointSlices, and services, and then builds an xDS snapshot that it advertises directly to gRPC clients. The client dials an address like xds:///ranking.ranking-platform and gets back everything it needs, including routing rules, endpoint lists, load balancing configuration,  without any proxy sitting in the middle.

Instead of Client -> Proxy -> Backend, the architecture becomes Client -> Backend. The gRPC client itself handles routing and load balancing, programmed remotely by the control plane. This means you don’t need a sidecar, or a gateway.

If you want to understand the system in depth, my colleague Sotiris Nanopoulos gave a talk at gRPConf last year about it: Building a gRPC Proxyless World: How Reddit Scaled Resilience with xDS. It covers the control plane architecture and how we handle client-side observability as well. I would really recommend watching it if this topic interests you.

For this post though, all you need to know is this: Portal Transporter lets a service owner define their routing table once, and every client in the fleet gets it automatically. No client code changes needed. That is the foundation everything else in this post builds on.

The Goal

Here is the thing, to migrate one of the most important customers in the company to our infrastructure golden path, you cannot just show up at their door and say "hey folks, please use our toys." 

I approached this migration by asking myself a series of questions:

• In an ideal world where engineering cost is irrelevant, what is my perfect end goal?
• How much effort do my team and the team I am migrating have to pay to reach that end goal?
• Okay, that is very expensive. How can I find a common ground that achieves 90% of the dream goal without burning everyone out?
• And most importantly: how can I make the migration safe for both my team and my customer, and build trust in the process?

After working through these questions, here is what I landed on:

• Migrate the Ranking Platform to Portal Transporter without changing any of their existing architecture. I did not want to walk in and tell the Ranking team to restructure their services. The migration should be invisible to them as much as possible and the clients should only dial one address similar to how they used to do with Envoy.
• Cross-namespace routing was our biggest blocker. When I sat down and studied the Envoy Gateway configuration in detail, I realized we could not do this yet. The Ranking Platform team runs a cross-namespace setup, with services spread across multiple Kubernetes namespaces but unified behind a single Envoy routing table. Portal Transporter did not support that at the time.
• There were other challenges too. Envoy was a major source of SLO metrics and alerts for the Ranking team. They had even written a custom WASM filter to add tailored metrics to Envoy, which made replacing the observability layer harder than just swapping the data path. We needed to bring equivalent or better observability to the table before we could ask anyone to move.
• And above all, I had to do this safely. This is not some low-traffic internal tool, it serves traffic that powers some of Reddit's most important user experiences. A bad migration here would be very visible.

On the positive side, I was not coming empty-handed. I personally believe standardization is valuable, even if not everyone shares that belief as strongly, so I knew I needed something more concrete to bring to the table: CPU-aware load balancing (ORCA), which lets gRPC clients route requests based on how busy each server actually is. In practice that means better pod balance under load and real CPU savings, which are the things teams actually optimize for.

In the upcoming sections, I will walk through how we tackled each of these blockers and how we arrived at the architecture below.

https://preview.redd.it/i3osbty2lo4h1.png?width=1999&format=png&auto=webp&s=0ff839859e3050054ada7164e99039c3d90cc942

The missing piece: cross-namespace routing

Before we could migrate anything, we had to solve a fundamental gap in Portal Transporter. The Ranking Platform's services all live in separate Kubernetes namespaces but are served through a single unified routing table. Clients call one address and the routing layer figures out which namespace to send the request to based on the gRPC method and headers. Portal Transporter had no concept of this. It assumed every GRPCRoute lived in the same namespace as the service it was routed to.

To make things harder, we already offer a public API to service owners across Reddit that supports exactly this kind of parent-child relationship across namespaces for the Ingress layer. Our ingress infrastructure relies on it, and our cross-cluster routing is also using it. Whatever we built for Portal had to fit into that same API. I would not introduce a new, ambiguous API or force changes on teams already using it. 
One API surface, two backends, that was the constraint.

This took a lot of iteration to get right. My first instinct was to look at what the Gateway API spec already offered. The Gateway resource has a model for cross-namespace routing, and it seemed like a natural fit. But we realized it was built on different assumptions than our infrastructure. So we stepped back and looked for something simpler.

What we ended up with is actually quite clever, and I say that knowing it took me three attempts and a lot of help from my colleagues to get there. We decided to use the GRPCRoute's backendRef field to reference other GRPCRoutes from different namespaces. Here is a simplified example:

The root GRPCRoute, let's say in the reddit-service-streaming-pipedream namespace, defines the main address that clients dial. But instead of routing directly to a service, some of its rules point to GRPCRoutes in other namespaces as backend references:

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: ranking-platform
  namespace: reddit-service-streaming-pipedream
spec:
  rules:
    - matches:
        - method:
            service: reddit.rankingplatform.ranking.v1.Ranking
      backendRefs:
        - kind: GRPCRoute
          name: ranking-comments
          namespace: reddit-service-ranking-comments
        - kind: GRPCRoute
          name: ranking-channels
          namespace: reddit-service-ranking-channels
        - kind: GRPCRoute
          name: ranking-onefeed
          namespace: reddit-service-ranking-homefeed

Each GRPCRoute in its own namespace then defines its own specific routing rules and points to its actual service. When Portal Transporter's control plane encounters these backend references, it expands them by pulling in the child routes and building a combined routing table that it serves to clients as a single xDS snapshot.

This design maps cleanly to the parent-child model our public API already supports for ingress. Service owners define their setup once, and it generates the right resources for both our ingress layer and Portal Transporter. No breaking changes or any special cases.

I want to be honest about the process though. This was not a clean straight path. The first design went through code review and was ultimately closed. The second attempt at combining GRPCRoutes had the right idea but the wrong execution. It was only on the third iteration, with significant input from my teammates, that we landed on the backendRef approach. It took several months from the initial proposal to the merged PR. 
I think there is something worth saying about that: good infrastructure design is not about getting it right the first time. It is about iterating and maybe discovering the real requirements as part of the process and it is about having colleagues who will tell you when an approach is not working, and being willing to throw away code that you spent weeks on (it is not easy when your “ego” is involved, by the way) and I am grateful my team gave me that.

Metrics and Alerts

Envoy was a major source of SLO metrics and alerts for the Ranking team. This was not a small detail but a real blocker.

At Reddit, we offer a standardized set of gRPC metrics and recording rules that teams can use to build dashboards and alerts. The Ranking Platform has a specific challenge: they use the same gRPC service and method across all their components, and they rely on custom headers like x-route to direct traffic to separate deployments. To get the breakdowns they actually needed, we had to add a custom layer on top of the standard metrics.

There was also the client-side SLO question. Client-based SLOs are an area we are still investing in, so there was no off-the-shelf path I could lean on for this migration. Aggregating metrics across multiple client sources is also not trivial, which meant I had to be pragmatic about scope.

This is where the "ideal world vs reality vs common ground" framework came back. A perfect, like-for-like replication of the existing observability layer would have taken months. But I could deliver a solution that gave the Ranking team most of what they actually needed:

• We built custom server and client metrics using grpc-go's StatsHandler, with the required headers as labels.
• We injected these metrics into the Ranking Platform components on the server side and into the clients.
• We used the server-side metrics to build alerts for each ranking tenant.
• We use the largest client as a stand-in for client-side SLO measurement.

Is it a one-to-one replacement for Envoy's observability? No. Does it produce a reliable outcome that the Ranking team can actually use? Yes. And honestly, I think that is the right tradeoff because perfect parity would have taken months of additional work and delayed the entire migration. 

Good enough, delivered now, was the better call.

De-risking the rollout

Here is a secret on how you make your managers and users happy at the same time: reliability. If there is one north star we operate under at Reddit Infrastructure, it is reliability. Everything else flows from that.

How do you deliver reliability when you are shuffling many RPCs?

Before migrating any traffic, I wanted to assure correctness to the best of my knowledge. And I want to be honest here, I really do not understand everything going on with the Ranking Platform and its clients. I do not know exactly what Notification Platform is calling or how Answers gives you all the information you need about the best daily shoe recommendations (spoiler: it is the Adidas Evo SL, thank me later). So what could I do to build confidence without pretending I understood every edge case?

I broke it down into three requirements:

1. Whatever development environment the engineers who work on Ranking components use must support Portal Transporter, not Envoy.
2. The Ranking Platform CI needs to run tests against my new setup and Envoy at the same time, and I should expect 1:1 correctness if my routing table is right.
3. My migration knobs should be able to turn traffic up and down between Envoy and the proxyless mesh in under two minutes.

For number one, we designed Portal to be development-friendly from day one, so there was no fundamental issue here. I just had to do some tweaks to make cross-namespace routing work in our development environment, but nothing major.

For number two, thanks to the Ranking Platform team, they already had very well thought-out smoke tests in their CI running against Envoy. All I needed to do was hook up the same calls via Portal alongside the existing tests and hope for the best. To my relief, it worked, the routing table was correct and the smoke tests passed on both paths.

Number three is where it gets interesting. We built a gRPC sampling client that has been part of Portal Transporter's migration toolkit since the early days. The idea is simple: we spin up two gRPC clients at the same time, one for the existing path (Envoy) and one for the new path (Portal Transporter). We use our dynamic configuration framework to provide a deterministic sampling knob that allows us to shift traffic between the two on the fly, straight from a UI. If something looks wrong at 10%, flip it back to 0% in seconds.
Under the hood, the implementation is actually quite neat. One of our engineers had the idea to override the ClientConnInterface interface in grpc-go, so that when Invoke is called, it checks against our rate sampler to decide which client handles the RPC. This means the sampling is completely transparent to the application code and the service owner does not need to change anything.

Teaching Servers to Tell Clients How Busy They Are

Remember when I said standardization itself is valuable, but I'm bringing something extra to the table? This is it.

With Envoy out of the picture, your gRPC clients now connect directly to server pods. The default load balancing strategy is round robin where every pod gets roughly the same number of requests. Sounds fair, right? The problem is that "fair" and "efficient" aren't the same thing as not all pods are equal. Some run on busier nodes, some share CPU with noisy neighbors, some just got unlucky with garbage collection timing. Round robin doesn't care, it sends traffic to an overloaded pod just as happily as it sends traffic to an idle one.
We implemented ORCA (Open Request Cost Aggregation) which is a standard that lets servers whisper back to clients: "hey, here's how busy I actually am." Each server pod reports its CPU utilization in gRPC response trailers, and the client uses these signals to compute weights. 
More loaded? Fewer requests. Less loaded? More requests. 
This is Client-Side Weighted Round Robin, and it's beautiful in its simplicity.

For Ranking Platform, the impact was significant: tighter CPU distribution across pods, improved availability, and a meaningful reduction in the number of CPU cores needed (we're talking about hundreds of cores saved).

My colleague wrote an excellent deep dive on ORCA at Reddit: Cheaper, Safer Scaling of CPU-Bound Workloads. If you're curious about the weight formula, the observability setup, and the operational lessons, go read that. It's worth your time.

After the migration

• We secured another nine on Ranking Onefeed.

https://preview.redd.it/s4md2jzhlo4h1.png?width=1999&format=png&auto=webp&s=96d48270e5aaa3eddecc6aed6c715e85f3064f5d

• Hundreds of cores reclaimed across Ranking workloads, thanks to ORCA.

https://preview.redd.it/xeswmt8jlo4h1.png?width=2048&format=png&auto=webp&s=4401bc5eb935f2a80169152ef0c8c9378c0748da

https://preview.redd.it/lv6smm5vlo4h1.png?width=1999&format=png&auto=webp&s=b70fd137af2516d76e5d8111c8defac49b86ad35

• Latency stayed flat
• Envoy Gateway fully decommissioned
• Simplified architecture
• Standardization: Ranking now uses the same infra golden path as the rest of Reddit

Conclusion

I started this post by saying that sometimes the best thing infrastructure can do is disappear. I think we did that here.

The Ranking team didn't have to change their architecture. The Transport team moved one of the most important deployments in the company to the golden path. The migration was seamless and honestly, the best compliment we got was that nobody realized it happened. For an infrastructure team, that silence is very important because it means trust, and in infrastructure, trust is your currency.

We improved availability for our users, saved a lot of CPU cores, and deleted a system that no team should have had to own in the first place. 

This work was also recognized internally, and that recognition mattered to me and the team. Migrations of this scale are never the work of one person. They happen because engineers across Transport, Ranking Platform, and the surrounding teams chose to invest in each other, and I am grateful to all of them.

A very happy ending 😄

written by Nomi Khedawala, Technical Program Manager

Intro

I’m Nomi (Know-Me).

I joined Reddit as a Technical Program Manager in October 2024. I came from a background in product operations and technical program management, and what drew me to Reddit was the pace and the scale of the problems (and being a longtime lurker). I’ve had the opportunity to work on so many different areas of the business in my tenure here so far: Games on Reddit, Age Verification, Data Foundations, and even Search and Answers!

I'm part of the Tech PMO team, Reddit's centralized team for Technical Program Managers. Each of us manages programs for a specific org, set of teams, or one-off high priority initiatives. I recently wrapped up a company-wide Code Yellow focused on our consumer data foundations, and I'd like to share with you what a typical Thursday has been like for me during this Code Yellow.

If you're not familiar with Code Yellows at Reddit: they're our mechanism for escalating specific operational issues. When a problem is too big or too cross-cutting to fix through normal prioritization, we declare a Code Yellow. Think 4-6 week sprints with clear exit criteria and a temporary but significant shift in engineering priorities. They're designed to converge fast on a set of goals.

The Code Yellow I led alongside Paul Raff, our Data Foundations Lead, spanned the full consumer data supply chain across 10 surface areas. We defined how events should be instrumented, stood up real-time data quality monitoring, built curated data tables from raw events through aggregates and cubes, migrated metrics, and established a steady state operating model so the work would persist beyond the Code Yellow. It's the kind of program where the scope keeps wanting to expand and the TPM's job is to hold the line on what "done" actually means.

What follows is a snapshot of a typical Thursday during the Code Yellow. Thursdays were my heaviest day. Here's what one looked like.

Morning: 8:00 AM - 12:00 PM

8:00 AM - I start the day by facilitating a session of our Collaborative Problem Solving forum. This is something I built and run for our TPM team. The intent is that we get together every other week to work through real challenges we're each facing on our programs and look for opportunities to standardize how we operate. It's part peer support, part playbook development. Today I'm running a quarterly check-in to make sure the forum is still hitting the mark for the team.

9:00 AM - CPS wraps and I shift into Code Yellow mode. I'm reading through Slack messages that came in overnight and earlier this morning,. answering Qquestions from engineers on the Code Yellow and a thread with Paul about one of our exit criteria. I'm also reviewing every tracker, timeline doc, and milestone artifact to make sure nothing material has changed since we sent the Steer Co agenda out 48 hours ago. The Steer Co is a meeting with our executive sponsors and stakeholders, it’s later this morning and I want to be sure we walk in with a complete and current picture.

10:00 AM - Paul and I co-host office hours for any Code Yellow participant withquestions about scope, timelines, goals, or non-goals. These sessions are some of the most useful meetings on the calendar because the questions get genuinely technical. Today we're working through how to structure the data supply chain from raw events through fact tables, aggregates, and cubes. Someone raises a question about backfill strategies and the cost tradeoffs for different look-back periods. Another engineer wants to talk through whether we should have separate fact tables for web vs. mobile or consolidate by platform group. Paul and I don't always have the answer on the spot, but we talk it through with our counterparts and agree on a path forward whether that's deferment, fast-follow, or recommendation to senior leadership.

10:30 AM - More prep for the Steer Co. I'm pressure-testing each section of the agenda we drafted earlier this week. Is the status accurate as of this morning? Are the risks framed clearly enough for a room of senior leaders and our execs to act on? Do we need to add more context about a risk?

11:30 AM - I lead the Steer Co. This is our steering committee meeting with our CTO, CPO, VPs, and Directors. We share the program's current health, walk through progress against our five exit criteria, flag risks and blockers, and lay out what they should expect between now and our next check-in. Steer Cos are not the place for surprises. The prep I did all morning is so that every question gets a clear answer.

Afternoon: 12:00 PM - 4:00 PM

12:00 PM - Straight from the Steer Co into the cross-functional engineering sync. This is where I gather status updates from all of the eng teams contributing to the Code Yellow. We talk through their progress, timeline changes, risks, and I share relevant outputs from the Steer Co we just finished. Information has to flow quickly upwards and outwards during a Code Yellow. The engineers executing the work need to know what leadership is thinking, and leadership needs to know what progress is being made week over week.

12:30 PM - I meet 1:1 with a teammate to talk about how we can make the CPS forum even better. This week's session was my quarterly assessment, so we're comparing notes on what's working, what's not, and how to make the outputs more actionable. My goal is to take these findings, write up a retro, and present it to my manager and Sr. Director later to establish a baseline and start folding the best ideas into our team's standard operating procedures.

1:00 PM - Lunch. I close the laptop, make some coffee, and spend time catching up with my wife. One of the genuine perks of working from home is being able to take an actual break with the people you live with. Thumper, our cat, hates to see me coming, but I have to get a good arm wrestling match in with him so he doesn't sleep all day.

https://preview.redd.it/d1jk14d28w1h1.jpg?width=1999&format=pjpg&auto=webp&s=31938427152fe2c857bdaf24259437e70746c550

2:00 PM - Back at my desk. I go through my raw notes from the Steer Co and start synthesizing them into a stakeholder summary: key takeaways, discussion points, action items, and owners. Before I send it out, I reach out to each action item owner individually via Slack to confirm they're aligned on ownership and timelines. I learned a long time ago that assigning someone an action item in a meeting recap they didn't agree to is a fast way to create confusion and lose trust.

2:30 PM - I start drafting the weekly update that goes out to our Code Yellow participants and leadership. Milestones, health of each exit criteria, timelines, what moved forward this week, what risks or blockers surfaced, and how we're addressing them. This is one of those artifacts where the quality of the writing matters as much as the substance. If leaders can't parse your update in under 3 minutes, they stop reading it.

3:00 PM - I switch gears to the CPS retro. I'm taking the notes from my 1:1 earlier and incorporating them into the retrospective document I'm drafting for my manager and Sr. Director. The goal is to establish a clear baseline for the forum's performance and propose specific improvements that could be codified into how we run the CPS forum. This is the kind of work that doesn't show up on a sprint board but compounds over time.

3:30 PM - I'm starting to wrap up. I respond to Slack messages I couldn't get to earlier, schedule a few messages for teammates in different time zones, and share my draft weekly update with my program leads so they can add their content. I'll revise and edit tomorrow morning and get it out before the end of day Friday.

What a TPM Actually Does

A TPM's job is to make sure that the right things are happening at the right time and that everyone has what they need to focus on what they're good at. I handle the coordination, the communication, the risk identification, the stakeholder management, and the organizational scaffolding that lets a program with dozens of contributors across multiple teams actually converge on a set of milestones and objectives.

I love being a TPM because I love to be a part of other people's successes. Being a TPM means that you wear multiple hats so that the people around you can have peace of mind that all of the peripheral stuff is being handled while they can focus on their craft. TPMs are enforcers, enablers, coaches, communicators, therapists, and most importantly... friends not food! 🐟

If you're interested in becoming a Snoo, check out our open roles: https://redditinc.com/careers