Problem:

Let's say I have a domain like messaging-app.europa.eu, with an SSL certificate, and I use it to distribute a PWA. In the case of an internet blackout I would still like for the app to retrieve updates from a local mesh.

Possible solution:

This is the solution I'm currently imagining:

• Deploy a bunch of OpenWRT routers with some custom software
• The smartphone connect to one such router via wifi
• The PWA tries to connect to the domain https://messaging-app.europa.eu in the absence of internet
• The router has a copy of the SSL certificate, so it can intercept the request and generate valid HTTPS responses from a locally running server

A user would connect to the wifi, exchange messages with other peers connected to the same station without relying on an internet connection.

Questions:

• Do I just need a copy of the SSL certificate on each router, or is there other part of the stack that I should patch? (e.g.: DNS)
• Is it possible to intercept and patch the connection as I mentioned?
• Am I missing something?

Prodrome:

I work for the government and I've been asked to demo a solution for enabling communication during an internet blackout (earthquake, war, ...) using a mesh/p2p network.

The path I'm taking right now is to build a PWA that would work offline, and rely on customized openwrt routers, because it seems that the other solution, WiFi NAN, works well on android phones but not on iphones, due to malicious compliance on Apple side.

Problem:

Let's say I have a domain like messaging-app.europa.eu, with an SSL certificate, and I use it to distribute a PWA. In the case of an internet blackout I would still like for the app to retrieve updates from a local mesh.

Possible solution:

This is the solution I'm currently imagining:

• Deploy a bunch of OpenWRT routers with some custom software
• The smartphone connect to one such router via wifi
• The PWA tries to connect to the domain https://messaging-app.europa.eu in the absence of internet
• The router has a copy of the SSL certificate, so it can intercept the request and generate valid HTTPS responses from a locally running server

A user would connect to the wifi, exchange messages with other peers connected to the same station without relying on an internet connection.

Questions:

• Do I just need a copy of the SSL certificate on each router, or is there other part of the stack that I should patch? (e.g.: DNS)
• Is it possible to intercept and patch the connection as I mentioned?
• Am I missing something?

Prodrome:

I work for the government and I've been asked to demo a solution for enabling communication during an internet blackout (earthquake, war, ...) using a mesh/p2p network.

The path I'm taking right now is to build a PWA that would work offline, and rely on customized openwrt routers, because it seems that the other solution, WiFi NAN, works well on android phones but not on iphones, due to malicious compliance on Apple side.

At work I built a custom datastore / multimodel database on top of io_uring and Rust, which I would like to expose to the public via Cloudflare workers. Unfortunately there is a cost in setting up a new connection to the backend from a cold worker, cost which I'm trying to amortize very much like Hyperdrive does for SQL databases.

I imagined a solution like placement groups:

• I spin up a durable object which will set up the backend connection
• A bunch of workers, can connect to this durable object via service bindings for reduced cold startup time
• The number of durable objects scale with the number of workers
• Ideally the workers are on the same machine as the durable object, for reduced latency

Now my questions are:

• Is this the best solution for my problem?
• Is it possible to achieve this with Cloudflare workers?
• Is it possible to setup the load balancer affinity, so that I can route a connection always to the same placement group?

At work I built a custom datastore / multimodel database on top of io_uring and Rust, which I would like to expose to the public via Cloudflare workers. Unfortunately there is a cost in setting up a new connection to the backend from a cold worker, cost which I'm trying to amortize very much like Hyperdrive does for SQL databases.

I imagined a solution like placement groups:

• I spin up a durable object which will set up the backend connection
• A bunch of workers, can connect to this durable object via service bindings for reduced cold startup time
• The number of durable objects scale with the number of workers
• Ideally the workers are on the same machine as the durable object, for reduced latency

Now my questions are:

• Is this the best solution for my problem?
• Is it possible to achieve this with Cloudflare workers?
• Is it possible to setup the load balancer affinity, so that I can route a connection always to the same placement group?

At work I maintain a NestJS microservice which is used by many other engineers. One of the features depends on a third party API for which I need to implement rate limit:

• The API has a soft limit of 20 req/s
• There is no way to programmatically monitor this limit on their end
• Surpassing the limit means that when someone looks at the dashboard then they will manually disable us, and we have to start another manual process to unblock us
• This API is owned by the government, so we can't ask or hope for changes

How would you implement rate limit for this external dependency? Here's what I thought:

• Have a token bucket limiter inside each service instance, but then scaling
• Store the above token bucket in a database, like mongo or dynamo, but it would be very inefficient
• Use redis, but I would have to spin up and maintain an additional dependency just for this feature

Can you think of a better approach?

At work I maintain a NestJS microservice which is used by many other engineers. One of the features depends on a third party API for which I need to implement rate limit:

• The API has a soft limit of 20 req/s
• There is no way to programmatically monitor this limit on their end
• Surpassing the limit means that when someone looks at the dashboard then they will manually disable us, and we have to start another manual process to unblock us
• This API is owned by the government, so we can't ask or hope for changes

How would you implement rate limit for this external dependency? Here's what I thought:

• Have a token bucket limiter inside each service instance, but then scaling
• Store the above token bucket in a database, like mongo or dynamo, but it would be very inefficient
• Use redis, but I would have to spin up and maintain an additional dependency just for this feature

Can you think of a better approach?

At the moment I work for a state funded company that turned out to be very political. My role is to design cryptographic protocols and writing libraries for p2p authentication and authorization, applied to the renewable grid.

To put things simple, there are two teams:

• Team A, made by the high performers
• Team B, where I work, with the people performing not as well

A few months ago a big mess happened:

• Most PMs and engineers make HEAVY use of LLMs
• People read neither the tickets nor the code, which means we have a high hallucination rate
• On my side I always try to read and understands, and I consistently write RFCs/ADRs to document the work
• When team A tried to use my library a huge chaos happened because the interfaces and the flows were incompatible

I got heavily criticized by team A, and my reaction was to point towards the RFCs/ADRs, but also to take blame so we could focus on finding a way forward.

Team A manager complained a LOT about me, while my manager defended me with great energy. Also the PM in my team is pissed at me because all the hallucinations in her tickets finally came to evidence.

Now team A manager approached me in private and proposed me to join their team, but my manager really dislike him and is pissed about having mostly not-so-good engineers to work with. I'm afraid I would hurt him if I were to switch teams, and I'm also afraid the manager of team A is not a nice person. But on the other hand joining them would be a defacto promotion.

What would you do in my place?

For fun an profit I'm building a microvm-like serverless environment, using webassembly. Basically my demo looks like this:

• Layer 4 load balancer written in rust + io_uring, owning the public sockets
• wasm runtime to run containers
• The load balancer scale up and down replicas based on load

Now I'm trying to mitigate cold start in the scale to zero scenario. Let's say each load balancer is owned by just one tenant, and each microservice in the load balancer needs to call a given third party HTTP API very often. Instead of opening the connection anew with each container, I could have the load balancer manage a pool of open HTTPs connections that are kept alive, so containers don't have to open a new socket on each cold start.

• Does this approach makes sense? What could be the blockers?
• Can this approach be generalized to other protocols, like the postgres protocol? How?
• Can this approach be generalized to layer 2, to recycle TCP/TLS connections? How?

For fun an profit I'm building a microvm-like serverless environment, using webassembly. Basically my demo looks like this:

• Layer 4 load balancer written in rust + io_uring, owning the public sockets
• wasm runtime to run containers
• The load balancer scale up and down replicas based on load

Now I'm trying to mitigate cold start in the scale to zero scenario. Let's say each load balancer is owned by just one tenant, and each microservice in the load balancer needs to call a given third party HTTP API very often. Instead of opening the connection anew with each container, I could have the load balancer manage a pool of open HTTPs connections that are kept alive, so containers don't have to open a new socket on each cold start.

• Does this approach makes sense? What could be the blockers?
• Can this approach be generalized to other protocols, like the postgres protocol? How?
• Can this approach be generalized to layer 2, to recycle TCP/TLS connections? How?

I'm working on finishing a paper on high velocity distributed datastores, but I'm struggling a bit to find an optimal solution for a class of problems, and hence why I'm asking for help here. I remember reading that this problem has been solved at twitter, but not with a general solution, and I cannot find the relative paper anymore.

The problem:

Let's say we have an dynamo like store with a lot of objects (1 trillion?), and we have lots of users (1 billion?) who want to react to changes to the store. Each user could be interested in watching a few objects or many (a thousand? a million?).

My question:

What design could be used to solve this problem efficiently? Anyone has sources to link?

Possible solutions:

Naive solution:

The naive solution is for each user to periodically request the objects again. Very expensive, very wasteful.

List of subscribers

Another solution is to store a list of users interested in changes for a given object. The problems with this solution are:

• Doesn't scale. One object could potentially have millions of subscribers
• Not live: a user could be interested in changes to an object, but he might be offline and hence can't receive updates

Compare and fetch with merkle trees:

Like in the naive solution each user could maintain a list of objects he's interested into, and periodically scan for changes, but instead of requiring each object each time he could send a list of object he's watching together with an hash (etag in dynamo terms) of the last content he witnessed. The server then compare the hash (etag?) and either answer with a "no changes" or send the updated content. Merkle trees would make this efficient by allowing to compare hundreds of object at once.

Note: This thread was cross posted on experienced devs