Building fully on-chain games is a lot fun, and often challenging. Recently been seeing an issue on @PirateNationNFT causing some tx's to fail. Just diagnosed, and deployed a fix. Sharing what we learned and built to resolve:

You can see these transactions at

Previously, we were doing some basic load balancing between the relayers, but sometimes we saw a relayer get stuck for various reasons (in this case, it was retrying transactions with nonces out of order).

This got us into a state where we would have a partial outage Some transactions would go to healthy relayers and be fine Some transactions would go to the unhealthy relayer and fail. Not great.

We were able to pull this Relayer out of the rotation manually with oncall, but it cost us time and a few late nights - we saw this happening at 1-3am sometimes. This also takes time for a human to be paged, respond, get online, understand the problem then take action.

So, we put in a change that now checks how many transactions are pending per relayer, and will route new transactions to the relayer with lowest pending tx. This in theory should allow any relayers that are stuck with nonce issues to self-heal.

So let's test it. We saw a relayer starting to have the issue, so we deployed the fix. After deploying observed that the failed relayer was pulled out of rotation, and then put back into rotation once its pending transactions were cleared up - in record time! Success!

All in all, a fire drill that resulted in a new, interesting piece of tech, and now a better experience for our players. I'm going to share more behind-the-scenes stuff like this as we go. Drop me a follow / feel free to share if you found this useful!