We host in a SOC-2 certified IDC with 2N power and cooling, physically redundant fiber, biometric security. Our servers are in a locked private cabinet, to which we have keyed 24/7 access.
We operate servers that are always provisioned with redundant power and storage so that single points of failure are minimized.
Our network infrastructure includes diverse paths in the IDC infrastructure so that a single power or network failure in the facility will not bring our servers offline.
We operate validator / baking nodes on hardware servers in our private network. We operate an infrastructure of public facing sentry nodes across a number of cloud providers, including Google Cloud, AWS, Azure, and no less than 3 smaller providers. Our validator / baking nodes have no connectivity to public blockchain nodes.
Our private network in the IDC is fully blocked from the public internet, with no inbound traffic of any kind. Connectivity is mediated by VPNs and private links between our private network and our public cloud networks.
We implement a sentry architecture for networks like Cosmos and Tezos, and variants of this architecture for other networks as technical details permit.
Tezos presents unique challenges for validators who need to bake securely and manage risks appropriately. At the moment there are two common methods of storing the private keys that are required for baking: encrypted on disk or using a Ledger Nano-S hardware wallet.
Neither method is entirely satisfactory. Storing keys on disk, even encrypted, is not an acceptable option, for obvious reasons. The Ledger Nano-S is excellent consumer hardware, but it is not intended for continuous server operation. Nonetheless, it is the best available option, so it is what we use for the time being, with a process to mitigate the hardware’s weaknesses.
The Tezos baking node has a Ledger Nano-S attached to a USB port. That Ledger holds the private key of our Baker, and has the Tezos Baking app installed. In order for the node to bake, this Ledger needs to be unlocked with a PIN. This PIN is stored encrypted and separate from the servers and is only decrypted in the secure facility. The PIN is long enough to make it hard to remember and is rotated frequently so that the user does not memorize it. The Ledger needs to be unlocked with the PIN in order to bake, and subsequently only needs to have the PIN re-entered if it is power cycled. This requires a visit to the facility, which is inconvenient, but since PINs and keys never move across any remote access facility it does enhance security. As the Tezos platform matures, we expect that enterprise-level custody solutions will emerge, and enterprise HSMs will replace the Ledgers.
In addition to the security of the private keys, the Ledger guards against the possibility of double baking. A double bake occurs when the same private key is used to sign two versions of the same block. This risk is managed by the Ledger, as the Ledger will only sign a given block once, and will refuse to re-sign it. The only way to double bake using Ledger is to have two connected and active at the same time. Some bakers do this in order to mitigate the risk of downtime. Since the liveness penalties in Tezos are very small compared to the double sign penalty, we maintain only a single Ledger connected and active. We have a standby server active, in-sync and ready to bake in case of a failure of the primary, but it requires a visit to the IDC to move the Ledger, or replace the Ledger in the event that it is the failure point. The small loss of revenue that we risk in the event of missed bakes is accepted as a business risk.
At this time we operate physical servers in a single facility. We will be establishing a second facility as a warm disaster recovery site, but for the planned future we will maintain the policy of requiring hands-on intervention to move baking from a failed primary to a standby facility.
The baking Ledger only has the baking app installed on it. Even if this hardware was accessed by an unauthorized person in the IDC, it is not capable of spending/sending transactions. The baking operation receives and needs to distribute rewards to delegators. This process is carried out using a second Ledger Nano-S, which is replicated and holds the same keys as the baking device, however this device has the Tezos Wallet application installed. This Ledger is stored in a safe deposit box at a bank. In order to access funds a physical visit is required, and transactions are executed using this Ledger without leaving the bank. This Ledger never leaves the bank. We make bulk transfers of the XTZ that we anticipate needing for 2 weeks of payouts from this Ledger to a float account, which is managed on a third Ledger wallet.
The float wallet is managed without the inconvenient security measures of the two baking account wallets. It is still highly secure, but it is in the possession of an individual responsible for cycle by cycle payouts. In the future, we may implement a hot wallet system to facilitate fully automated payouts. Minimal funds are held in the float wallet, so normal security measures for storing and handling cryptocurrency suffice.