Faster bringup of a Polkadot validator with Polkashots.io

by Nicolas Ochem

When operating a Polkadot validator, it is critical to maximize your uptime in order to avoid being penalized.

Good practices of a node operator incude disaster recovery: the ability to quickly repair your operations in case of failures. For example, if your hard drive dies, you need to recreate your node from scratch. As time passes and the number of historical blocks grow, it can take a prohibitive amount of time to synchronize your node to the head of the chain. During this time, you may be marked offline and lose your rights as a validator.

We are proud to introduce Polkashots.io, a website where you can download recent snapshots of the Polkadot chain, regularly updated and fast to download. This website was developed thanks to a Web3 Foundation grant.

Compressed database snapshots

Substrate has an export-blocks and import-blocks function. These are native commands to package blocks into a file. However, this format has several limitations. We have chosen instead to provide compressed snapshots of the underlying database files.

There are two types of database that Substrate node operators can use: ParityDb and RocksDb:

• ParityDb is the new experimental database format that provides very good performance, and is expected to become the default soon,
• RocksDb is a mature database format, currently the default for Polkadot and all substrate chains, and is less performant than ParityDb.

We provide filesystem-level snapshots for both these database types. They are compressed using the 7z format.

These snapshots are “pruned” meaning they contain just enough data to enable you to validate. Their compressed size is about 2Gb at time of writing. Unlike “archive” nodes, they do not contain every block and extrinsic since chain inception. While it is recommended that you also run archive nodes in order to preserve copies of the chain history, these snapshots are optimized for quickest bringup of a validator node and therefore are not “archive” snapshots.

Don’t trust, verify

As a validator, you must take good custody of the network. Do not just blindly download a snapshot from the Internet.

We make it easy for you to verify authenticity of our snapshots: the webpage mentions the block height and block hash at which the snapshot was taken. Links to Polkascan/Polkastats are provided so you can verify separately that the snapshot contains the true chain history.

Additionally, we provide a sha256 signature for all snapshots that we generate.

How to use

Install and start the polkadot program. It will create the necessary folder structure in ~/.local . In this example, we will be using the ParityDb format:

polkadot --db=paritydb

The node will start syncing from the genesis block, which is not what we want. Terminate the program with Ctrl+C.

We provide links that reliably point to a recent snapshot. To download a recent ParityDb snapshot, do:

wget https://dot.polkashots.io/snapshot -O polkadot.ParityDb.7z

Then, uncompress the file with the 7z utility:

7z x polkadot.ParityDb.7z -o~/.local/share/polkadot/chains/polkadot

It will uncompress the snapshot into ~/.local/share/polkadot/chains/polkadot/paritydb

Start the node again:

polkadot --name "My node" --db=paritydb

The node will continue syncing from where the snapshot engine left off. It should only take a few minutes for it to be fully synced and show as Idle.

You may then kill it again with Ctrl+C and start it as validator with the following parameters:

polkadot --name "My node" --validator --unsafe-pruning --pruning=1000

How it works

The snapshot generator is hosted on Google Kubernetes Engine and uses several cutting-edge features to get the job done:

• volume snapshotting to run the compression job on a “frozen” filesystem state while the node is still running,
• workload identity to regularly start the aforementioned volume snapshotting and compression job with a Kubernetes CronJob

The snapshot webpage is statically built using Jekyll each time a new snapshot is created, and hosted on Firebase.

The snapshot generation infrastructure is deployed using infrastructure-as-code. The entire codebase is open-source.

Who we are

We are MIDL.dev, a staking-as-a-service company. We can help you validate your DOT, and develop custom cloud infrastructure solutions for your parachain project. Get in touch.

We maintain the polkadot-k8s project, which allows you to deploy a Polkadot validator on Google Cloud Platform in just one command. The most recent release will now download a pre-synced database snapshot from polkashots.io, so it takes even less time to get up and running as a cloud validator!