Tezos upgrade analysis — Carthage

by Oksana Protsukha

The clock is ticking before the Tezos blockchain upgrades itself to the next protocol Delphi. Mark the date: Nov 12th, 2020 12:28 pm UTC. The details on the proposal can be found here.

As part of our preparedness for the upgrade we took a comprehensive look at the impact of the previous upgrade to Carthage had on Tezos network performance and Tezos baking community. Carthage upgrade was executed on Mar 5th, 2020 04:38 am UTC before cycle 208.

We aimed to answer the following questions:

1. Were bakers ready for the upgrade?
2. What was the impact of the failures to upgrade on-time on the network?
3. How long did it take bakers to recover following the initial upgrade failures?
4. Were public bakers more successful in their upgrade efforts than solo bakers?

Summary

• The findings suggest that the upgrade was smooth for 75.6% of the Tezos baking community, while 24.4% were negatively impacted by the upgrade.
• As expected, public bakers were on top of things while some smaller bakers and few outliers among large solo bakers were significantly impacted.
• In a few instances it took a baker more than 5 cycles after the update to recover a bakery. Few bakers came back online several months later or were eventually shut down.

How the data was collected:

• TzStats API was used to retrieve most of the data needed for the analysis.
• TzKT API was used to retrieve public/private attributes for individual bakers.
• Anyone can replicate and rerun the analysis. Link to the Jupyter notebook.

We applied the following assumptions to the analysis:

1. Bakers that failed to endorse more than two slots and did not recover from the failure within two hours of the upgrade were considered as impacted by the upgrade.
2. Validators who had an average reliability of less than 20% during the four cycles before the upgrade were excluded from the analysis of impacted bakers.
3. Active bakers were calculated as the total number of bakers who had rights in the cycle immediately after the upgrade (cycle 208)
4. The analysis only considered data for the four cycles before the upgrade and five cycles after the upgrade.

Were bakers ready for the upgrade?

Out of 435 bakers who had rights in cycle 208, 329 bakers (75.6%) were ready for the upgrade and performed as expected; 106 bakers (24.4%) were insufficiently ready for the upgrade and experienced continuous failures in operations in one or more cycles after the upgrade.

What was the impact of the failures to upgrade on-time on the network?

Lost block priorities and endorsements due to lack in upgrade readiness account for missed income of more than 10,000 tez in the cycle that followed immediately after the upgrade. Additional income was lost due to the slow response from the bakers who failed to recover their operations for more than 1 cycle. More importantly, failures to upgrade lead to missed blocks and endorsements, which directly contribute to the decrease in the network speed and security.

How long did it take bakers to recover following the initial upgrade failures?

Out of all impacted bakers, 50% recovered within one cycle. Surprisingly, a high number of bakers — 24.5% of the total impacted — took longer than 5 cycles (over 15 days) to recover and some eventually became deactivated. One fourth of all impacted bakers recovered their operations within 2 to 5 cycles after the upgrade. The total share managed by impacted bakers accounted for 12% of the total tez supply.

These findings suggest that while the majority of the baking community was well prepared for the Carthage upgrade, it still took a significant number of bakers by surprise. It is likely that the impacted bakers either:

• were not aware of the upgrade,
• lacked sufficient knowledge to upgrade their operations,
• lacked time to upgrade their operations.

Were public bakers more successful in their upgrade efforts than solo bakers?

Overall, public bakers with a higher number of delegators either smoothly switched over their operations to Carthage or were able to upgrade their operations within the first cycle after the upgrade had happened. This is expected as these bakers have a lot at stake and are watched closely by their delegators. What was not expected was to see solo bakers with a large number of tokens failing to upgrade on time.

Note: bakers who took longer than 15 days (5 cycles) to recover are excluded from the chart.

So, what to do to successfully switch over to the next version of the Tezos Protocol?

1. Make sure you know when the next upgrade will go live. The details on each protocol version are available on TzStats under voting section. Stay involved in protocol development and vote.
2. If you use the Tezos Suite from MIDL.dev, upgrade your code to the latest Tezos Suite v1.2. The latest changes support running two protocol versions simultaneously and can be turned on/off by passing the protocol version as a parameter.
3. If you are using Kiln, make sure to upgrade to the latest version.
4. If you manage baking operations yourself, upgrade to version 7.4. This version has baker and endorser binaries for both Carthage and Delphi. Run them both side-by-side. You may also switch over manually when the protocol upgrade happens.

If you are a delegator, check that your baker has upgraded to version 7.4. It is visible on TzStats under the “baker” tab.

Conclusion

Tezos is the most successful experiment in on-chain governance to date. However, it only works if bakers are sufficiently engaged.

This analysis aimed to create a meaningful benchmark for the future upgrades of the Tezos network. We will follow-up with the analysis of the Delphi upgrade after it has been running in production for a few cycles. We surely hope that the baking community is more ready for the upcoming upgrade.

MIDL.dev is a cryptocurrency staking-as-a-service company. We provide a non-custodial solution to help you spin your baker on our secure and scalable cloud infrastructure platform with full monitoring.