Ossification (announcement and call for feedback)

Methodology & Framework
1

Ossification

in which we attempt to quantify an ancient blockchain security primitive of permissionless, immutable smart contract code

We are working on a new L2BEAT metric: the Ossification Score.

The frontend and backend machinery is still being assembled, while the methodology and dataset already exist. So instead of waiting until the feature is shiny and emotionally harder for us to change, we are posting it here first.

Please attack it.

Dataset: https://sekuba.github.io/ossification-dataset/
Repo: https://github.com/sekuba/ossification-dataset

Old code gud?

There is some evidence from traditional software that old, unchanged code tends to be less vulnerable, while fresh and heavily modified code tends to contain more bugs (1, 2, 3, 4).

This seems especially interesting for smart contracts: they are public, attackers can stare at them early and indefinitely, and successful staring can occasionally produce several hundred million dollars. So we define code ossification as the accumulation of evidence that code has matured by successfully securing value over time (not automatically: old code = secure).

The :bone:-clock

L2BEAT already monitors project contracts and their state with our discovery system. For ossification, we define a perimeter of critical contracts and configuration. Every time something critical changes, the project’s :bone:-clock resets.

code age = time since the last critical change

We did not want to declare that 2 years = 83 score because it felt spiritually correct, so we collected exploits instead.

The score

We started, together with a robot swarm, an open dataset of EVM blockchain exploits and reviewed, among other things:

  • what was exploited,
  • how old the vulnerable code/configuration was,
  • how much money was lost.

That gives us an empirical distribution of code age at exploit time. The Ossification Score is derived from this distribution. For example:

A score of 67 means that 67% of exploits above $1k loss targeted younger code.

You can play with the current dataset and thresholds here:

:backhand_index_pointing_right: https://sekuba.github.io/ossification-dataset/

But there is $10 in this contract

Three-year-old code securing $50k has survived a somewhat different adversarial environment than three-year-old code securing $10B. So we add another metric: battle-tested exposure.

This is the integral of TVS since the last critical change, measured in dollar-years. If a contract secures $1B unchanged for one year, that is roughly $1B-year of exposure. Very advanced physics. A project like Arbitrum or Base can have relatively young code while already having accumulated enormous economic exposure.

We currently prefer showing these things next to each other rather than mathing them into One Final Security Number:

  • code age,
  • ossification score,
  • battle-tested exposure.

Ossification = security confirmed?

No, the metric does not tell you:

  • who can upgrade the contracts (check L2BEAT),
  • whether users have an exit window (check L2BEAT),
  • whether the code was audited (working on that!),
  • how large the last code change was (working on that!),
  • whether someone will lose the multisig keys in a boating accident,
  • basically anything outside the onchain code/configuration.

Why?

Very few blockchain systems are actually ossified today. That means there is something to incentivize.

Just as Stages make decentralization progress visible, we would like ossification to make another property visible: building systems that eventually become simple, robust, permissionless and boring enough that nobody needs to keep upgrading them forever.

Contribute

Before we ship, we are especially interested in attacks on:

  • the dataset: missing exploits? bad classifications?
  • the clock: what exactly should reset it?
  • the perimeter: what counts as critical code/configuration?
  • the score: is this percentile-style construction actually useful?
  • battle-tested exposure: what do you think about dollar-years?
  • obvious ways people will misinterpret this: find them before CT does.

Dataset again: https://sekuba.github.io/ossification-dataset/
Repo (open for PRs/issues): https://github.com/sekuba/ossification-dataset

5 Likes
2

here’s a preview of what the ossification summary page could look like

image
image2328Ă—1909 401 KB

3

I like the idea and the thesis makes sense to me. Have you considered treating contract storage changes (config update) like code changes?

1 Like
4

yes, we already do! our disco update monitor can watch storage as well as public view funcs and both can be labeled as critical changes. some examples are pointers to verifier contracts, program hashes, quorum fraction/threshold changes,…

5

Have some feedback (I used L2BEAT TVS data for the below) :

Battle-tested Exposure
Battle-tested exposure is the integral of TVS since the last critical change: E = integral of TVS(t) dt, measured in dollar-years. Two properties make it fragile: the integrand is linear (every dollar counts equally, every day), and the input is TVS which can be easily gamed. I give Blast as an example:

Blast accrued an incredible amount of TVS very quickly and has a 99% drawdown from its peak, and it was just announced that it will wind down. Here are the dollar years it accrued:

Period Dollar-years accrued
Pre-mainnet only (101 days, chain did not exist) 0.29 $B-yr
Mainnet to end of 2024 1.81 $B-yr
Total, Nov 2023 to Oct 2026 2.55 $B-yr

Compare Blast to Optimism: Accrued 15.65 $B-year after 5.7 years of operation. Blast accrued 16% of Optimism’s lifetime battle-tested exposure in under three years, way before the chain existed, on new code that was behind a 3/5 multisig.

How to fix it (in Bens opinion)

The linear fails at the bottom and the top end of the TVS chart. If you have $50k sitting untouched for 10 years or extra billions added to a top chain - both gather exposure but only one is a real target.

I would try to do capped dollar-years

E = integral of min(TVS(t), $1B) dt. / Dollar-years secured, counting at most $1B per day.

I think having a global cap that is identical for each projects lets us directly compare dollar-years better. I chose $1B by checking the largest single event extractions, and p99 is $121M / Max $611M. No attacker has been able to hack $1B (yet), and any TVS above that hasn’t added any adversial information. It is similar to bounty economics where you have X% of funds at risk that are capped to $XM. I think the cap would need to be rederived based on the loss distribution.

Here is an example:

Measure Blast Optimism Arbitrum Polygon Ethereum
Operating span 2.9y (0.3y pre-mainnet) 5.7y 5.4y 6.4y 9.0y
Naive: integral of TVS dt 2.55 $B-yr 15.65 52.02 23.84 300.31
Capped at $1B 1.52 4.62 5.07 5.49 7.10
Concave alternative: integral of log2(1 + TVS/$100M) x $100M dt 0.75 2.35 3.24 3.00 5.84

I think the cap here works quite well, Blast drops to 33% of Optimisms exposure, and Ethereum is compressed to 7.1. At a certain point gathering more dollars does not lead to more evidence and this smooths output to be comparable.

I think for additonal context you could add days above $XM / $XB, but this wouldn’t impact the score.

Chain Days above $100M Days above $1B Days above $10B
Blast 897 375 0
Optimism 1923 1485 1
Arbitrum 1853 1851 944
Polygon 2044 1988 0
Ethereum 2833 2307 2159

The missing denominator (dollars lost per dollar-year lived)
The ossification score is the share of historical exploits that hit code younger than yours: P(age | exploited). I think a better score would be P(exploited | age). It would be good for a user to know: how much value-time was lived at each code age across all projects, exploited or not.

Without the denominator, even the direction of the age-risk relationship is unidentified. The dataset gives the numerator:

Code age at exploit Exploits Dollars lost
< 30 days 276 (47%) $968M (28%)
30 days to 1 year 196 (34%) $2,155M (63%)
1 to 2 years 32 (5%) $184M (5%)
2+ years 81 (14%) $134M (4%)

Looking at the shares above, younger code looks incredibly dangerous, but the shares depend on how much code existed. In 2020 to 2022, 96% of exploits were on code that was under a year old, as DeFi TVL exploded from $1B to $180B at its peak, meaning the large majority of dollar-years lived were also in code under a year old. If over 96% of exposure was young, young code was under-represented among exploits per dollar-year.

How to fix it (in Bens opinion)

  1. Build the exposure panel. For each perimeter-instance i (a project’s critical contracts between two resets), born at its last critical change t0_i and exiting at exploit time or censored today, accumulate exposure per age bucket: E(a) = sum over i of the integral of TVS_i(t) x 1[age_i(t) in bucket a] dt.
  2. Report the loss rate per bucket: h(a) = L(a) / E(a), dollars lost per dollar-year lived, with Poisson confidence intervals.
  3. Validate against the dataset itself. Add TVS-at-exploit-time to each incident record and check whether h(a) actually falls with age.
6

hey thanks for the feedback! we thought about this battle-tested exposure a lot, and e.g. AI always suggests way more complicated curves too, but it felt like this metric is already quite far out in a way, and not using the simplest, kind of physically intuitive function for it would make it even less useful.

blast would also very likely not have been shown on our ossification page, especially not during the pre-farming, because ossification does not mean anything for projects that are not at least permission-reduced or aiming at permissionlessness. which projects we opt into ossification will be purely editorial

7

Thanks for the thoughtful analysis of our ossification project!

To comment on your second proposal, this is definitely a shortcoming of our current metric. Ossification score does not tell you how likely a particular protocol to be hacked (statistically), because it doesn’t include data on all the protocols that were not hacked. It tells you rather “if the protocol is ever hacked, when is it most likely to be hacked?”, which is a different thing.

We are limited by the data at our disposal. To compute what you propose, we would need to track:

  • all projects that could theoretically be hacked and end up in the exploit database
  • their TVS changes over time
  • their security perimeters and critical upgrade timestamps.

It’s beyond our reach, unfortunately, so we settled on a simpler yet feasible metric. Would be cool if someone gathered all the data though!