Storage staking price

[illia]

Join the open discussion on Feb 1

We are going to have an open discussion on Feb 1st 3pm UTC (7am PT), to join the call add this Google Calendar invite: Google Calendar - Sign in to Access & Edit Your Schedule

It will also be streamed on Youtube and you can also ask questions / comment there: Decision Discussion: Reducing Storage Cost - YouTube

Join the call if you have a strong opinion, watch stream if you just want to be informed.

TLDR

This is opening a discussion to reduce storage staking price per byte from 1E20 yN to 1E19 yN based on adjusting the assumptions from initial calculations.

Additional ideas for addressing experience issues are also proposed for discussion, like burning storage cost and including it into gas instead of as a separate cost.

Parameters of the decision:

• Proposed: Jan 18
• Decision deadline: Feb 1
• Parties whose input should be included: active developers, validators and token holders with economic background or active position on economics.

Details

Problem

Currently the price of the storage becoming unrealistic for storing larger contracts. Rust contracts that take 300kb would require 30N, which at price of 2.5 $USD 2.5 per $NEAR is 75 $USD. Clearly this is already very expensive and directly correlates with any potential price increase.

Adding extra key (e.g. NEAR Wallet “log in”) to the account is ~100 bytes, currently 0.01N. Compare this with tx fee - 0.00004N.

Similarly, things like storing information about user’s token holding, currently can take 168 bytes or 0.0168N.

Developer side, it’s also means that all these operations need to contain extra payment included in the call to other contracts. Which makes the development more complicated. The exact cost of storage may be unclear from the caller perspective. The balance on the account is now also needs to be split between “active” and “used for storage” (in addition to “staked”).

Another point, is that even though NEAR is “staked” for storage, for the most cases it’s not expected that storage will be deleted. E.g. accounts, contracts and user’s account info inside the other contracts. E.g. the amount “staked” is almost never returned.

Historical context

State cost is one of the harder problems in blockchains.

Storage staking came in early 2020 as an alternative to storage rent that was implemented in NEAR protocol. Storage rent was highly unintuitive for users and complex to support in the tooling. Contracts potentially could run out of payment for storage and become deletable. Same for user’s accounts, which user potentially could loose an account if didn’t have keep enough or didn’t top off at the right time. Even worse, someone potentially could re-create it and claim all the assets that this user had in other contracts.

Ethereum uses a form of storage staking & burning, where gas gets burned when things get stored and 75% of gas gets released when data gets deleted. The important piece here is that it’s valuing storage in gas instead of in the base currency. But Ethereum doesn’t have any long term mechanics alignment for storage with storage size growing linearly for past years to over 100 GB.

One of the ideas in Ethereum, is to switch to stateless validation, which indeed provides a convenient way to avoid state storage problem for miners / validators… But really just shifts the problem to whoever will be composing blocks, requiring them to have a separate economic model to store data. One idea there is that block composers are professional “frontrunners” (MEV) and hence extract value by forming blocks. Generally it allows a smaller subset of full nodes to maintain data and set of application specific nodes that contain data for specific applications.

Storage staking cost

The assumptions were made for calculating the cost of storage to account for rate of replication R and current cost C in $USD of the active cloud storage of 1 byte per year. Exchange rate E of $USD to $N was used.

The model used was:

• Given network must store 1 byte for next year, the validators must be paid X $N to store it.
• If $N is “locked” inside the account for this year, it pretty much “generates” inflation on it (as alternatively it could have been delegated and earn the rewards instead), which is global parameter I.

E.g I * X = R * C * E

Parameters used:

• R = 100 (nodes per shard)
• C = 1.12E-9` (High-Performance Block Storage– Amazon EBS Pricing – Amazon Web Services gp2 is $0.1 GB/mo)
• E = 1
• I = 0.05

Solving for X => 0.05 * X = 100 * 1 * 1.12E-9 => X = 2.24E-6 or 2.24E19 yN.
Given there is pretty large overhead for storing state data (merkle tree, various meta data, etc), it was rounded up to 1E20 yN.

Proposals

This is still TODO to figure out all the details based on the feedback and information.

Proposal 1

Description: reduce cost by 10x
Impact: minimal
Implementation time: couple of weeks
Reasoning: current costs are too high for users and developers

Proposal 2

Description: reduce cost by 100x
Impact: potentially people filling in cheaper storage and then release it later to get back NEAR
Implementation time: couple of weeks
Reasoning: current costs are too high for users and developers

Proposal 3

Description: reduce cost by 100x and burn the storage cost, including it in gas spent
Impact: reduces complexity of building applications, reduces complexity of frotnends like wallets and explorer by removing “minimum balance” concept; there is no motivation to release storage.
Implementation: 1-2 months
Reasoning: current costs are too high for users and developers; developer experience suffers from needing to send storage $NEAR around; people are not releasing storage anyway which leads to the $NEAR effectively being burnt

This proposal can be split into one step to reduce costs in short term and second step to include storage cost into gas which will take more to implement.

The storage cost can be measured still in $NEAR or in gas. If it’s measured in $NEAR, the charging will only charge amount of gas to cover storage cost (e.g. gas_spent = bytes_added * storage_byte_cost / gas_cost).

Potentially, gas can be returned to the transaction which releases storage.

This is pretty much Ethereum model. It’s known limitations are mainly around lack of motivation to release storage and issues around high gas prices to use “storage” as reserve of cheaper gas. The problem with reserving gas can be minimized by using formula above where charging happens at fixed $NEAR cost instead of current gas cost.

Motivation around releasing storage seems missing in general. It seems like a more advanced technical solution is needed to handle growing size of the active state in the blockchain. For example, stateless validation mentioned in “Historical Context”.

[hodlr]

I like where this is going: Include storage cost into gas spent, as it finally forces developers to consider storage per-interaction, not just a bulk “ill pay for it overall”. It also means contracts can decide to keep or throw away storage per function call, say if too much storage was getting used beyond what was being paid extra.
Seems like there needs to be some simulations to the economics here tho.
It would be possible to offload some of this to a secondary market, similar to gas station network?

[evgenykuzyakov]

To comment on this. I’d love to see 10X drop in storage pricing on-chain. But eventually we’ll solve replicated contracts as a protocol upgrade by making it free to deploy an instance of a contract from the global registry.

Also we’re working on no_std support for near-sdk for Rust, which should bring contract sizes down.

[illia]

(Alternative storage charging model considerations rambling)

I think one important question for blockchain state storage that I’m having trouble resolving: do people expect to remove things from blockchain state?

Current model expects that applications and users actually will be cleaning up storage when it’s not needed. At the same time we currently the state where people don’t want to delete anything. For example, people asked specifically in SputnikDAO to keep all proposals in the state. This data problem potentially can be fixed by having available historical data around in queryable form.

But general fact stands: contracts are deployed forever, data stored in them mostly as well; users don’t expect to remove their accounts; etc.

If we approach from perspective that data will be only added to the state, we actually get into different state where burning $N (or gas) for storage makes more sense. This benefits validators in the future by reducing the token supply. Also this would clean up interfaces, because would remove need to keep track of “active” balance vs full balance. This is closer to current Ethereum approach (except instead of burning gas for storage goes to current miner that included).

[crasskitty]

Big fan of reducing storage costs.

That’s all

[alex.shevchenko]

Thanks for bringing this on the table @Illia!

I do believe there should be some simulations done to adjust the model, so it would be hard to make a final decision on 1st of February. Also, validators should be well-informed of the changes in the tokenomics, so we won’t end up in the clash like with EIP-1559.

I’m supporting the idea of burning $N / gas for the storage as it will make UX much better.

I suggest to do now an intermediary step of reducing the storage cost 10 times (pretty fast), but continue this discussion and figure out how we should change the storage acoounting.

Also, please bear in mind that we need it to be consistent with EVM Runtime to not introduce kludges like ‘let’s not account for EVM storage, since it has different storage economics’. Probably we would like to make this uniform.

[nearmax]

Similarly to other parameters in Runtime, storage cost can only be reduced. If we ever increase storage cost, some factory contracts might stop working. Therefore I would be maximally conservative, e.g settle with 2E19 yN price first.

However, I am strongly against changing economic model, like burning gas instead of staking. We have sufficient experience in the past discovering how storage models can have unforeseen consequences on DevX/UX or create complex game-theoretic dynamics, like chi token on Ethereum. The only way to know about all second-order effects is to run storage model sufficiently long in production, and we had only two models sufficiently long in production: storage staking and storage rent.

[oysterpack]

Given the removal is not expected, burn the $NEAR for storage.

Does 30% of the burnt $NEAR go towards the contract reward? The answer affects my question below. Based on the example outlined below, I would conclude that it can’t be, but I want to confirm what is being proposed.

If data does get removed, $NEAR can be returned back at current storage cost

To be clear - who earns the refund for storage deallocation, i.e., deleting data?

• Is it the account that paid for the storage allocation? If yes (and I think it should be), how would you track which storage was allocated by which account generically? I don’t think this option is feasible because the accounting would add much complexity and overhead.
• Is it the account that issued the transaction that resulted in data being deleted?
  • if yes, then how would this work for transient / short term storage? I can think of many use cases for transient / short term storage. How does this exactly work with contract rewards and refunds? Let’s use a message queue smart contract and simple round numbers to illustrate:

ASSUME: Contract is rewarded 30% of storage costs

1. message producer pays 100 NEAR for storage allocation
2. contract earns 30% of that - which means 30 NEAR is deposited to contract balance and effectively 70 NEAR is burnt
3. message consumer deallocates storage that is currently priced at 100 NEAR - what happens here?
   • if the storage cost is refunded to the message consumer, then the model is broken. Message consumers would be rewarded NEAR for deleting data, which breaks the economic model for the message producer. Ideally, the refund should be sent to the account that allocated the storage, in this case it would be the message producer. With the current storage staking model, this can be designed to work correctly today because the contract can collect the storage fees and escrow the storage fees for the message producer. As message consumers result in deleting data, then the contract can release funds from the message producer escrow. How would this be supported in the proposed model?
   • how would storage refunds play with contract rewards? If the contract is rewarded storage allocation costs, then even if the refund would be sent back to the message producer, the message producer effectively paid 30% of the long term cost for short term “rental” space. Based on this, I would conclude when the NEAR is burned for storage allocation, that it cannot be included in the contract reward because it breaks the economics.

• Decision deadline: Jan 29

This would be a huge paradigm shift and radically changes the economic model. The proposed deadline to make such a huge decision that impacts the entire community and long term future economics is unrealistic and concerning from a governance perspective.

[illia]

Update the post with the call information and separated proposals.

@evgenykuzyakov

But eventually we’ll solve replicated contracts as a protocol upgrade by making it free to deploy an instance of a contract from the global registry.

Yes, this will help for sure. But currently the account creation and adding keys costs are pretty high as well. I think the fact that developers are not expecting that this cost will be returned to them is somewhat established shows that initial assumption of more malleable state is not valid.

@nearmax

I am strongly against changing economic model, like burning gas instead of staking.

I mentioned in the Proposal 3 section that with specific burning based on fixed $NEAR cost per byte we can actually avoid pitfalls of Ethereum. That said Ethereum’s model works pretty well for developers as it’s not really in their face. And opens up an easy way to later reduce costs for storage by going to stateless verification or whatever other approach.

@oysterpack

Does 30% of the burnt $NEAR go towards the contract reward?

I don’t think it should.

@oysterpack

who earns the refund for storage deallocation

Right now actually it belongs to the contract. So developer can decide who to return it.
In the model of burning gas for storage, it will be returned in the transaction that releases the space, similar to Ethereum’s approach. It will also be only the amount of actual $NEAR spent for storing these bytes added back in the form of gas (through refund will return it to balance of the user).

@oysterpack

transient / short term storage

In this case indeed the storage is refunded to consumer of the storage.

That said using blockchain storage for message queues seems highly suboptimal. One can use the blockchain transaction data for that with way better cost efficiency and scalability.

[evgenykuzyakov]

Proposal 0 - no changes. With the current pricing, it’s also not as bad. Fungible token is in works to account for account registration, the funds are recoverable if you don’t need them anymore.
The problem is large contract sizes, which we have to resolve because it’s not only storage, but also execution cost of them. The larger the binary the slower the initiation. This can be address by optimizing Rust builds.
For repetitive contracts we should instead focus on contract reusability to allow spawn free instances of such contracts.

Proposal 3 is a huge change to the entire system, it’s unrealistic to think we should select it.

Proposal 1 is my favorite so far, due to NEAR cost increase we can account for a cheaper storage by keeping the current economics model.

Proposal 2 seems to be an overkill and creates the dangerous long-term effect if someone want’s to break the system. It wouldn’t be easy to revert such decision.

[evgenykuzyakov]

During the call we’ve discussed the possibility to drop storage by 20X instead of 10X as it was mentioned in the proposal.

I vote against 20X, since it was not discussed in the proposal before and was briefly mentioned at the end of the call. We’ve also didn’t finalize the decision to go with 20X instead of 10X.

If anyone wants to see 20X instead of 10X, please provide strong arguments why 20X is better than 10X. The question is about 100Tb of state that will be added for the full supply if we go with 20X vs 10X.

Summary:

• 10X reduction brings maximum state to 100Tb from 10Tb
• 20X reduction brings maximum state to 200Tb from 10Tb

20X discussion in the video: https://youtu.be/Ge1Cfq-0y5w?t=3860

[zavodil]

Agree, I would like to see more math before to switch from 10X to 20X.

[FreshNEARs]

10X, but in parallel I’m interested in hearing on possible solutions to reduce state size. Pruning? Rent?

[Lastblock]

I vote for 10x, introducing the cheap storage may also introduce unexpected problems.

[eatmore]

I vote against 20X. We should encourage efficient use of resources, including storage.

[nategeier]

I lean on @evgenykuzyakov for security concerns or anyone else deeper on the topic. I’m not sure about shard capacity or how ready we are to go to a new shard if the storage gets maxed out and don’t have the time to research it. However, seems @illia doesn’t think the extra storage is that big of a deal so of course, I’ll always vote on cheaper if secured. If storage really is an issue, then that means sharding is an issue right? Mintbase needs a drastic move in storage costs as deploying a store, which is an ERC721 (nep4) contract now costs around 31 N which is around $90. We moved to NEAR because the store deployment on Ethereum was too insane, was $2 then shot up to now $90-$300 on average.

[evgenykuzyakov]

It’s possible to decrease the storage cost further, but it’s not easy to increase storage price back.

[blaze]

I agree with @evgenykuzyakov that a 10x reduction helps alleviate the current price pressure, along with the fact that @nearmax pointed out that storage costs can only be reduced.

The shared contract state will also help for reusable contracts.

However, these are not likely to be long-term measures by themselves as the price of NEAR continues to increase over time.

Would storing default profile info (name, email, nick, etc) in a shared state help? In this way, each application could avoid storing it again. It seems the more shared contract services that could be stored in a global state would help with storage and composability. Is there a set of general contracts/functions that could be used by all apps?

Providing an interface that would allow end-users to view their storage (indexer) and initiate the contract call to reclaim it, would be a great way to help manage storage. This could be similar to managing Facebook app permission each app would implement an interface to reclaim storage categorized by object types (profile, cache, delete all, etc.). I’m not sure how practical this is. This would be similar to the Tron bandwidth indicator.

[evgenykuzyakov]

Thanks for participating in this discussion.

The 10X storage cost reduction was merged to nearcore/master: feat: Lower storage cost 10X by evgenykuzyakov · Pull Request #3881 · near/nearcore · GitHub

[evgenykuzyakov]

@blaze

Would storing default profile info (name, email, nick, etc) in a shared state help? In this way, each application could avoid storing it again. It seems the more shared contract services that could be stored in a global state would help with storage and composability. Is there a set of general contracts/functions that could be used by all apps?

I think right now we’re mostly trying to solve Rust contract size issue. There are still some challenges with multiple users per contract, so contract has to manage this storage for them. This discussion is happening here: Account Storage Standard (formerly Account Registration) · near/NEPs · Discussion #145 · GitHub