Off-chain computation framework

Development
1

This is general idea for creating off chain computation that has various levels of security.

Problem

Off-chain computation can span from providing external data to on-chain (e.g. price feed, twitter API) to complex computation over on-chain data (e.g. average over transactions in the last 24 hours, merkle root of all users across all contracts).

All these use cases can benefit from security that is provided by validators of the chain, but there are few reasons why this is not possible to achieve directly on chain as part of the protocol:

  • Data provided is external to chain
  • Computation takes too long (seconds to minutes) or too much memory (load the whole tx history)
  • In sharded environment, it may also be also too expensive because it requires lots of cross shard communication

Another use case for such off-chain computation that is still shares the validator set is various bridging.

Proposal

In the generic case, a new worker client is built.

This worker client reads the job description posted on-chain (wasm file to execute, periodicity, where to take arguments from). It then executes this job locally and posts results back on-chain.

There is a jobs contract on-chain, where developers and other contracts that need off-chain jobs to be done can post what needs to be executed. They pay $NEAR or new token for this job to be executed that then gets split among the runners of worker clients.

Now, validators are generally open to provide more utility to the network. They can be running such worker clients.

One trick that’s possible on NEAR is that jobs contract can actually link the chain validators and these worker runners based on who is staking at any moment. This means that for any job executed, you can see how much of the stake executed this job.

Jobs can implement aggregation / challenge system: from simple that super majority needs to agree, median of price with 2 standard deviations, etc. if the validators disagree their stake can be “challenged” by smart contract (this would require staking smart contract modification to support it).

First version of this though can just rely on social consensus around delegations - most validators are leveraging delegations and thus providing invalid data will lead to loosing the delegations and loosing their spot as validator. More advanced version with challenges can be then added as validators migrate to next version of the staking smart contract.

Validator’s job is easy to just run such worker client and register another account or a key in jobs that this worker is operating under (it’s not the same account as the one that does validation to avoid any issues).

More specific version of worker can be developed where it’s required (e.g. wasm is too slow for complex computation or may be some external operations are not supported). Validators adopting workers will register which types of tasks they support.

Extra: Bridge

Bridge can leverage this framework to use the same security of validators to sign with secp256k1 blocks or transactions that must be relayed to other chains.

This ways protocol doesn’t need to directly do this, while majority of validators can be running separate piece of software and with challenge functionality providing the same level of security.

Thoughts?

4 Likes
2

This relies on the assumption that every validator uses some smart contract for staking. That is true for the time being, but it may change in the future. Also it is not clear why we should limit this to validators. It appears that we could allow anyone to stake to the contract for the sole purpose of executing the jobs off-chain with some minimum stake threshold.

3

Jobs can implement aggregation / challenge system: from simple that super majority needs to agree, median of price with 2 standard deviations, etc. if the validators disagree their stake can be “challenged” by smart contract (this would require staking smart contract modification to support it).

I’m thinking to build something similar in order to bring off-chain price data on chain in a decentralized way. Regarding Bowen’s concern I think the most challenging thing is a well-designed economics, so that

  • people are incentive to run the off chain worker
  • there’s reasonable forfeit for malicious off chain workers
  • there is a balance in number of participating off chain workers, so that it’s not too expensive for the caller, but also profitable for off chain worker, and secure enough in consensus sense that result is trustful.

So that Illia proposed see how much total staked worker is execute is a good indication of security, if non validators can participate too, then there needs other means of token locked to ensure safety

4

Also the requirement of the data consensus should be configurable by the caller, whether it is “should be exactly same” or “should be within 1% difference in number”, etc.

5

@illia This worker client reads the job description posted on-chain (wasm file to execute, periodicity, where to take arguments from). It then executes this job locally and posts results back on-chain.

Where would the wasm file be hosted / distributed through? How would one upload it to a Jobs contract, perhaps without external dependencies like IPFS, conveniently referencing by hash, letting anyone check the wasm’s validity by compiling its source instructions.

Speaking of source, does the wasm file describing computation steps for a Job need to be implemented as an SDK contract, or adhere to some alternative requirements to be interpreted properly by the client?

Does the Worker client itself need to inherit from, or even plug into neard somehow? Would a controller process in the daemon need to dispatch the running of Worker, in order to,

benefit from security that is provided by validators of the chain

and if not, then which component would govern the provisioning of “read request from chain, process off-chain, and call back with results”?

To my understanding, Worker may need to sit on top of an Indexer in order to be capable of doing Jobs that require reading an on-chain contract’s entire state. It probably would not be necessary given reads could be chunked into separate Jobs, for instance, each reading one struct from a persistent map at a time rather than reading the entire map at once.

@illia First version of this though can just rely on social consensus around delegations - most validators are leveraging delegations and thus providing invalid data will lead to loosing the delegations and loosing their spot as validator.

In my way of thinking, Jobs would be split according to the owner of the input data they work on, input being structured as { ‘contract-account-name’ ; ‘getter function therein’ ; ‘owner as param into getter’ }.

The Job would be backed by an owner's STAKE, abstracted over by an stNEAR-style basket contract auxiliary to the existing @oysterpack implementation. Such a basket contract would be one-to-one coupled with a Jobs registry contract.

All the various owners of data processed through Jobs would be stakers, spread over all validators pertaining to the basket. Every validator in the basket would run the same Job for a given owner regardless of whether owner is staked to that validator or not.

Assuming all-or-nothing consensus, if one validator’s result for a given Job differs from the results of 2/3 of the rest, and it so happens that the relevant owner is staked to that validator, then owner's STAKE is automatically booted out from the validator and, for instance, spread equally to the 2/3. The basket contract later prevents owner from ever staking back to the validator in the wrong, which is also useful if owner was STAKEd to a different validator.

If the validator had not been in the wrong, they would keep a small cut of owner's STAKE rewards for themselves as payment for successfully fulfilling the Job. An owner's stake traveling between validators in a basket would have no impact on owner's basket share balance, changing only the revenue stream associated with the balance.

1 Like
6

They can be stored on-chain for example to start and then a more robust storage solution can be added.

It needs to be a different SDK for this. Because there is no immediate write access to storage and there is ability to call out into the internet. So might need a modified version of SDK or just few extensions for this.

This is not required for security - the only thing required for security is validators adopting it and registering that they are running it via the key they use for validation.

It might be useful to plug into nearcore to be able to read state of blockchain, but I think v1 doesn’t need that. It can be just running standalone runtime to avoid writing more code.

This is just part of Worker that reads off the RPC or whatever the requested jobs and starts the runtime to process things and later signs a transaction to post it back also via RPC. Of cause can be bundled with a validator but that will create more security issues, so it’s better to make it isolated.

You can do a lot there to ensure security over time, but I think first version can be way simpler.

3 Likes