This post is a follow up / proposal in response to another indexer post, linked below. This post is to kick off discussions of what another iteration of that idea could look like.
Many dapps require data for their apps to work well, and extend the UX of NEAR interactions.
Indexing data is required to make data access faster and more specific. Currently many apps must run servers that are costly, or access centralized “free” servers and hope that there arent any downtimes that affect their app.
There are several solutions to aid the indexing for dapps, as previously identified in this post:
However, there are several drawbacks:
- This service would likely be a centralized service and could potentially not grow the way dapps need.
- The service would create and maintain a framework, but is less likely to be fully open sourced depending on the business model.
- Data verification and proofs were not part of the core concept.
Here are the benefits of a different approach:
- Still utilize several of the solutions of the IaaS model.
- Provide a decentralized P2P approach where many participants can get paid to index, and index in different ways.
- Provide a decentralized access to indexed data
- Provide open source code bases
- Grow beyond dapps & near, could grow into multi-chain (Aurora, +)
What does the future look like?
Onchain indexing
- index contract registry
- contracts for specific indexing logic
- Use cases for things like: SMA, MACD, historical timeseries data, larger single total aggregations, analytics
- NOTE: onchain indexing will be released as a part of the https://cron.cat ecosystem. Follow croncat for updates on this!
Offchain indexing
- Run a node + indexer
- Source data to decentralized data
- node can source/submit data to onchain contracts
- node can validate on chain data within contracts
- node can keep track of client usage for charging access to data to cover operation costs.
- node can validate other clients usage to prevent overcharging, DOS or bad actors
- node can provide RPC & websocket services to clients
- SDK will be available in Rust & typescript for integrations
Business model:
- Indexing nodes get paid to source data
- Applications/Users pay a fee to access the data
- Onchain contracts with aggregation logic get royalties
Indexer Collection Stack:
- neard / nearcore
- kafka/rabbitmq producer
- monitoring stats
- signing keys + small balance to produce registry records
- signing keys for decentralized storage
Indexer Validation Stack:
- neard / nearcore
- monitoring stats
- signing keys + small balance to produce registry validation hashes
Indexer RPC Stack:
- neard / nearcore
- monitoring stats
- signing keys + small balance to produce access proofs (keep track of node usage from clients)
I’m SOLD! When can i use it?
- This system is currently in active development, and specifically being built by 1 person actively. It will take a while to get out to the ecosystem in a plug-n-play model. If you or someone you know are interested in helping, please DM me!
Isn’t onchain data free?
- It costs real money to run RPC servers, average cost in AWS is $1200 / mo.
- It also costs money to store data long term, even though the initial data store is expensive, it requires fees to maintain for longer periods of time.
Aren’t there RPCs available for free now?
- Currenlty NEAR Foundation kindly runs the bulk of RPCs for all dapps.
- A decentralized RPC service is in the works.
Can I get data from the beginning of time until now, any time i want?
- No, even with an archive node, the mainnet is running off a compact snapshot.
- Getting data farther back on RPC is slow and expensive to run, having a paid indexed set of data enables fast direct access to data.
Will this indexed data work out of the box for my dapps?
- In most cases yes, since Near represents data in JSON format. Your dapps will be able to consume the direct indexed data without any additional aggregations.
- In some cases, dapps may want a cleaner/more concise data indexing. This is where the core indexing provided should get re-aggregated for further use. An example could be thegraph.com utilizing the raw indexing and creating a subgraph dedicated to specific near dapp use cases.
Thoughts?
Please leave comments, questions & thoughts below.
If you know anyone wanting to work/build something like this, DM me! We’re hiring.