Indexer as a service. Proof of concept

Problem description:
Currently, at NEAR we have a tool NEAR Indexer Framework you may be familiar with.

I know a few teams who built indexers using Indexer Framework:

But a lot more teams avoid building indexers. The ones who built struggle.

Running an indexer is almost the same as running a NEAR node but you also have custom code and you can’t get binaries on nearcore updates. You need to maintain a “custom NEAR node” of some sort.

It might be expensive, it requires time and money investments and it distracts you from your business.

Solution we see:
To create Indexer as a Service.

We want to allow our users to focus on their business. We can achieve it by providing a tool with which they can:

  • benefit from indexer features
  • avoid running a NEAR node
  • decrease time and money investments for maintenance
  • focus on the business, not the tools.

We decided to start by providing simple yet useful tooling to check if users may want it.

Indexer as service nodes will store every single block on some blob storage (AWS S3 or alternative) in JSON format named by block hash. An API endpoint in Indexer as a service will return any block by its hash.

We’re going to define a set of triggers that essentially are going to represent some sort of events from the Network:

  • receiver_id is observed (receiver_id is user-provided)
  • The balance of the account has changed (account is user-provided)
  • Epoch has changed
  • etc.

It is considered that there are up to 20 types of events that can interest application developers.

In case of any event, a POST request with a relevant piece of data and block hash will be sent to a user-provided endpoint. By “relevant piece of data” we mean the piece from the block with something that corresponds to the user-selected event. For example, if a user is watching for “receiver_id is observed” and Indexer as service notices a Receipt with receiver_id provided by the user a POST request with block hash and the Receipt will be sent. All other transactions, receipts, and execution outcomes will be skipped for the user as non-relevant. The user will be able to request the entire block from blob storage via the Indexer as a service API endpoint.

The PoC will look as following (from the user perspective):

[To be discussed additionally] A user buys Indexer as a service fungible tokens on the service’s contract.

In the service contract, a user will set which predefined event they are interested in. They will provide a set of parameters relevant to the event and the endpoint to send a POST request. For security reasons, the endpoint might be encrypted with a service-provided public key.

Once the event is emitted on the network a POST request is sent to the user’s endpoint. The user is charged for checking every block against the user’s “filters” (a small yet reasonable amount to cover the CPU cost), either number of requests or the number of bytes sent to them.

Contract UI

Draft of the contract UI (made quick and dirty): https://www.figma.com/file/kc2Gnb333T98IyMslKhezC/Indexer-Service-Contract-UX-draft?node-id=0%3A1

Basically the flow:

  1. A user logs in with NEAR Wallet
  2. A user see all the so-called filters created earlier (if any)
  3. A user can toggle filters w/o deleting them
  4. A user can buy “Indexer coins” to spend them on checks and events
  5. A user can create a filter

Contract UX:

As for the contract (it’s all are about to be discussed and clarified though)

Assuming a user can buy 1000 of Indexer coins to spend on checks and requests for 1NEAR.

A user is buying coins and submits their filter. Indexer Service keeps track of the user balance and makes a decision to include/not include the user filters on every block handler step.

If the balance is positive Indexer Service checks the block against user filters. Right after the check, the service increments the number in an “invoice”.
If the event triggers the user filter and the service needs to send a request it checks the user balance again. If it is enough for a request then the service increments the number in the “invoice”.

Once in 10 minutes (think of it like the service epoch) the service performs a function call to the service contract owner method (using FUNCTION_CALL access key) to send the invoice. Once the invoice is received the contract decreases balances for users specified in the invoice.

Obviously, on the service side, the invoice is cleared and we start over a new 10 minutes cycle.

P.S. If the user balance reaches zero in the middle of the 10 minutes cycle the service just skips user filters.

How to handle it?

We imagine a user might have a simple HTTP-server that is ready to receive a specific JSON structure from the Indexer Service. After some manipulation with the data (if necessary) the handler can send a message to Telegram BOT or to store it in some MongoDB base for further use. Those handlers could even be hosted on AWS Lambda functions or alternatives.

The user code will also be able to pull the previous and the next blocks to fill up potentially missing information (e.g. cross-contract calls) or perform RPC calls to fetch relevant details if the data is pushed through the POST request from Indexer as a service is not sufficient.

Implementation of PoC plan:

  • Choose one of the possible events to implement
  • Write a contract for Indexer Service:
    • Fungible tokens
    • Possibility to choose an event and set necessary parameters
    • User charging
  • Indexer Service node
    • Read the contract state for events to track and send
    • Listen to every block for tracking events
    • Store block in blob storage
    • Send requests to users with tracking events
    • Function call to IaaS to charge users
  • Frontend
    • Indexer Service contract frontend
    • Login
    • Buy IaaS fungible tokens
    • CRUD events
    • Enable/Disable events

We encourage those who build their indexers to share their thoughts if this project might be interesting to you.
We encourage those who avoid building their indexers to share their use cases so we can empower them with a proper tool.
And we encourage everyone to share their thoughts about such a service.

  • Are you interested?
  • Do you need something like that?

Thank you!

10 Likes

This is an awesome idea and I definitely think that members of the community will be interested and use this service! I’ve worked with NEAR for a little under a year now and I personally had trouble setting up our indexer (which is open source if anybody wants to use it).

I feel as though there is a relatively high barrier to entry for new users and it can definitely feel overwhelming trying to understand how everything works. A very generic use case that a lot of users might have is keeping their own databases synced with the blockchain. For this, they would need to run an indexer that listens for events and updates their database accordingly.

Users might not care about how the indexer works and simply want to be able to use it. The proposed PoC is a great way to tackle this issue.

The only question I have is related to performance - what are the performance drawbacks of using the indexer as a service vs. running your own indexer.

1 Like

We don’t expect any performance drawbacks. Actually, our users will pay for the service to avoid complexity and additional drawbacks. However, we won’t be able to guarantee good performance on the user’s side (where the handlers are)

2 Likes

What do we want to achieve with the smart contract that charges for the service? It seems to me that handling it through smart contracts is expensive and also incurs additional overhead related to security and privacy.

Financial side of things and use the contract state as a backend/storage already shared between all nodes.

How does this contract work exactly? Do we charge a one-time fee or will there be some sort of recurring payments?

1 Like

Recurring.

We plan to charge a user who has active filters:

  • for each block the Indexer Service checks (it takes resources to check if the data in block matches the user’s filter)
  • if filter matches and a request has been sent we plan to charge for either number of requests sent or for number of bytes sent (not decided yet)

We consider to charge our users in bulk once in 10 minutes.

@Bowen I don’t get your concerns about the contract. IMO it’s the best way to handle financial part of the service. Also, I don’t think having external off-chain database and off-chain payment gateway would be better. On-chain solution for this purpose looks the best from my point of view: stable, reliable, transparent, and showing the power of Open Web.

1 Like

How does the UX work here? Could you explain the workflow in more details?

To be clear, I am not against this idea per se. I just want to better understand how it works in practice.

@Bowen I’ve created UX draft quickly. You can see it here https://www.figma.com/file/kc2Gnb333T98IyMslKhezC/Indexer-Service-Contract-UX-draft?node-id=0%3A1
(I’ve added the link to the original post)

Basically the flow:

  1. A user logs in with NEAR Wallet
  2. A user see all the so called filters created earlier (if any)
  3. A user can toggle filters w/o deleting them
  4. A user can buy “Indexer coins” to spend them on checks and events
  5. A user can create filter

Yeah, I’ve just realized you probably are not interested in UI :slight_smile:

As for the contract (it’s all are about to be discussed and clarified though)

Assuming a user can buy 1000 of Indexer coins to spend on checks and requests for 1NEAR.

A user is buying coins and submits their filter. Indexer Service keeps track of the user balance and makes a decision to include/not include the user filters on every block handler step.

If the balance is positive Indexer Service checks the block against user filters. Right after the check, the service increments the number in an “invoice”.
If the event triggers the user filter and the service needs to send a request it checks the user balance again. If it is enough for a request then the service increments the number in the “invoice”.

Once in 10 minutes (think of it like the service epoch) the service performs a function call to the service contract owner method (using FUNCTION_CALL access key) to send the invoice. Once the invoice is received the contract decreases balances for users specified in the invoice.

Obviously, on the service side, the invoice is cleared and we start over a new 10 minutes cycle.

P.S. If the user balance reaches zero in the middle of the 10 minutes cycle the service just skips user filters.

I really hope I’ve explained it clearly. Feel free to ask questions if my explanation is somehow messy.

1 Like

Thanks for the explanation. One more question: what is the workflow to ask users to top up their balances? Do we notify the users when their balances are low?

1 Like

It’s a great question. I’ve forgot about it. Though it’s solvable. I thinks we can come up with idea how do to it later.
We can go with TG bot or add email field for notifications (encrypted). I don’t think it is a blocker for proof of concept. What do you think?

2 Likes

Yeah sounds good to me

2 Likes