Standard / convention for structured logging or "events"

Given recent discussions with The Graph and @zcstarr on the multi-token standard, I feel like this is a good time to discuss structured logging. This can also be thought of as “events” that can be picked up by indexers. (See Ethereum-like logs on NEAR for more.)

Brief history

Some smart contracts on NEAR have historically used the logging features to write simple, human-readable message. These are unstructured. For example, check out the lockup contract:

Some other logs in this same contract add useful information, but it’s also in a human-readable form and would need to be parsed manually:

Lately, a convention has cropped up where logs are using more of a code than a human-readable message. Take this example from SputnikDAO v2:

Note that in the above case, this is a user error where the provided value is invalid. This is one kind of error but not the only kind. For instance, this next example comes from the function internal_execute_proposal, where the internal_ prefix has become a convention for internal logic, perhaps after user errors have been checked.

Hence, the previous snippet is an example of a potential contract-level error, which is different than user input error.

We also know that users may make mistakes in how they’ve constructed their transaction. For instance, perhaps they haven’t included enough gas or enough Ⓝ for storage staking. It would be quite valuable to have some structured logging/events that returns this information in a way that doesn’t need to be parsed or is subject to change.

Proposal

Let’s come up with structured logging events. The following is a place to start this conversation. First, here’s a short snippet from a main contract.

#[near_bindgen]
pub struct Contract {
  …
  // URI/URLs to download resource with fallback ordering
  // The resource is structured, containing translations of 
  // log codes, extended help text, etc.
  log_events_ref: Vec<String>,
}

It’s not terribly useful to have multiple language translations (or long text) in contract state. These can live in a file on IPFS or a public web2 server.

The next part of this is the proposal implemented in a separate file:


If an internal contract error has occurred, it can be filtered by an indexer that looks for the LogLevel of type “ContractError”. That would look like this:

env::panic(&LogEvent::new(
  Lvl::ContractError,
  "NO_USER_EXISTS").b()
)

(Note: LogLevel has been aliased to Lvl for brevity.)

If a user enters the wrong kind of input, a frontend can take the log event and display something useful:

let val = self.my_map(&user_input).expect(
  &LogEvent::new(Lvl::UserError, "NO_ABILITY")
  .advice(format!(
    "Please use one of these abilities: {}.",
    self.get_abilities()
  ))
  .to_string(),
);

Note that advice is meant to be a brief message worth storing on-chain, but the link(s) via the log_events_ref would have extended help / descriptions.

If a user has given the correct parameters but simply hasn’t included enough gas, it will show in the needs section, and so on. This will allow dApps to potentially retry with the knowledge returned from a failed transaction. (Might have to dig into the log_events.rs file linked above for details.)

Lastly, these structured log events would have a version associated with them so frontends and indexers can know what to expect from the structure.

I’d love to discuss thoughts in this thread and eventually make a formal NEP around the idea of structured logging.

The above examples spoke to the need for errors, but note that there are log levels for Info and Warn as well. These would play well as events that help an indexer / frontend react nicely.

Future considerations

At some point it might be reasonable to consider how blockchain has status codes similar to HTTP status codes (401, 500, 200, etc.). This proposal is perhaps a small step in that direction.

3 Likes

This is very interesting!

I like this idea, instead of running the indexer once again, we can just grab all the events in the contract, it also provides a good “backward-compatibility” for all the current live contracts to follow the new standard by manually indexing all the transactions and put it into a standard format and just push it to the contract.

Not really familiar with the logging and storage, but can we stringify object and put it in the log? With this we can structure something like ethereum did, the event title and then event params in object.

Some observations:

  • I think it is pretty much required for structured logs to be binary as oppose to formatted strings. NEAR is interoperating with other blockchains, like Ethereum, and it is much harder and error prone to parse string in Solidity as oppose to unpacking a binary object;
  • This post seems to be talking both about logs and assertion messages. AFAIR assertion messages do not output logs, and they are instead returned through ExecutionOutcome. Even though ExecutionOutcome is merkelized and therefore provable, it is much harder to work with than just regular logs, because it is a much more complex object to unpack. Besides, I am not sure we will be adding Bloom filters to execution outcomes.

I’m kind of of the same mind as @nearmax, that it seems like binary events would still be really desirable here. I think with structured logs it’s about being able to ingest and transform data that one context wants to consume with little fuss. I think logs in this format would be nice. My mind goes to building analytics, and aggregation statistics for my specific contract.

I think for doing things like hooking up something to grafana, splunk, ELK, or doing a big query type of ingestion, where you know your particular format, then it becomes an easy way for you to consume your log data and hook it up to some control system. I think that it makes sense.

With events however, I think that it would be nice to have a more strongly typed experience that perhaps binary data could provide.

One thing to conisder about the string events is communication of format and indexer expectations, might be harder as an indexer, it’s not as fun to write /^\s\s token_id:%d.*$/ . It’d be easier if you were able to point to a lib or import event data to then produce blob to index I think, even if it meant serializing it into the language of your choice. Probably the hardest thing about the binary format is communicating what the event format should be and having that be easy to deserialize into language context of your choice. :thinking:

For status codes, I know this was floated around a cpl years ago, some interesting ideas around status codes events for contracts that make sense:

1 Like

Also, we are already forced to replicate Ethereum’s ABI, since many of NEAR contracts use binary arguments. Which means we need metadata/ABI, but given that in Ethereum contract ABI also includes events/logs, I don’t see why we wouldn’t extend our future metadata design to include descriptions of logs too.

1 Like

Great thoughts. So what all needs to happen to remove this UTF-8 restriction we currently have? My understanding is we’d want to change this part of the Rust SDK:

which then requires a change to nearcore around here:

Perhaps we’d need to:

  • Reach out to partners/builders who are using indexers to make sure they can handle this change?
  • Consult with the Node Experience team who have been helpful in advising folks on implementing the node indexer framework.
  • Other thoughts?

Trying to get an idea of how big of a lift this would be.