Revisiting contract testing framework

Now that we have more people working in the Developer Platform team, it is time to revisit our approach to testing contracts, and improve on them.

Current state

Currently there are two ways Rust contract developers can test their contracts, and there is one way AssemblyScript developers can test theirs.
Rust:

  • Rust devs can use unit-tests like these to test small things. They are easy to use, but they are limited and cannot test full spectrum on interactions between contract and the blockchain, e.g. they don’t allow cross-contract calls or issuing transactions;
  • Rust devs can also use near-sdk-sim test like these to test complex interactions between all kinds of contracts, plus it allows contracts to test creation of transactions. Unfortunately, behind the scenes near-sdk-sim emulates part of the NEAR node, which is costly to maintain and does not match execution of the node 100%;

AssemblyScript:

  • AS devs use tests similar to near-sdk-sim in their functionality. The test are written in AssemblyScript and they leverage similar emulation environment behind the scenes;

Motivation

There are multiple reasons to change how we current approach testing:

  • It would be good to not maintain many testing frameworks especially if their functionalities overlap;
  • We are planning on allowing people to develop in other languages on NEAR (2 more languages are currently underway) and we want them to be able to reuse existing frameworks without creating a language-specific framework each time;
  • It would be good for the tests to be able to run on the live nodes, like local node, Testnet, or Mainnet. This will allow developers ensure that the test behavior can be replicated in production;
  • Maintaining an emulator of NEAR node is costly. Plus, it will inevitably diverge from the actual node causing bad DevX;
  • It would be good if testing frameworks were completely decoupled from SDKs to allow easier iterations on both.

Ideas

Several ideas were voiced:

  • We discussed getting rid of the emulator and replacing it with the NEAR node, similar to how Ganache is used by Truffle. We call NEAR version of Ganache – Sandbox;
  • There was an idea to allow users to run tests on code snippets from within our documentation webpage. Tests do not need to be hosted entirely in the browser webpage (i.e. they can run on the backend), but it would be cool if people could run them on Mainnet/Testnet for extra realism;
  • We might want to replace AssemblyScript tests with TypeScript tests;

Work group

The following people are involved with this project:

Additionally the following people are partially involved in the project:

  • @austinabell – helping with Rust code reviews;
  • @mikedotexe – orchestration from the Developer Platform side;
  • @matklad – orchestration from the Contract Runtime side.
3 Likes

One danger I see here is ending up in a state where we have three different testing stories, neither of which is better than the other. So I think it’s important to focus on transition story early on.

Rather then trying new exciting features, unlocked by sandbox, we probably should prioritize feature-parity – all the existing tests should be easy to port over to sandbox.

2 Likes

Thanks for the summary here. I share a similar sentiment to Kladov. I’d like to advocate for pumping the brakes on making new stuff until we’ve properly documented the solutions we have.

Over the weekend I wrote a long simulation test that had to produce blocks to “fast forward” until a certain point when transactions needed to be called. This ended up being fairly odd. This tells me that we haven’t sufficiently gone through the battle-testing and feedback cycle for simulation tests.

Perhaps we can survey partners to see if they know how to accomplish their testing needs currently. If they don’t, or have to spend tremendous time figuring it out, perhaps we shore up the “how to” side of things.

Basically, I would hate to leave partners/developers with an under-documented approach to what they need, as we charge forth on a different approach.

Several more (disjoint) considerations from a chat with @olonho:

Current testing approaches cause a couple of big architectural oddities. The first is that near-sdk-rs unit tests compile contract to native code, rather that to wasm. Rust contract compatibility with x86_64-unknown-linux-gnu is accidental complexity for contract developers. The second is that near-sdk-as emulation environment is based on subset of nearcore crates, compiled to wasm. This creates a dual bit of accidental complexity – parts of nearcore must be compatible with wasm32-unknown-unknown.

There are two non-fundamental, but significant issues we’ll face with developer experience sandbox.

First one is test performance – today sandbox RPC contain some wall-clock time waits, which makes the test significantly slower than in-the-process emulation. Note that the RPC itself is not a problem – shoveling large JSON between processes is fast on current systems. It’s mostly just fixable internal things like waiting for the blocks to be produced.

Second one is sandbox delivery – today’s test work fully within programming language default toolbox (cargo test). With sandbox, we’ll need to have a binary which exists outside of the language’s ecosystem, and delivering that might cause friction. As an extreme case, today you need to requite to compile sandbox from source. @olonho suggested an interesting idea here, that the sand-box shouldn’t necessary run on the local machine. We can imagine some kind of near-sandbox-as-a-service thing, where language sdks (an in-browser playgrounds) just make requests to some known host.

To be clear, I personally am convinced that language-independent sandbox is the best long-term solution, both for devx (its higher fidelity than any simulation, and tests being independent of the contract itself is a killer feature) and for near maintainers (only one repo to maintain). But yeah, moving into sandbox world is non-trivial, it needs to be a conscious, coordinated effort.