New NEAR CLI design proposal!

Development Proposals
1

:warning: This discussion will be closed on August 2, 2021. Please, share your thoughts!

near-cli Core NEP

Summary

NEAR CLI is one of the primary tools NEAR developers must use to develop and interact with NEAR networks and smart contracts. Today, NEAR CLI contains the base commands for account creation and deletion, login, viewing account state and keys, sending tokens, creating staking transactions, deploying, and calling smart contracts.

This proposal will cover best practices that we want to follow during the development process, propositions to improve our key management and distribution systems, ideas on how to make NEAR CLI more secure, modular, and configurable.

NEAR CLI is an excellent tool to attract and engage new NEAR developers, both core and contract level, and if it contains all the required features to select and configure and deploy NEAR networks, nodes, crypto-assets (i.e., keys), accounts, and wallets, it can be the ideal single-entry point for those potential new hackers.

Motivation

These enhancements are intended to simplify and enhance developer and user productivity by providing a single and at the same time modular command-line tool that exercises all major features of NEAR. As each major piece of NEAR functionality is added to NEAR CLI we should see the utilization of each of those features increase. Additionally, community mindshare and understanding of NEAR should increase as NEAR CLI will contain detailed help documentation for each major feature with clear examples of use, just like Linux man pages and/or other major CLI tools that have integrated help for each command.

First-class user experience

General principles

  • NEAR CLI is interactive, prompts and advice are used over errors
  • Colors, highlighted text, animated loaders and progress bars are widely used
  • NEAR CLI provides the possibility to setup auto-completion (generation of .env file and instructions)
  • Common answers are saved into config file, frequent and identical prompts can be annoying
  • NEAR CLI uses default parameters, sometimes questions are redundant
  • NEAR CLI works out of the box, no initial setup is needed
  • Multi-language support (For --help, errors, and logs. Commands are not translated, language can be changed in config following the ISO 639-1)
  • All docs are available in man near, near --help, and near <command> --help and near help <command>. There is no actual need to visit docs.near.org while working with NEAR CLI
  • Only relevant flags and commands are shown in near --help and near <command> --help
  • All links are hyperlinks, nobody wants to copy them to the browser
  • No magic! Users always know what they are doing
  • --verbose flag can be used to show additional information about the execution process

Note: propose additional best practices that NEAR CLI should follow to improve user experience.

Two modes of interaction with the user

Since NEAR CLI will not throw errors if some of the arguments were not specified, it will ask questions instead. It means that users will be able to work in two modes. The first one is for the engineers who are familiar with the CLI, and the second one is for easy, guided interaction.

  • Example of classical mode where user is entering command as a one-liner.
near send testnet NEAR --sender serhii.testnet --receiver vlad.testnet --amount 10N

serhii@x1 ~/P/N/h/n/n/t/debug (master)> ./near-cli

? Choose your action ›
✔ Choose your action · Transfer tokens

✔ What do you want to transfer? · NEAR tokens

? To construct a transaction you will need to provide information about sender (signer) and receiver accounts, and actions that needs to be performed.
✔ To construct a transaction you will need to provide information about sender (signer) and receiver accounts, and actions that needs to be performed.
                 
Do you want to derive some information required for transaction construction automatically querying it online? · Yes, I keep it simple

✔ Select NEAR protocol RPC server: · Testnet

What is the account ID of the sender?: ⏎

These two modes can be mixed. If the user entered only half of the parameters then they will be asked about the remaining half.

Scripting friendly

  • The output of the NEAR CLI is easy for parsing (--structured flag will return result in JSON)
  • The output of the NEAR CLI can be piped to the input of another CLI
  • Result of another tool can be used as an input to NEAR CLI (we should accept input as STDIN)
  • NEAR CLI should be OS agnostic (Linux, macOS and Windows)
  • NEAR CLI should use exit codes properly
  • Shell’s environment variables and any other environment variables should be available to the application
  • NEAR CLI shall detect whether it’s been used in an interactive shell and if not (means scripting mode), it should not show interactive prompts but exit with error code and structured error message

Note: propose additional best practices that NEAR CLI should follow to improve scripting experience

Following industry best practices

POSIX standards

  • <> for required parameters, [] for optional parameters
  • All flags has short versions ( --force has short version -f)
  • Flags can be merged (-a -b -c is equal to -abc)
  • We respect POSIX signals (such as Cntrl+C)
  • etc.

Error handling

  • Errors are Tractable (each error should have a code/name so it can be easily referenced)
  • Errors are Actionable (each error should give user a hint on how the error can be fixed)
  • NEAR CLI should have a debug mode with rich error output (provided with flag --verbose)
  • When the error is thrown NEAR CLI should provide a link to the GitHub issue template in the NEAR CLI repository. The new issue should be prefilled with available parameters.
  • Never shows stack traces (avoid uncaught throw in JS or panic in Rust)
  • When --verbose flag is not specified NEAR CLI stores error log in a file, and mentions file location in stderr

Note: The list of best practices can never be full. Anything we forgot about to mention? Add your propositions in the comments.

Secure

Key management

Current situation:

  • Private keys are stored as a plain text in home directory
  • Only one key is stored per account

In new NEAR CLI it should look like:

  • Private keys can be stored as plain text, in OS level key store or external key storage (like bitwarden)
  • Storage of any type can store multiple keys per account
  • User can choose what key they wants to use
  • Keys can not be overwritten, we should eliminate any possibility to lose keys
  • Users can specify key management solution is cli.config
  • Each key is a separate file. It will prevent loss of all keys during the file read/write process.

Proposed file structure:

─ ~/.near-credentials
 ├── mainnet
	├── alice
		├── ed25519:<PK>.json
		├── ed25519:<PK>.json
		└── ed25519:<PK>.json
	└── bob
		├── ed25519:<PK>.json
		├── ed25519:<PK>.json
		└── ed25519:<PK>.json
 ├── testnet
	├── alice
		├── ed25519:<PK>.json
		├── ed25519:<PK>.json
		└── ed25519:<PK>.json
	└── bob
		├── ed25519:<PK>.json
		├── ed25519:<PK>.json
		└── ed25519:<PK>.json
 └── ...

An example of ed25519:<PK>.json file from .credentials/<network_name>/<account_id> folder

{
	"hd_path": "<HD Path>",
	"private_key_location": "file", <- can be `ledger`, `system_key_store`, etc. 
	"private_key":"<key>"}⏎ <- can be empty if `private_key_location` is not `file`
}

Also, we are planning to investigate the approach of Amazon Vault system.

Note: if Rust will be chosen as the main language for the new NEAR CLI we will use .toml files for all keys

Note: Well-designed key management is critical for our success. Please leave comments!

Accessible and well distributed

  • Ship binaries for common OSes; distribute through brew, apt-get, AUR, yum, chocolatey, scoop, etc.
  • Minimize use of production dependencies
  • Cleanup configuration files during uninstall (but not keys!)
  • Each release should be signed, users should be able to verify them
  • Release process should be automated

Configurable

Certain NEAR CLI functionality requires an input of the same data continuously, this information can be stored in the config file not to bother our users.

NEAR CLI will have 3 types of config files:

  • cli.config for configurations that affects core NEAR CLI behavior
  • connections.config for the list of connections (connections are more flexible than networks)
  • X-extension.config for configurations that affects NEAR CLI extensions

Proposed file structure:

─ ~/.near-config
 ├── connections.config
 ├── cli.config
 ├── X-extension.config
 └── Y-extension.config

  • connections.config example:

    {
	"mainnet": {
	        nodeUrl: 'https://rpc.mainnet.near.org',
	        walletUrl: 'https://wallet.near.org',
	        helperUrl: 'https://helper.mainnet.near.org',
	        helperAccount: 'near',
	        explorerUrl: 'https://explorer.mainnet.near.org',
	},
	"testnet": {
	...
}

  • cli.config example

    {
	"default_keystorage": "home_folder",  ⟵ example: user chose to always save keys to the home directory when running "near login"
	"automatic_migrations": "always", ⟵ example: user wants to always run migrations automatically. "never" and "ask" options are also available
	"login_type": "browser"
}

    User will be able to manage this config with any text editor or built-in commands, such as near config set <parameter> <value>.

  • X-extension.config will similar to cli.config

Note: if Rust will be chosen as the main language for the new NEAR CLI we will use .toml files for all configs

Extendable

The current NEAR CLI has many user types. Each of these user types requires its own set of functionality. At the same time, we are trying to keep NEAR CLI minimalistic and easy to use. A variety of commands can be confusing.

The solution to this problem is extensions. It will allow the core NEAR CLI to remain as lightweight as possible and satisfy all the demands from our growing user base at the same time.

Lack of extendability forces developers to create their own tools from scratch. Extendability within our tools alleviates this and reduces strain on our team, while also fostering a stronger dev community.

Here is a list of functions that can become an extension:

  • Validation
  • Project functionality (creation, building, testing, etc.)
  • Linkdrop
  • nearup

Note: Some of the proposed extensions can be a part of the core NEAR CLI, leave your thoughts in the comments.

Extension Platform VS Library

We will not try to build a whole extension platform. The extension should be a separate and independent library that can be executed with NEAR CLI. NEAR CLI extension developers should be able to reuse convenience components from NEAR CLI (e.g. near-cli-key-management).

Note: Extensions are available in Rust. Good example is Cargo extensions.

Upgradable

As NEAR CLI matures, updates may/will cause a user’s configuration to become outdated. This proposal declares that NEAR CLI will have a mechanism to make such upgrades possible.

This endeavor cannot happen, however, without keeping track of which version of NEAR CLI was most recently used. We will store this version in a cli.config file.

Example:

...
"NEAR_CLI_VERSION": "2.4.0",
...

Migration scripts will be able to update all 4 types of config files (see Config section).

The folder structure within NEAR CLI will have migration files per minor version as illustrated here:

└── middleware
   └── migration
      ├── README.md
      ├── scripts
      │  ├── 0.19.x.js          ⟵ logic to run for 0.19.0 to 0.19.N
      │  ├── 0.20.x.js          ⟵ logic to run for 0.20.0 to 0.20.N
      │  └── x.x.x-template.js  ⟵ template for adding new migrations
      └── shell-upgrade.js      ⟵ checks for the current version against the stored version, etc.

Note: Migrations can also improve the experience by, for example, removing orphaned files, make safety checks and show warnings, etc.

Commands

The list of commands is highly affected by the design of our modular system and the decisions that we will make there. Here, we will try to explain the general approach.

near
- login
- send <network> <sender> <receiver> <amount> <- easy way to send tokens
- <other aliases of contract-transaction> <- we can move other popular functionality to the top level
- account <accountId>
	- create-subaccount <subaccountId>
	- state
	- delete <beneficiaryId>
	- keys <- will show account keys
		- generate
		- add
		- delete
- construct-transaction <parameters>
- config
	- set <parameter> <value>
	- show <parameter> <- should show all available options aswell
----------------------------------------------------------------
The functionality of the `dev` extension is still under development, it can be structured differently:
- dev
	- contract 
		- call
		- view-call
		- state
	- deploy
	- dev-deploy
	- build
	- test
	- create-project
	- repl
----------------------------------------------------------------
Validation extension:
- validators
	- list <epoch>
	- proposals
	- stake (can be moved to top level as an alias)
---------------------------------------------------------------
Explorer extension:
- explorer
	- tx-status
	- etc. (all the other functions from the explorer)
---------------------------------------------------------------
Users will be able to write and add their own extensions.

Should we write new NEAR CLI in Rust or JS/TS?

Motivation to write it in Rust

  • We have a cool project from Vlad Frolov that we can base our new CLI on
  • Reliable installation (a single binary installation that cannot get broken because of node.js upgrade or something like that)
  • Predictable operation:
    • scripting languages (JS, Python, …) are great for happy-path implementations, and really tough when you need to handle errors, so Rust just solidifies and secures the operations
    • scripting languages require longer start-up (0.5 - 1 second of additional delay)
  • Secure (single binary, reliable libraries)

Motivation to write it in JS/TS

  • NEAR CLI in JS is already live in production
  • Some say that it’s faster to develop software in JS/TS
  • More people know JS/TS means more external contributions
  • No build process during development! Just change code and execute

Some parts of the proposal are based on @mikedotexe work. Big thanks to @josh.quintal for collaboration and @frol for inspiration and Rust insights.

:warning: This discussion will be closed on August 2, 2021. Please, share your thoughts!

10 Likes
2

@Serhii Thanks for the comprehensive analysis and summary!

There are tons of dimensions touched here. Indeed, we already have two versions of CLI (JS and Rust), so we can now compare the two, so I can even invite you to give the Rust version a try yourself: near-cli/README.en.md at master · FroVolod/near-cli · GitHub.

Note, the layout of the command is different from the one presented in the post; the current implementation is action-first, while the current proposal suggests using resource-first. Let us know which one you like better. (I like both, but I have not tried to model all the commands in resource-first approach yet)

2 Likes
3

I really like the thinking here!

A couple thoughts –

A couple dev things i’d like to see

  • txn repeatability - would be great to be able to replay a txn with incremented nonce. Helpful for quick testing of transactions. Either a param like “repeat=10” or similar.
  • RPC Fallback - in the event near foundation rpc nodes are unreachable, would be great to have fallback configurations available.
  • Default preferences/configuration: Specify things like NEAR amount, gas, etc for ongoing use.
  • Access keys: Need a way to view these and potentially tag them locally. This is helpful for understanding which keys go to what client or dapp
4 Likes
4

“Should we write new NEAR CLI in Rust or JS/TS?”

I vote Rust!

"NEAR CLI will have 3 types of config files"

I vote one .nearconfig file, or a .near folder with a config.toml / config.json. But a single config file.

This still allows nested sections for connections and extensions. In toml, it would look like:

default_keystorage = "home_folder"
automatic_migrations = "always"
login_type = "browser"

[connections.mainnet]
node_url = "https://custom-rpc.example.com"
# other settings omitted, to use defaults from near-cli

[extensions.validators]
something = "some other thing"

In JSON:

  "defaultKeystorage": "home_folder",
  "automaticMigrations": "always",
  "loginType": "browser",
  "connections": {
    "mainnet": {
      "nodeUrl": "https://custom-rpc.example.com"
    },
    "extensions": {
      "validators": {
        "something": "some other thing"
      }
    }
  }
}

(Note that I’m also implying that JSON should use camelCase setting names and toml should use snake_case.)

Why one file?

  1. It matches the pattern of other projects such as cargo, git, babel, etc.

  2. In the future, this could allow for easier config nesting.

    Say for my main job I build a production NEAR app that has a custom RPC server, so I want to override connections.mainnet.node_url for those projects. But for hobby projects and hackathons, I still want to use the default RPC server provided by NEAR Foundation.

    For cases like this, it would be great if I could throw a .nearconfig or .near/config.toml in any folder and have its settings automatically picked up by NEAR CLI when I run commands from that folder and its subfolders. Naturally (and as with the other projects I linked to above), the more deeply a config file is nested in my filesystem, the higher priority it has.

Configuration via environment variables

Just as with cargo config, if predictably-named environment variables like NEAR_CONNECTION_MAINNET_NODE_URL (or maybe NEAR_CONNECTION_NODE_URL) are set, they should override any settings from config files.

The contract functionality should not be part of an extension

In the Commands section, you list:

The functionality of the `dev` extension is still under development, it can be structured differently:
- dev
	- contract 
		- call
		- view-call
		- state
	- deploy
	- dev-deploy
	- ...

In my use of NEAR CLI so far, I call contracts frequently that I did not deploy and which I do not contribute to. Making interaction with existing contracts a part of the dev extension seems wrong to me. And philosophically, we want to make it easy to interact with NEAR contracts from the CLI; we shouldn’t make people install a “dev” extension to do so.

I think near contract should be a subcommand that’s shipped with the core NEAR CLI. I also think we should call the next subcommand view instead of view-call, but I feel less strongly about this. So the proposed change looks like:

- config
	- set <parameter> <value>
	- show <parameter> <- should show all available options aswell [note the typo, copied from original post 😉]
- contract
    - call
    - view
    - state
----------------------------------------------------------------
The functionality of the `dev` extension is still under development, it can be structured differently:
- dev
    - deploy
    - dev-deploy
    - ...

The dev subcommand has multiple issues

Maybe we can save this discussion for a future forum post. I see multiple issues here:

  • Some commands are for creating & deploying contracts, others are for bootstrapping frontends with helpful conventions using near-api-js. Maybe it should be two separate extensions?
  • near dev dev-deploy is awkward; maybe it needs a better name?
  • If one of the extensions is for creating & deploying contracts, could it extend the core near contract subcommand I advocated for above? Then instead of just call, view, and state, you could also near contract build, near contract deploy, etc. This seems like expectable naming, but I’m not sure it’s possible or advisable.
  • Should repl be moved to near-api-js?
  • What is near dev test supposed to do?
  • The only proposed command for bootstrapping frontends is create-project. Should this be part of core NEAR CLI? We could avoid bloating the core distribution by fetching template files from a separate GitHub project when someone actually runs the command. If we envision eventually having more commands for working with frontend codebases, we could split this into a separate extension; maybe near project, with a further subcommand near project new or near project create.

Should we show near extensions subcommand?

We will need a near extensions subcommand (name up for discussion) to manage extensions. Should we add it to the list above for clarity, or is it too in-the-weeds?

near
- login
...
- config
	- set <parameter> <value>
	- show <parameter> <- should show all available options aswell
- extensions
	- list     ← lists installed extensions
	- list-all ← lists all available extensions; similar to list/list-all from asdf
	- add
	- rm
	- update
1 Like
5

This is a very compelling reason for JS/TS, honestly. In addition, (rumor has it) simulation testing will be deprecated in favor of a code sandbox, which kind of removes a major incentive for AssemblyScript developers to learn Rust.

6

On the contrast side I want to mention that most of JS/TS developers are doing frontend, and it does not help them to write backends (just a subset of frontend devs get to backend development even given Node.js platform) let alone cli apps.

Actually, we can already tell that extensive use of near-cli JS did not attract many of external contributors: Contributors to near/near-cli · GitHub (27 unique contributors), so I believe we should not take this point into account since successful Rust projects can attract more than that (e.g. ripgrep has 291 contributors)

4 Likes
7

Yes, we are planning to move near-account-utils functionality to the core NEAR CLI, thanks!

Probably we will start with the OS level key stores, should work on MAC. Linux and Windows can be challenging.

  • hm… automated nonce management can be helpful. Thank you for the insight!
  • we are planning to have several RPC providers in the future, fallbacks are possible. Also, the connection can be configured in a config file.
  • Yes, we know that we want to use multiple keys per account, but the design of it is still an open question. Tagging, or giving each key a name can be helpful.
1 Like
8

Most of the extensions will be built by the community. It means that somebody can accidentally damage the config file. It’s safer to have a separate file for each extension.

Looks like it can be done with multiple files as well.

What benefits it will give us?
The downside is that users can be puzzled, “Where it’s configured?”. KISS is not followed :slight_smile:

I don’t have a strong opinion on this. I think that dev extension should be an extension so other similar extensions can be created by the community. If we will move contract outside of the dev extension - it can be a part of core NEAR CLI.

Yes, I think that dev extension should be discussed separately.

Yes! But it depends on the implementation of the extension system. Also, I think that only whitelisted extensions should be on this list.

2 Likes
9

This discussion will be closed on August 2, 2021. Please, share your thoughts!

Some updates from me (after discussions with other people):

More best practices to follow:

  • Use dashes only as word separator in options (–contract-name)
  • –help should print options that are specific for each command
  • we should print warnings when options are ignored
  • we should use more explicit optional arguments order: [arg1 [arg2 [arg3]]]
  • we should not show command-specific arguments in top-level help (to save space)
  • we should get language from the system locale (if not overridden in the config file)

We don’t have any commands that needs to accept stdin. Interactive mode is only for humans. All other info will be specified with commands and options.

They are available to any CLI tool by default. The ability to use env variables in an interactive mode is overengineering, we should avoid that behavior.

We should add the ability for explicit control of interactive/non-interactive mods.

We can have more commands than letters. Only SOME of the options should have a shot version. In general, the POSIX standards block is redundant.

We can save it in /tmp folder and use some library for cross platform support (like tempfile in Python).

For Linux, we can use libsecret and gnome keyring libraries.

In Linux, $XDG_CONFIG_HOME is responsible for a path where configs are stored.
By default it’s ~/.config/tool-name. Lot’s of projects are using it.

2 Likes
10

Regarding Rust vs JS implementation. As we discussed with @jim today this decision shouldn’t come exclusively from NEAR Inc engineers. In the past hackathons and workshops we have observed that blockchain developers struggled equally and in big measure with both installing JS version of near-cli and with compiling Rust contracts. So it is unclear which language is better for the CLI. We should make a thorough research by collecting opinion from the community and the partners, including asking about the option of providing near-cli for all platforms through package managers.

11

Well, we should probably compare installing near-cli through npm vs installing a single binary. If we go with a single binary, we may even package it into npm package if necessary.

My point is that we should not consider a case where someone needs to compile near-cli from the source code as our default installation strategy.

1 Like
12

I’m fine with leaving this out for v1.

I can think of a couple benefits off the top of my head, but they’re not huge wins.

  1. I want near-cli and near-api-js to have similar behavior, and I think we should encourage env vars as the primary configuration approach for near-api-js. (If I’m using parcel or another application runner that respects .env and .env.local files, I should be able to set different env vars in these files to configure RPC node, wallet URL, etc. We can deprecate & remove src/config.js in favor of this more familiar & expectable approach.)

  2. Ease of copy-pasting commands from docs and elsewhere, with config set via env vars at the beginning (NEAR_RPC_URL=blah near do thing)

  3. Honestly I find it more expectable than not being able to do this. Cargo allows it. near-cli allows it now (though inconsistently). And it just feels right to me. But my intuition might be off here, and I’m curious to see how near-cli feels without it.

13

One other request:
There could be a flag to format a gas breakdown on transaction responses.
I want to be able to view a table list of the breakdown of contract/method and how much gas/tokens burned.

I find myself doing this checking by going to explorer or RPC and sifting through. Would save a lot of time developing gas things :slight_smile:

1 Like
14

Here are the real issues with the CLI implemented in Node.js causing all sorts of issues like “too old Node.js”, “too new Node.js”, “upgraded Node.js, so need to re-install near-cli”:

3 Likes
15

Thank you everybody for this great discussion! Now we will move to the active development phase.

It was decided to build the new NEAR CLI in Rust with the use of Clap framework. It will be based on an existing project from @FroVolod, you can check it here.

All other propositions and some of the open questions that we had here will be addressed with a more hands-on approach. Development of the new CLI will be an iterative process so please, keep an eye on our repositories, create issues, and contribute!

4 Likes
16

Absolutely. We’re definitely wanting to do this. Thanks for the bump! We’ve got a few tickets in our icebox regarding this. :slight_smile:

2 Likes
17

Have we collected and processed the feedback from the community and partners yet? If not then we cannot move to the development. We cannot make product decisions in isolation from the community. Please consider it a blocker. CC @josh.quintal , @jim , @frol

18

We will iterate on our rust solution for CLI while re-engaging the community to help inform the structure, interface patterns and implementation going forward. We’re looking forward to demoing progress on our current path to the community and continually soliciting feedback that will guide our path forward.

1 Like