NEAR Python Tooling Community Group

I would like to bootstrap a space for Python developers that want to collaborate on building Python developers tooling for NEAR.

In short, the state of Python tooling on NEAR is lacking. Yet, there is a raising demand, especially after the success of NEAR SDK for JS.

I am going to update the following sections with useful references. Please, share your progress with the community in the comments to this post.

On-chain programs development in Python

near-sdk-py

It does not exist yet, and there is still a need to build a proof-of-concept that can compile Python code to Wasm that is capable to be executed in NEAR’s Wasm environment.

Here are the projects that might help to achieve that:

  • Nuitka - Python compiler that is capable to produce standalone binaries that do not require Python installation on the host.
    • Nuitka does not compromise on features (latest Python features, C-extensions are supported) and even provides some performance optimizations
    • There was a discussion on github years ago about Wasm support, but I feel it is outdated and things got improved since that time
  • cowasm (aka zython) - CPython compiled with Zig to Wasm target.
    • It supports C-extensions in Wasm, yet it seems that the resulting files are quite beefy (their demo loads 24MB over the network). It is a mix of JS and Wasm files, so it is not clear whether it is possible to package everything into a single Wasm file.
  • RustPython - Python interpreter implementation that is compilable to Wasm.
    • Its output is a single wasm, yet it is also quite heavy (their interactive demo loads 25MB Wasm file). Nevertheless, I feel it should be possible to tweak it and selectively include only the necessary features and thus slim it down.

Off-chain apps development in Python

Low-level off-chain JSON RPC API packages

High-level off-chain packages

near-api-js provides some abstractions for accounts and keys management, but near-api-js design does not feel Pythonic, so it should not be copied blindly.

End-to-end testing in Python is also needed, so workspaces-js and workspaces-rs needs to be re-implemented in Python.

There are also other ways to get the data from the blockchain:

  • near-lake-framework-py is a small package that provides a stream of blocks that are finalized on NEAR
  • Pagoda Enhanced API provides RESTful API to get information about assets (NEAR, FTs, and NFTs) and history in an easy way. It has OpenAPI spec, so low-level Python client can even be automatically generated, yet maintenance is still required.

Prior discussions

10 Likes

Hey all, just wanted to tell you about Pyonear, a project I’m working on to provide Python wrappers for near-primitives and other Rust crates that comprise the backbone of near-api-rs.

I’m currently seeking a grant for this, so here are some details on what I’m building and why:

Pyonear (PyO3 + NEAR) is a high-performance Python library for interacting with NEAR, built primarily by wrapping the NEAR Rust crates (e.g. near-primitives) in Python. It uses PyO3 to run NEAR’s Rust code in Python, providing Python users with fast, familiar and robust solutions for foundational NEAR problems (e.g. public key cryptography, building transactions and parsing accounts). The first release will provide comprehensive Python wrappers for the Rust primitives that enable building, signing and serializing transactions.

Why me?

I have a lot of experience building Python blockchain Python libraries, including libraries that call Rust under the hood. I am the top contributor to solana-py, and I am the author of solders, a Rust-based Python library used by solana-py under the hood.

Why wrap Rust code instead of just using Python?

Here are the main benefits to using the NEAR Rust crates:

  • The hard problems are already solved. We don’t need to reimplement things like signing and serialization because we are calling code that already takes care of it.
  • Ease of upgrades: related to the above, it’s easier to accommodate changes to the NEAR protocol. For example, when Solana added a new Transaction version, I was able to add support for it in solders before I even understood how it worked.
  • Performance: while there are many cases where speed isn’t as important as people think, it is still very useful for things like data engineering where N is large. Based on what I observed with Solders, I estimate Pyonear would be 10-100x faster at the things it does than pure Python.

Rust is becoming increasingly popular for Python libraries. Two recent examples are polars, the very fast dataframe library, and pydantic, which recently started using Rust to speed up its core operations.

Do end users need Rust installed?

No! I sometimes see this in comment sections and the like so I thought I’d take the time to clarify it. People who run pip install pyonear are installing a prebuilt wheel for their platform. They don’t need the Rust toolchain at all for this.

5 Likes

This is exciting! I must admit, I was not immediately sold on the idea of building on top of Rust crates even those I am a huge fan of Rust and Python (I use Alpine Linux [which uses musl libc] for Docker images and Mac with M1 arm64 platform, and often I face problems with Python packages unable to build on these platforms), but after I tried solders and faced zero issues, I completely changed my mind. Great job!

4 Likes

Just an FYI. If we succeed with making Python interpreter run in Wasm contract (similarly to our near-sdk-js) then we would need many of these primitives to be re-implemented in pure Python.

3 Likes

is there a chat for this? Also we’ve been exploring transitioning the builder/community groups to a telegram (needs 200+ people) with topics for each group to better streamline conversation.

1 Like

To what frol just commented I would add that another possible route for creating .wasm file from python code would be to use mypyc to convert typed-python code to C. We can use similar to what we use for the JS SDK to go from C to .wasm. Of course, this would mean that developers can make limited use of the Python ecosystem, as many packages are not entirely typed.

2 Likes

Yes, please provide your telegram and I can add you !

1 Like

There is no chat for NEAR Python Tooling Community Group yet. Let’s decide which platform works best for us now (we can revisit it later).

There are a bunch of NEAR communities hosted on Telegram while nearcore uses Zulip. The benefit of Zulip and Github Discussions is that in order to read the messages, you don’t need to sign up, so if users don’t have Zulip/Github account, they still can still read public channels while it is not true for all the other chat platforms listed below.

(You can select and vote for up to 3 options on the poll)

  • Telegram
  • Zulip
  • Github Discussions
  • Discord
  • WeChat
  • Whatsapp
  • Signal
  • No chat, please
  • Other

0 voters

mypyc vs Python experience will be similar to AssemlyScript vs JavaScript, and I would argue it does not make people happy. Most of the time I observed people were tricked by the fact that AssemlyScript looked similar to JavaScript, but then the lack of packages, ugly errors, missing features, and lack of documentation were completely killing the mood, and most of the users strongly considered switching to Rust.

2 Likes