I would like to bootstrap a space for Python developers that want to collaborate on building Python developers tooling for NEAR.
In short, the state of Python tooling on NEAR is lacking. Yet, there is a raising demand, especially after the success of NEAR SDK for JS.
I am going to update the following sections with useful references. Please, share your progress with the community in the comments to this post.
On-chain programs development in Python
near-sdk-py
It does not exist yet, and there is still a need to build a proof-of-concept that can compile Python code to Wasm that is capable to be executed in NEARās Wasm environment.
Here are the projects that might help to achieve that:
Nuitka - Python compiler that is capable to produce standalone binaries that do not require Python installation on the host.
Nuitka does not compromise on features (latest Python features, C-extensions are supported) and even provides some performance optimizations
cowasm (aka zython) - CPython compiled with Zig to Wasm target.
It supports C-extensions in Wasm, yet it seems that the resulting files are quite beefy (their demo loads 24MB over the network). It is a mix of JS and Wasm files, so it is not clear whether it is possible to package everything into a single Wasm file.
RustPython - Python interpreter implementation that is compilable to Wasm.
Its output is a single wasm, yet it is also quite heavy (their interactive demo loads 25MB Wasm file). Nevertheless, I feel it should be possible to tweak it and selectively include only the necessary features and thus slim it down.
near-api-js provides some abstractions for accounts and keys management, but near-api-js design does not feel Pythonic, so it should not be copied blindly.
End-to-end testing in Python is also needed, so workspaces-js and workspaces-rs needs to be re-implemented in Python.
There are also other ways to get the data from the blockchain:
near-lake-framework-py is a small package that provides a stream of blocks that are finalized on NEAR
Pagoda Enhanced API provides RESTful API to get information about assets (NEAR, FTs, and NFTs) and history in an easy way. It has OpenAPI spec, so low-level Python client can even be automatically generated, yet maintenance is still required.
Hey all, just wanted to tell you about Pyonear, a project Iām working on to provide Python wrappers for near-primitives and other Rust crates that comprise the backbone of near-api-rs.
Iām currently seeking a grant for this, so here are some details on what Iām building and why:
Pyonear (PyO3 + NEAR) is a high-performance Python library for interacting with NEAR, built primarily by wrapping the NEAR Rust crates (e.g. near-primitives) in Python. It uses PyO3 to run NEARās Rust code in Python, providing Python users with fast, familiar and robust solutions for foundational NEAR problems (e.g. public key cryptography, building transactions and parsing accounts). The first release will provide comprehensive Python wrappers for the Rust primitives that enable building, signing and serializing transactions.
Why me?
I have a lot of experience building Python blockchain Python libraries, including libraries that call Rust under the hood. I am the top contributor to solana-py, and I am the author of solders, a Rust-based Python library used by solana-py under the hood.
Why wrap Rust code instead of just using Python?
Here are the main benefits to using the NEAR Rust crates:
The hard problems are already solved. We donāt need to reimplement things like signing and serialization because we are calling code that already takes care of it.
Ease of upgrades: related to the above, itās easier to accommodate changes to the NEAR protocol. For example, when Solana added a new Transaction version, I was able to add support for it in solders before I even understood how it worked.
Performance: while there are many cases where speed isnāt as important as people think, it is still very useful for things like data engineering where N is large. Based on what I observed with Solders, I estimate Pyonear would be 10-100x faster at the things it does than pure Python.
Rust is becoming increasingly popular for Python libraries. Two recent examples are polars, the very fast dataframe library, and pydantic, which recently started using Rust to speed up its core operations.
Do end users need Rust installed?
No! I sometimes see this in comment sections and the like so I thought Iād take the time to clarify it. People who run pip install pyonear are installing a prebuilt wheel for their platform. They donāt need the Rust toolchain at all for this.
This is exciting! I must admit, I was not immediately sold on the idea of building on top of Rust crates even those I am a huge fan of Rust and Python (I use Alpine Linux [which uses musl libc] for Docker images and Mac with M1 arm64 platform, and often I face problems with Python packages unable to build on these platforms), but after I tried solders and faced zero issues, I completely changed my mind. Great job!
Just an FYI. If we succeed with making Python interpreter run in Wasm contract (similarly to our near-sdk-js) then we would need many of these primitives to be re-implemented in pure Python.
is there a chat for this? Also weāve been exploring transitioning the builder/community groups to a telegram (needs 200+ people) with topics for each group to better streamline conversation.
To what frol just commented I would add that another possible route for creating .wasm file from python code would be to use mypyc to convert typed-python code to C. We can use similar to what we use for the JS SDK to go from C to .wasm. Of course, this would mean that developers can make limited use of the Python ecosystem, as many packages are not entirely typed.
There is no chat for NEAR Python Tooling Community Group yet. Letās decide which platform works best for us now (we can revisit it later).
There are a bunch of NEAR communities hosted on Telegram while nearcore uses Zulip. The benefit of Zulip and Github Discussions is that in order to read the messages, you donāt need to sign up, so if users donāt have Zulip/Github account, they still can still read public channels while it is not true for all the other chat platforms listed below.
(You can select and vote for up to 3 options on the poll)
mypyc vs Python experience will be similar to AssemlyScript vs JavaScript, and I would argue it does not make people happy. Most of the time I observed people were tricked by the fact that AssemlyScript looked similar to JavaScript, but then the lack of packages, ugly errors, missing features, and lack of documentation were completely killing the mood, and most of the users strongly considered switching to Rust.
Has the communication channel been decided? Would appreciate if you could share links.
Very interested in this working group. I am coming from the end-user perspective, currently have beginner Python skills and super keen to be able to develop on NEAR.
There are two key things I am interested in and where I think I can contribute;
Communications and recruitment - Iāve just heard about this working group for the first time now after watching Dennis presentation at PyChain. I reckon I can help communicate progress to the world, ideally to onboard more developers who can actually help with implementation and to rally a community of would-be builders like myself - hopefully this can also help is accessing grant for development.
No-code/Low-code Platforms - Iāve been big on no-code platforms for a while. Thereās one in particular I think could have massive potential called Anvil which enable you to build web apps with Python (has all the front end components, integrations, etc.) Would be great if we could set up a smaller working group to explore integrating the current near-api-py with Anvil, maybe create a couple of tutorials, set up some bounties, etc. Could be a big first step while the SDK implementation is underway