[Proposal] Hyperfiles Funding Request (February 2024)

Near Community and nodes Build DAO
1

Title: Hyperfiles Funding Request (February 2024)

Date: January 5, 2024

Prepared by Elijah Spina, PhD

Organization: OpenCann, Hyperfiles, Build DAO

1. Summary

  • Project Description: A self-organizing on-chain knowledge graph leveraging the scientific method for sense-making through empirical hypothesis testing.
  • Funding Requested: $3000 for February 2024
  • Implementation Timeline:

    • July 2023 - August 2023

      • Idea conceived and core schema outlined

    • September 2023

    • October 2023

    • November 2023

      • Recruited additional contributors and stakeholders
      • Designed Hyperfiles attestation framework

    • December 2023

      • Built IPFS adapter
      • Deployed demo Hyperfiles frontend

    • January 2024

      • Create framework for Potlock attestation schemas (fileformats)
      • Potlock Attestations
        • Deploy attestation schemas to potlock.hyperfiles.near
          1. Project
          2. Contributor
          3. Donor
          4. Contribution/Donation
          5. Round Patron
          6. Integration
          7. Partner

    • Deploy resolver contract to validate on-chain conditions for Potlock attestation schemas (e.g. transactions executed by donate.potlock.near).

    • February 2024

      • Deploy job types
      • Deploy OpenCann fileformat library (tripartite DAO governed data marketplace)
        • Data sets
        • Algorithms
        • Impact certificates
      • Integrate Hyperfiles with OpenCann dashboard
      • Publish new documentation containing guides for how to use Potlock and OpenCann fileformat libraries

    • March 2024 and Beyond

      • API(s) and JS SDK
      • Fileformat data validation
      • Marketplace price determination
      • Permissions and encryption for individual records
      • Expand Scientific Fileformat Ontology
      • Implement “Relative Trust Networks”, magnitude of attestation from -1 (complete refutation) to +1 (complete attestation of truth). Expected probability that a statement is true.
      • Implement Hyperbole (exaggerations, untruths, negative data/lack of evidence, saying what something is not, strengthens positive evidence by ruling out alternative hypotheses)
      • Implement Counterfactuals (backtesting reality against alternative hypotheses, “what if?”)
      • Integrate with Voyager and Near FS
      • Integrate AI
        • Create: data validation & categorization
        • Query: Near AI
        • Run: Flowise

2. Background

  • Problem Statement:

    • Developing a universal reputation layer based on empirical hypothesis testing requires automated provenance and contribution tracking of objects at the atomic level during publication and subsequent modification (e.g. via reference, analysis, recombination, etc).
    • Creating liquidity for data requires enabling explicit composability between different types of objects via a coherent knowledge graph.

  • Objective:

    • Become the standard for data composability tooling on Near Protocol, the most robust and scalable multi-chain data availability layer.
    • Create an objective method for evaluating subjective truth and impact of scientific theories that more accurately represents the objectivity of evidence (individual facts) interpreted collectively and iteratively via the scientific method.
      • Observations (data, objects, events)
      • Statements (references/attestations)
      • Facts (statements supported by observations)
      • Hypothesis (testable and falsifiable proposed explanations of the world)
      • Theories (collections of facts that have gathered extensive evidence through hypothesis testing and whose predictions are logically consistent with observations of the natural world)

3. Project Description

  • Scope: build a toolkit & consumer facing app to easily leverage Near data availability between any two sources, whether on or off-chain.

  • Strategy: define conceptual framework to satisfy requirements for different object types in the Hyperfiles knowledge graph

Hyperfiles (b) is an abstracted version of everything.dev (a) and consists of a core library of type schemas that define objects:

a. Thing = (path, blockheight), nodes on the graph
b. Record = contents of a thing, a piece of data

a. Type = a collection of properties of predefined types
b. Fileformat = a collection of fields of predefined types

a. Property = a type of predefined primitive
b. Field = a type of predefined primitive

a. Attestation/Reference = an edge on the graph
b. File = a collection of records (a reference/attestation linking more than one object/record)

a. Metadata = a self-referencing edge that creates context for the object it references
b. Hyperfile = a metadata schema, a collection of files (references/attestations); with explicitly defined external references (source/adapter)

Hyperfiles objects contain embedded references (attestations) that possess the following properties:

  1. All objects MUST have an owner and CAN reference/attest to other objects.
    1a. Object owner and provenance (objects used to derive a given object) are indexed as implicit attestations.
    1b. Hyperfiles explicit attestation objects can reference an unlimited number of objects (compared to EAS which only enables a single attestation to reference two other objects: an account and another attestation).

  2. Objects following the attestation type are indexed as “explicit” attestation objects.

  3. All attestation objects MUST directionally reference other object(s).

  4. Reference type (attestation schema - the UID of the schema used to make the attestation) is embedded in the explicit attestation type.

  5. Users/accounts are just a type of object that are owned by “self”. When users create objects, those objects inherently provide attestations referencing the creator user object.

Steps to be taken:

  1. Build a library of attestation schemas for:
    1a. Potlock (decentralized retroactive public goods funding platform), and,
    1b. OpenCann (tripartite DAO governed data marketplace encompassing data, algorithm, and impact markets).

  2. Build BOS frontend components to enable non-technical users to interact with the Hyperfiles knowledge graph
    2a. Create: publish a Hyperfile (dataset, algorithm, or impact certificate)
    2b. Explore/Query: sort/filter and text search to find data, algorithms, and impact certificates (three separate marketplaces on the frontend but the graphs are linked via references specified during object creation/update)
    2c. Run: run a job, use an algorithm to transform existing data into new object(s), output can be fed to the create component after job is completed to create new objects

  3. Build Hyperfiles API (REST/GraphQL) and Javascript SDK to enable technical users to easily integrate the Hyperfiles knowledge graph into other apps.
    3a. Deploy adapter
    3b. create(fileformat, source, adapter path, content)
    If fileformat = fileformat, create type;
    Else, create thing;
    3c. Query/Explore: Near Query API and social API (social.index)
    3d. Run

  4. Build enhanced query features for hypothesis testing, interpreting evidence, and evaluation of truth.
    4a. Add network statistics to sort/filter parameters and enable the option to view distributions for selected nodes and/or edges.

    • Node Degree Distribution
    • Clustering Coefficient
    • Topological Coefficient
    • Shortest Path Length
    • Neighborhood Connectivity
    • Betweenness Centrality
    • Closeness Centrality
    • Stress Centrality

    4b. Network visualization via force directed layouts

Key Stakeholders: OpenCann, Build DAO, all open web builders

4. Budget

  • Total Cost: a full estimate of project cost has not yet been calculated. The project has been developed to its current state on an ad-hoc basis. Elijah began contributing to the project full-time in January 2024 with the support of Build DAO, under the guidance of James Waugh and Elliot Braem.

  • Funding Request: funding will be used to support developer hours for one (1) full-time contributor, Elijah, for February 2024.

5. Implementation Plan

  • Timeline and Milestones:
    • 1st Week: Deploy job types
      • IO file (JSON specifying IPFS CIDs of input & output data for a job, as well as tool config used)
      • Tool config (JSON specifying instructions and parameters to run a containerized app, e.g. a Docker instance)
    • 2nd Week: Deploy OpenCann fileformat library (tripartite DAO governed data marketplace)
      • Data sets (NFTs and IO files)
      • Algorithms (tool configs)
      • Impact certificates (Hypercerts and derivations)
    • 3rd Week: Deploy BOS components to integrate Hyperfiles with OpenCann dashboard
    • 4th Week: Publish new documentation containing guides for how to use Potlock and OpenCann fileformat libraries

6. Evaluation and Measurement

  • Success Criteria:

    • Total data publish and consume volume (MB)
    • Number of fileformats/schemas created
    • Number of objects created
      • “Data”: events, transactions, strings, integers, CSV/TSV, etc.
      • Algorithms: instructions and parameters to execute containerized apps
      • Attestations (explicit): objects following the Hyperfiles attestation schema

  • Monitoring and Evaluation:

    • Monthly evaluation and reporting of success criteria from tools such as everything.dev, Near Atlas, GitBos, SourceScan, and Nearblocks.io.

7. Risks and Mitigation Strategies

  • Risks:

    1. Competing solutions gain widespread adoption and create a network effect that is difficult to disrupt.
    2. Empirical hypothesis testing doesn’t gain traction as an on-chain knowledge curation mechanism.

  • Mitigation:

    1. Create Hyperfiles adapters for competing knowledge graphs to complement network effect strength of alternative solutions.
    2. Assess strengths and weaknesses of empirical hypothesis testing for on-chain knowledge curation then adjust marketing and PR strategy. Resort to logic, reason, and evidence of technological advancement achieved through application of the scientific method.

8. Conclusion

  • Summary: Hyperfiles is a self-organizing on-chain knowledge graph leveraging the scientific method for sense-making through empirical hypothesis testing. Developing a universal reputation layer based on empirical hypothesis testing requires automated provenance and contribution tracking of objects at the atomic level during publication and subsequent modification (e.g. via reference, analysis, recombination, etc).

Data is potentially the most valuable real-world asset that can easily be represented on-chain due to being digital in nature. Creating liquidity for data requires enabling explicit composability between different types of objects via a coherent knowledge graph.

  • Call to Action: Hyperfiles aims to become the standard for data composability tooling on Near Protocol, the most well-positioned multi-chain data availability layer. We’re building an objective method for evaluating subjective truth and impact of scientific theories that more accurately represents the objectivity of evidence (individual facts) interpreted collectively and iteratively via the scientific method.

The Hyperfiles knowledge graph can be universally generalized and is meant to be a proof of concept system for decentralized data governance. Decentralized coordination requires reputation layers to automate value creation & attribution in the pending world of ubiquitous and powerful AI’s.

9. Appendix

Attachments:

  1. Hyperfiles Frontend
  2. Demo
  3. Official Docs
  4. Ideation and v0 Docs
  5. OpenCann Dashboard
  6. Near Attestation Service
  7. Hypercerts on BOS
  8. Potlock Attestations

Previous Work:

  1. Interpreting Evidence (presentation)
  2. Quantifying Reputation in DAOs (part 1)
  3. Quantifying Reputation in DAOs (part 2)
  4. A microRNA-mRNA Expression Network During Oral Siphon Regeneration in Ciona

External References:

  1. Assembly Theory - Sharma, … Cronin, et al (2023)
  2. Objects as Reference: Toward Robust First Principles of Digital Organization
  3. Building a knowledge graph for biological experiments
  4. Sensemaking Networks: Incorporating science social media into the scientific process
  5. Counterfactual: Generalized State Channels on Ethereum | by Liam Horne | State Channels | Medium
  6. Counterfactual Delegation | Delegatable
  7. Ethereum Attestation Service
  8. Relative Trust Networks - Ethereum Attestation Service
  9. Schema.org
  10. Archetype
  11. NerdBrain
  12. Nostradamus: Prove You Knew It
  13. GitHub - 41hulk/near-attestation: Digital records signed by an individual, company, or organization to verify information about another person, entity, or thing
  14. Attestation [not started] - PotLock
  15. Complex Network Statistics
  16. Force Directed Layouts for Complex Networks
  17. Force Directed Layout in tlDraw

10. Contact Information

2 Likes
2

I’m glad to support this proposal ($2500 in February) because it will help bring more builders into the NEAR ecosystem from various communities of developers, including users of IPFS, GitHub, etc. Also, I’m excited to see Hyperfiles create a solution for impact attestations used on PotLock.

2 Likes
3

Update regarding progress toward January milestones:

Over a dozen unique fields have been mapped to custom types that can be used in creating attestation schemas for events on Potlock. Eight additional schemas have been developed that can be used to reference objects and events during the donation process. We have also begun creating a scoring system to integrate relevant attestations into a trust or impact evaluation algorithm for both individual contributors and projects registered on Potlock.

Hyperfiles Potlock Integration - Main Page
Potlock Impact Evaluation Flow
Potlock Attestation Schemas & Fields
Potlock Impact Scoring System
Potlock Attestation Framework For Impact Evaluation

Attestation schemas are currently being deployed to attestations.near instead of potlock.hyperfiles.near. In the future, schemas may be registered to a fork of socialDB with custom permissions (e.g. to enable optional non-revocability).

Hyperfiles Registry Contract

Additionally, tremendous progress has been made to complete Near Attestation Service (NAS). This will enable anyone to easily create new schemas and publish attestations with a UX similar to Ethereum Attestation Service.

Hyperfiles is exploring implementations of the attestation primitive that leverage the unique benefits of the Near account model and open social graph. These include collaborative type definition, categorization, and use of types in standardizing references (attestations). Implementing conditional logic on top of attestation schemas traditionally requires independently deployed resolver contracts to validate the content of an attestation before granting a user permission to publish a related object using the schema in question. This makes it hard to deploy schemas with conditional logic and creates a large amount of redundant effort by schema deployers whose needs could be addressed by a basic resolver toolkit to choose from.

Thinking about how to make reusable/composable attestation “schema resolvers”:

  1. Resolver factory contracts: e.g. allow users to deploy custom “attester resolvers” where users specify an allowlist of accounts that can use a schema they deployed
  2. BOS feed components that render “attestations” meeting specific conditions (e.g. Near social posts mentioning @Potlock using the hashtag #isMilestone or #isProgress).

We’re developing a suite of reuseable and composable resolver components instead of single-use resolver contracts. The goal is to pre-populate NAS with optional “resolver components” that users can select from and input relevant parameters (e.g. allowlist, deposit required, etc) directly while creating schemas.

Resolver Contracts

On-chain updates have been posted at:

1-19-24
1-26-24
2-4-24

Notable accomplishments include expanding the Hyperfiles type library, Hyperfiles documentation, and improving Near Attestation Service (NAS).

4

Hyperfiles, February 2024 milestone progress report:

Work Completed

Next Steps

Forum Posts

  • 2-19-24: Hyperfiles type library for impact evaluation, attestations, and scientific sensemaking

On-Chain Updates

1 Like