Unlock the power of Sui Indexing Framework for seamless onchain data access! 💪🚀

Customizable Onchain Data Feeds: Unleashing the Power of Sui Indexing Framework 🚀

The Sui Indexing Framework provides a unique opportunity to access Sui’s onchain data in a customizable way through a robust data ingestion framework, as shared by The Sui Blog. This framework allows for the collection of raw onchain data and derived datasets by various software, regardless of whether they operate onchain or offchain.

By leveraging this framework, developers can create personalized data feeds that enable the development of software and products responsive to onchain events. Addressing the limitations of traditional blockchain data structures, which are often not optimized for random data access across their entire history, the customizable data feeds built using this framework empower developers to effectively utilize onchain data for real-time analytics and responsive applications.

Unlocking the Potential of Onchain Data Feeds 🌐

Imagine a scenario where a musician utilizes NFTs to distribute music to their fans. Through creating a non-transferrable NFT collection, each NFT could provide automatic access to an audio file stored in an offchain database upon minting. With a custom indexer developed using the Sui Indexing Framework, tracking the minting transactions associated with these specific NFTs becomes possible, enabling an offchain service to carry out actions like transferring audio files triggered by events monitored through the custom indexer.

This framework is especially beneficial for individuals looking to set up a leaner Full node. Without an indexing solution, Full nodes tend to retain the history of every transaction. By creating a custom indexer with the Sui Indexing Framework, which feeds checkpoint data to be stored separately from the Full node, more efficient infrastructure setups become achievable as Full nodes can be pruned more aggressively.

Furthermore, the Sui Indexing Framework plays a critical role in the development of onchain data dashboards, serving as the foundational element for data ingestion that these applications rely on.

Understanding the Inner Workings 🛠️

The process of data ingestion with the Sui Indexing Framework commences with subscribing to the checkpoint stream from Sui to receive the most recent data. A straightforward approach involves subscribing to a remote store of checkpoint data, such as those provided by Mysten Labs:

Testnet
Mainnet

To process the checkpoint data, a worker function needs to be created. The main application then triggers the worker function whenever it detects an event in the remote store.

use async_trait::async_trait;
use sui_data_ingestion_core::{setup_single_workflow, Worker};
use sui_types::full_checkpoint_content::CheckpointData;

struct CustomWorker;

#[async_trait]
impl Worker for CustomWorker {
    async fn process_checkpoint(&self, checkpoint: CheckpointData) -> Result {
        println!("processing checkpoint {}", checkpoint.checkpoint_summary.sequence_number);
        // custom processing logic
        ...
        Ok(())
    }
}

#[tokio::main]
async fn main() -> Result {
    let (executor, term_sender) = setup_single_workflow(
        CustomWorker,
        "".to_string(),
        0, /* initial checkpoint number */
        5, /* concurrency */
        None, /* extra reader options */
    ).await?;
    executor.await?;
    Ok(())
}

For individuals operating their own Full node, they can establish their checkpoint stream by adding the following checkpoint-executor-config information to the Full node configuration file:

checkpoint-executor-config:
  data-ingestion-dir:

Upon configuration, the Full node dumps checkpoint data into a local directory, and the indexer daemon listens for checkpoint events and processes the data as new checkpoints arrive. The checkpoint data returned is a CheckpointData struct, familiar to current apps. The indexer can then process the data in the same manner as hosted subscriptions.

The Sui Indexing Framework supports both pull-based and push-based processing methods, offering developers the flexibility to choose between a straightforward implementation or reduced latency. This adaptability is essential for applications prioritizing real-time data access and responsiveness.

Delve Deeper into the Possibilities 🌟

Whether you are developing apps that respond to real-time blockchain events or managing general data and infrastructure, the Sui Indexing Framework provides the flexibility and reliability required. For detailed implementation guidance, delve into the Sui Custom Indexer documentation. To witness the framework in action, explore the specialized indexing pipelines utilized by Mysten Labs, SuiNS, and the Sui Bridge.

Hot Take: Embrace the Customizable Onchain Data Experience 🚀

Empower your development journey with the Sui Indexing Framework, unlocking the true potential of onchain data feeds for real-time analytics and responsive applications. Dive into the world of customizable data ingestion and enhance your infrastructure setups with leaner Full nodes. Seize this opportunity to revolutionize your data handling capabilities in the blockchain space!