# Introduction

With IPC, dApps can reach planetary scale through recursively scalable subnets, sub-second transactions, robust compute workloads, and highly adaptable WebAssembly runtimes tailored to developer requirements.

## What is IPC?

[Interplanetary Consensus (IPC)](https://www.ipc.space/) is a framework that enables on-demand horizontal scalability of networks, by deploying "subnets" running different consensus algorithms depending on the application's requirements.

Let's break that down.

### What is horizontal scalability and why is it important for dApps?

[Horizontal scalability](https://en.wikipedia.org/wiki/Scalability#Horizontal_or_scale_out) generally refers to the addition of nodes to a system, to increase its performance. For example, adding more nodes to a compute network helps distribute the effort needed to run a single compute task. This reduces cost per task and decreases latency, while improving overall throughput.

In web3, horizontal scalability refers to *scaling* blockchains, for *desired* performance. More specifically, *scaling* the ability of a blockchain to process transactions and achieve consensus, across an increasing number of users, at *desired* latencies and throughput. IPC is one such scaling solution, alongside other popular layer 2 solutions, like [sidechains](https://ethereum.org/en/developers/docs/scaling/sidechains/) and [rollups](https://ethereum.org/en/developers/docs/scaling/#rollups).

For decentralized applications (dApps), there are several key motivations to adopt scaling - performance, decentralization, security. The challenge is that these factors are known to be conflicting goals.

### How does IPC achieve horizontal scalability?

IPC is a scaling solution intentionally designed to achieve considerable performance, decentralization and security for dApps.

It achieves scaling through the permission-less spawning of new blockchain sub-systems, which are composed of subnets.

[Subnets](/concepts/subnets) are organized in a hierarchy, with one parent subnet being able to spawn infinite child subnets. Within a hierarchical subsystem, subnets can seamlessly communicate with each other, reducing the need for cross-chain bridges.

Subnets also have their own specific consensus algorithms, whilst leveraging security features from parent subnets. This allows dApps to use subnets for hosting sets of applications or to [shard](https://en.wikipedia.org/wiki/Shard_\(database_architecture\)) a single application, according to its various cost or performance needs.\
\
IPC-powered networks will also be able to dynamically adjust their throughput by spawning and closing temporary subnets as needed.


# How IPC works

Overview of how IPC works.

## **Interplanetary Consensus (IPC)**

IPC is designed to scale blockchains through the creation of child chains, each with its own unique consensus mechanism, known as a subnet. This scalable architecture is facilitated by a parent chain, which runs a set of Solidity smart contracts—referred to as [actors](https://docs.filecoin.io/basics/the-blockchain/actors)—that manage the creation, recording, and oversight of subnets along with their validator and staking mechanisms.

## **Initialization and Operation of Subnets**

Each subnet is deployed as a standalone chain utilizing [CometBFT](https://docs.cometbft.com/) for its consensus process. The chain is initially configured within the parent chain, where the base validators set and their stakes are also recorded. Following this initial setup, the chain nodes must be started and connected to the parent chain to begin operations. The subnet deploys its own actors (smart contracts) on the [Filecoin Virtual Machine (FVM)](https://docs.filecoin.io/smart-contracts/fundamentals/the-fvm), which manage subnet-specific operations. Additionally, subnets are fully [EVM (Ethereum Virtual Machine)](https://ethereum.org/en/developers/docs/evm/) compatible, allowing for seamless integration with existing Ethereum-based tools and systems. This design allows subnets to operate semi-independently while still being part of a larger network managed by the parent chain.

## **Communication Between Chains**

Communication between the parent chain and its subnets can occur in both directions—top-down (from parent to subnet) and bottom-up (from subnet to parent). This inter-chain communication is facilitated by a component known as the relayer, which transmits messages between actors on different chains. Typical communications include actor-to-actor messages, updates to the validator set, and periodic checkpoints sent from the subnet to the parent chain.

## **Data Handling with IPLD and IPFS**

Instead of transmitting actual data directly, IPC utilizes [InterPlanetary Linked Data (IPLD)](https://spec.filecoin.io/libraries/ipld/) to create data links. These links are then resolved using a resolver that fetches the actual data stored on the [InterPlanetary File System (IPFS)](https://docs.ipfs.tech/), ensuring efficient and secure data management across the network. Additionally, the CometBFT validators run a quorum to agree on the top-down messages from the parent chain, ensuring they can achieve consensus and end up with the same state.

## **Architecture**

For a detailed exploration of the IPC's underlying structure and design principles, please refer to the [Architecture section](https://github.com/consensus-shipyard/ipc/blob/main/docs-gitbook/overview/overview/architecture.md). This section provides in-depth coverage of the technical framework and operational guidelines for IPC.


# Use cases

## Applications of IPC

Here are some practical examples of how IPC improves the performance of dApps:

* **Distributed Computation**: Spawn ephemeral subnets to run distributed computation jobs.
* **Coordination**: Assemble into smaller subnets for decentralized orchestration with high throughput and low fees.
* **Localization**: Leverage proximity to improve performance and operate with very low latency in geographically constrained settings.
* **Partition tolerance**: Deploy blockchain substrates in mobile settings or other environments with limited connectivity.

With better performance, lower fees and faster transactions, IPC can rapidly improve horizontal and vertical markets with decentralized technology:

* **Artificial Intelligence:** IPC is fully compatible with [Filecoin](https://docs.filecoin.io/basics/what-is-filecoin), the world’s largest decentralized data storage. Leveraging Filecoin, IPC can enable distributed computation to power hundreds of innovative AI models.
* **Decentralized Finance (DeFi):** Enabling truly high-frequency trading and traditional backends with verifiability and privacy.
* **Big Data and Data Science:** Multiple teams are creating global-scale distributed compute networks to enable Data Science analysis on Exabytes of decentralized stored data.
* **Metaverse/Gaming:** Enabling real-time tracking of player interactions in virtual worlds.
* **DAOs:** Assemble into smaller subnets for decentralized orchestration with high throughput and low fees. Partition tolerance: Deploy blockchain substrates in mobile settings or other environments with limited connectivity.


# How IPC compares

IPC is a standout framework that strikes a considerable balance, to achieve breakthroughs in scaling.

## **Benefits of IPC Design**

The IPC framework offers several significant benefits:

* **Scalability**: By enabling the creation of subnets, IPC allows for on-demand horizontal scalability, effectively managing increased network load by distributing transactions across multiple chains.
* **Flexibility**: The ability to tailor stakeholder incentives per subnet caters to diverse application needs, optimizing performance and security. However, switching consensus mechanisms may not be straightforward at the current stage.
* **Interoperability**: Full EVM compatibility ensures that subnets can seamlessly integrate with the broader Ethereum ecosystem, leveraging existing development tools and community resources.
* **Decentralization and Security**: The hierarchical structure of subnets supports a robust security architecture while promoting greater decentralization, as subnets can operate independently but are still connected to the main network.

## **Highly customizable without compromising security**

Most L2 scaling solutions today either inherit the L1's security features but don't have their own consensus algorithms (e.g. rollups), or do the reverse (e.g. sidechains). They are also deployed in isolation and require custom bridges or protocols to transfer assets and state between L2s that share a common L1, which are vulnerable to attacks. In contrast, IPC subnets have their own consensus algorithms, inherit security features from the parent subnet and have native cross-net communication, eliminating the need for bridges.

## **Multi-chain interoperability**

* IPC uses [Tendermint Core](https://tendermint.com/core/) as a generic blockchain SMR system, without defaulting to the Cosmos SDK (written in Go). This allows IPC to plug in our own application logic regardless of what language it’s written in: it can be Go, Rust, Java, Haskell, Scheme, etc.
* IPC uses the [Filecoin Virtual Machine (FVM)](https://docs.filecoin.io/smart-contracts/fundamentals/the-fvm) as its transaction execution layer. The FVM is a WASM-based polyglot execution environment for IPLD data and is designed to support smart contracts written in any programming language, compiled to WebAssembly. This enables multi-chain support and gives developers the flexibility to build with familiar tools. Today, IPC is fully compatible with Filecoin and Ethereum and can use either as a rootnet, with more multi-chain support in the roadmap.

## **Compute-Storage Interoperability with Filecoin and more**

IPC is designed to seamlessly integrate with Filecoin and EVM-compatible chains (with more to come), allowing developers to embed IPC subnets within these ecosystems. In particular, IPC unlocks new compute possibilities with the data-centric L1, [Filecoin](https://docs.filecoin.io/basics/what-is-filecoin), which is the largest decentralized storage network. IPC can leverage its storage primitives, like [IPLD](https://spec.filecoin.io/libraries/ipld/) data integration, to deliver enhanced solutions for data availability and more.

## **Increased performance**

IPC’s modular runtime enables the creation of truly flexible blockchains to increase throughput while managing gas efficiency. Developers can dynamically adjust their throughput by spawning and closing temporary subnets as needed.


# Architecture

Architectural components in the IPC framework.

\
![](https://github.com/consensus-shipyard/docs/blob/main/assets/architecture.png?raw=true)

## Validator nodes

Operators of a subnet run a full validator node for both the parent and the child subnet. Requiring the nodes of child subnets to run the nodes of parents is a security parameter to ensure [checkpointed](https://github.com/consensus-shipyard/ipc/blob/main/docs-gitbook/key-concepts/broken-reference/README.md) states of the subnet are appropriately stored, at the right time in the parent.

The following components make up a node:

### Tendermint

[Tendermint Core](https://tendermint.com/) is a byzantine fault tolerant (BFT) consensus engine for blockchains. It acts as a generic [state machine replication](https://en.wikipedia.org/wiki/State_machine_replication) (SMR) in a subnet, talking to other Tendermint instances in the subnet and ensuring a consistent ledger is maintained across validator nodes. It talks to the Application in the node, via ABCI++.

### ABCI++

The [ABCI++](https://members.delphidigital.io/learn/abci) interface is implemented in order to handle the IPC ledger logic and transaction handling, using the [Filecoin Virtual Machine](https://docs.filecoin.io/smart-contracts/fundamentals/the-fvm) (or Ethereum-compatible FVM). The ABCI can pass [checkpointed](https://github.com/consensus-shipyard/ipc/blob/main/docs-gitbook/key-concepts/broken-reference/README.md) headers to the parent and use the ledger to gather relevant signatures.

An ABCI++ application can contact the [IPLD](https://docs.filecoin.io/basics/project-and-community/related-projects#ipld) [resolver & store](https://github.com/consensus-shipyard/ipc/blob/main/docs-gitbook/overview/broken-reference/README.md) to read and write data so that it is IPLD addressable.

### Filecoin Virtual Machine (FVM)

The [FVM](https://docs.filecoin.io/smart-contracts/fundamentals/the-fvm) enables on-chain programmability and is built as a polyglot VM. It is currently compatible with Filecoin and Ethereum and has plans to support more chains in its [roadmap](https://fvm.filecoin.io/).

FVM is included as a transaction execution later in the subnet, allowing use cases enabled by smart contracts to be built on top of subnets.

You may want to build an dApp on FVM that requires a transaction throughput higher than Filecoin provides. You can deploy a subnet that has a higher transaction settling frequency, and deploy the dApp on the subnet with FVM.

### IPC Actors

Communication between subnets on IPC is done by two [actors](https://docs.filecoin.io/basics/the-blockchain/actors) instantiated in each subnet--the *IPC Subnet Actor (ISA)* and the *IPC Gateway Actor (IGA)*.

The IGA is an actor that contains all IPC-related information and logic associated with a subnet that needs to be replicated within the subnet. The IGA implements the hierarchical consensus logic and enforces some level of security for account balances, e.g. by ensuring that it is not possible to withdraw more native tokens from the subnet than were used to fund the subnet.

The ISA is the IGA’s parent-side counterpart; that is, it is deployed to a subnet’s parent and contains all data and logic associated with the particular child subnet. For a subnet to be able to interact with its parent, it needs to have a registered subnet actor on the parent network.

Primitives enabling cross-subnet communication include those for transferring funds between accounts in different subnets, saving checkpoints of a child's state in a parent chain, and submitting transactions on one subnet based on inputs from actors on another subnet.

### Relayer

The role of relayers is to pass messages between parent and child subnets. They have to follow both the parent and the child consensus, subscribe to events, re-package the messages in the appropriate formats and resend them. Relayers can be incentivized. Both parent and child subnets can have an entirely different block structure and consensus, and only the relayers understand both, by being purposefully constructed to act between certain combinations.

### Lotus rootnet

[Lotus](https://lotus.filecoin.io/lotus/get-started/what-is-lotus/) currently serves as the rootnet for IPC subnets. It is the reference implementation for Filecoin nodes.


# Deploy a subnet

Tutorial to deploy your first custom IPC subnet

Ready to test the waters with your first subnet? This guide will deploy a subnet with three local validators orchestrated by `ipc-cli`. This subnet will be anchored to the public [Calibration testnet](https://docs.filecoin.io/networks/calibration/details/). This will be a minimal example and may not work on all systems. The full documentation provides more details on each step.

Several steps in this guide involve running long-lived processes. In each of these cases, the guide advises starting a new *session*. Depending on your set-up, you may do this using tools like `screen` or `tmux`, or, if using a graphical environment, by opening a new terminal tab, pane, or window.

### Step 1: Prepare your system

#### Install the basic requirements for IPC

{% tabs %}
{% tab title="Linux" %}

* Install system packages: `sudo apt install build-essential clang cmake pkg-config libssl-dev protobuf-compiler git curl`.
* Install Rust. See [instructions](https://www.rust-lang.org/tools/install).
* Install cargo-make: `cargo install --force cargo-make`.
* Install Docker. See [instructions](https://docs.docker.com/engine/install/ubuntu/).
* Install Node.js. See [instructions](https://nodejs.org/en/download).
* Install Foundry. See [instructions](https://book.getfoundry.sh/getting-started/installation).

Also install the following dependencies ([details](https://lotus.filecoin.io/lotus/install/prerequisites/#supported-platforms))

```
sudo apt update && sudo apt install build-essential libssl-dev mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl clang hwloc libhwloc-dev wget ca-certificates gnupg -y
```

{% endtab %}

{% tab title="MacOS" %}

* Install Xcode from App Store or terminal: `xcode-select --install`
* Install Homebrew. See [instructions](https://brew.sh/).
* Install dependencies: `brew install jq`
* Install Rust. See [instructions](https://www.rust-lang.org/tools/install). (if you have homebrew installed rust, you may need to uninstall that if you get errors in the build)
* Install Cargo make: `cargo install --force cargo-make`
* Install docker. See [instructions](https://docs.docker.com/desktop/install/mac-install/).
* Install Node.js. See [instructions](https://nodejs.org/en/download).
* Install foundry. See [instructions](https://book.getfoundry.sh/getting-started/installation).
  {% endtab %}
  {% endtabs %}

#### Building

{% hint style="info" %}
NOTE: this step may take a while to compile, depending on OS version and hardware build
{% endhint %}

{% tabs %}
{% tab title="Linux" %}

```
# make sure that rust has the wasm32 target & use stable version of rustc
rustup target add wasm32-unknown-unknown
rustup default stable

# add your user to the docker group
sudo usermod -aG docker $USER && newgrp docker

# clone this repo and build
git clone https://github.com/consensus-shipyard/ipc.git
cd ipc
make

# building will generate the following binaries
./target/release/ipc-cli --version
./target/release/fendermint --version
```

{% endtab %}

{% tab title="MacOS" %}

```
# make sure that rust has the wasm32 target & use stable version of rustc
rustup target add wasm32-unknown-unknown
rustup default stable

# clone this repo and build
git clone https://github.com/consensus-shipyard/ipc.git
cd ipc/contracts
make gen
cd ..
cargo build --release

# building will generate the following binaries
./target/release/ipc-cli --version
./target/release/fendermint --version
```

{% endtab %}
{% endtabs %}

### Step 2: Initialize your config

* Initialize the config

{% tabs %}
{% tab title="Linux/MacOS" %}

```
alias ipc-cli="cargo run -q -p ipc-cli --release --"
ipc-cli config init
```

{% endtab %}
{% endtabs %}

This should have populated a default config file with all the parameters required to connect to calibration at `~/.ipc/config.toml`. Feel free to update this configuration to fit your needs.

You have two options for setting up the contracts: use the public shared contracts or deploy your own private contracts.

#### Option A: Use Public Shared Contracts

The IPC stack is changing rapidly. To make sure you use the latest contracts deployed on Filecoin Calibration:

* Run `nano ~/.ipc/config.toml` to see your configuration

```
keystore_path = "~/.ipc"

[[subnets]]
id = "/r314159"

[subnets.config]
network_type = "fevm"
provider_http = "https://api.calibration.node.glif.io/rpc/v1"
gateway_addr = "<GATEWAY_ADDR>"
registry_addr = "<REGISTRY_ADDR>"
```

* **Replace** the `gateway_addr` and `registry_addr` with the following values. Click on the badges below to take you to the source to copy and paste them or go to [this link](https://github.com/consensus-shipyard/ipc/blob/cd/contracts/deployments/r314159.json).

  [![Gateway Address](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Fconsensus-shipyard%2Fipc%2Fcd%2Fcontracts%2Fdeployments%2Fr314159.json\&query=%24.gateway_addr\&label=Gateway%20Address)](https://github.com/consensus-shipyard/ipc/blob/cd/contracts/deployments/r314159.json) [![Registry Address](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Fconsensus-shipyard%2Fipc%2Fcd%2Fcontracts%2Fdeployments%2Fr314159.json\&query=%24.registry_addr\&label=Registry%20Address)](https://github.com/consensus-shipyard/ipc/blob/cd/contracts/deployments/r314159.json)

#### Option B: Deploy Your Own Private Contracts

If you want to deploy your own custom IPC stack of contracts (recommended for production or private testing):

{% hint style="info" %}
**When to use your own contracts:**

* You want full control over the IPC infrastructure
* You're developing custom functionality
* You need a private testing environment
* You want to avoid potential conflicts with other users on shared contracts
  {% endhint %}

**Step 1: Prepare for contract deployment**

```bash
cd contracts
```

**Step 2: Ensure you have a funded address**

Make sure you have an Ethereum address with funds from the [Calibration faucet](https://faucet.calibration.fildev.network/):

1. Get your address: `ipc-cli wallet show --wallet-type evm`
2. Fund it at: <https://faucet.calibration.fildev.network/>
3. Note your address - you'll need it for the deployment command

{% hint style="info" %}
**Why you need a funded Ethereum address:**

Throughout IPC, you'll use Ethereum-format addresses (0x...) for:

* **Contract deployment** (paying gas fees)
* **Subnet operations** (creating, joining, managing subnets)
* **Cross-subnet messaging** (sending funds between subnets)
* **Checkpoint relaying** (submitting checkpoint transactions)

This address must be funded with testnet tokens and stored in your IPC keystore.
{% endhint %}

**Step 3: Compile contracts**

```bash
# Build the contract artifacts
make build
```

**Step 4: Deploy the contracts using ipc-cli**

```bash
# Deploy the complete IPC contract stack
../target/release/ipc-cli deploy \
  --url https://api.calibration.node.glif.io/rpc/v1 \
  --chain-id 314159 \
  --from <YOUR_ADDRESS_HERE> \
  --contracts-dir ./out/
```

{% hint style="info" %}
**About the Submitter Address (`--from` parameter):**

The submitter address is **your own Ethereum address** (0x format) that will:

* **Deploy the contracts** to the blockchain
* **Pay for gas fees** during deployment
* **Own the deployed contracts** (you'll be the contract owner)

**Requirements:**

* Must be an Ethereum-format address (0x...), not Filecoin format (t4...)
* Must have sufficient funds to pay for deployment gas costs
* Must be in your IPC keystore (the address you got from `ipc-cli wallet show --wallet-type evm`)
  {% endhint %}

{% hint style="warning" %}
**Important:** Replace `<YOUR_ADDRESS_HERE>` with your actual Ethereum address that has funds from the faucet. Use the address from step 2 above.
{% endhint %}

**Step 5: Note the deployed contract addresses**

After deployment, look for output similar to this in your terminal:

```
======================== Deployed Contracts ========================
Registry: 0x0987654321098765432109876543210987654321
Gateway : 0x1234567890123456789012345678901234567890
================================================================
```

{% hint style="warning" %}
**Important:** Save these addresses! You'll need them for configuration and they represent YOUR deployed contracts.
{% endhint %}

**Step 6: Update your IPC configuration**

Run `nano ~/.ipc/config.toml` to edit your configuration file and replace the addresses with your newly deployed contracts:

```toml
keystore_path = "~/.ipc"

[[subnets]]
id = "/r314159"

[subnets.config]
network_type = "fevm"
provider_http = "https://api.calibration.node.glif.io/rpc/v1"
gateway_addr = "0x1234567890123456789012345678901234567890"    # Your GatewayDiamond address
registry_addr = "0x0987654321098765432109876543210987654321"   # Your SubnetRegistryDiamond address
```

**Step 7: Verify deployment (Optional)**

You can verify your contracts are working by checking them on the [Calibration Explorer](https://calibration.filfox.info/en):

* Go to <https://calibration.filfox.info/en>
* Search for your Gateway and Registry contract addresses
* Verify they exist and have been deployed correctly

***

{% hint style="info" %}
**Note:** If you deployed your own contracts, make sure to use YOUR contract addresses in all subsequent steps in this guide, particularly when setting up the validators with `PARENT_GATEWAY` and `PARENT_REGISTRY` environment variables.
{% endhint %}

### Step 3: Set up your wallets

Since we are setting up a subnet with multiple validators, we will create a set of wallets to spawn and interact within the subnet.

{% hint style="info" %}
TIP: Note down wallet and subnet addresses and keys as you go along
{% endhint %}

* Create four different wallets (we recommend a minimum of 4 for BFT security)

```
ipc-cli wallet new --wallet-type evm
ipc-cli wallet new --wallet-type evm
ipc-cli wallet new --wallet-type evm
ipc-cli wallet new --wallet-type evm
```

* You can optionally set one of the wallets as your default so you don't have to use the `--from` flag explicitly in some of the commands:

```
ipc-cli wallet set-default --address <DEFAULT_ETH_ADDR> --wallet-type evm
```

* Go to the [Calibration faucet](https://faucet.calibnet.chainsafe-fil.io/) and get some funds sent to each of your addresses

{% hint style="info" %}
NOTE: you may hit faucet rate limits. In that case, wait a few minutes or continue with the guide and come back to this before step 9. Alternatively, you can send funds from your primary wallet to your owner wallets.
{% endhint %}

{% hint style="info" %}
TIP: If you'd like to import an EVM account into Metamask, you can use export the private key using `ipc-cli wallet export --wallet-type evm --address <ADDRESS>`. More information is available in the [EVM IPC agent support docs](https://github.com/consensus-shipyard/ipc/blob/main/docs/ipc/usage.md#key-management).
{% endhint %}

### Step 4: Create a child subnet

* The next step is to create a subnet under `/r314159` calibration. Remember to set a default wallet or explicitly specify the wallet from which you want to perform the action with the `--from` flag.

```
ipc-cli subnet create --parent /r314159 --min-validator-stake 1 --min-validators 4 --bottomup-check-period 300 --from <PLEASE PUT ACCOUNT ADDRESS> --permission-mode collateral --supply-source-kind native
```

This will output your subnet ID, similar to the following:

<pre><code><strong>/r314159/t410fx2xy6x6idpy6yfywiilp6uitq4eerhpdr72wtmi
</strong></code></pre>

Make a note of the address of the subnet you created because you will use it below.

### Step 5: Join the subnet

Before we deploy the infrastructure for the subnet, we will have to bootstrap the subnet and join from our validators, putting some initial collateral into the subnet and giving our validator address some initial balance in the subnet. For this, we need to send a `join` command from each of our validators from their validator owner addresses.

```
ipc-cli subnet join --from=<PLEASE PUT ADDRESS 1> --subnet=<PLEASE PUT SUBNET ID> --collateral=10 --initial-balance 1
ipc-cli subnet join --from=<PLEASE PUT ADDRESS 2> --subnet=<PLEASE PUT SUBNET ID> --collateral=10 --initial-balance 1
ipc-cli subnet join --from=<PLEASE PUT ADDRESS 3> --subnet=<PLEASE PUT SUBNET ID> --collateral=10 --initial-balance 1
ipc-cli subnet join --from=<PLEASE PUT ADDRESS 3> --subnet=<PLEASE PUT SUBNET ID> --collateral=10 --initial-balance 1
```

### Step 6: Deploy the infrastructure

First, we need to export the validator private keys for all wallets into separate files which we will use to set up a validator node.

```
ipc-cli wallet export --wallet-type evm --address <PLEASE PUT ADDRESS 1> --hex > ~/.ipc/validator_1.sk
ipc-cli wallet export --wallet-type evm --address <PLEASE PUT ADDRESS 2> --hex > ~/.ipc/validator_2.sk
ipc-cli wallet export --wallet-type evm --address <PLEASE PUT ADDRESS 3> --hex > ~/.ipc/validator_3.sk
ipc-cli wallet export --wallet-type evm --address <PLEASE PUT ADDRESS 4> --hex > ~/.ipc/validator_4.sk
```

Let's start our first validator which the rest of the validators will bootstrap from. Make sure you have docker running before running this command.

{% tabs %}
{% tab title="Using Public Shared Contracts" %}

```
cargo make --makefile infra/fendermint/Makefile.toml \
    -e NODE_NAME=validator-1 \
    -e PRIVATE_KEY_PATH=<PLEASE PUT FULL PATH TO validator_1.sk> \
    -e SUBNET_ID=<PLEASE PUT SUBNET ID> \
    -e CMT_P2P_HOST_PORT=26656 \
    -e CMT_RPC_HOST_PORT=26657 \
    -e ETHAPI_HOST_PORT=8545 \
    -e RESOLVER_HOST_PORT=26655 \
    -e PARENT_GATEWAY=`curl -s https://raw.githubusercontent.com/consensus-shipyard/ipc/cd/contracts/deployments/r314159.json | jq -r '.gateway_addr'` \
    -e PARENT_REGISTRY=`curl -s https://raw.githubusercontent.com/consensus-shipyard/ipc/cd/contracts/deployments/r314159.json | jq -r '.registry_addr'` \
    -e FM_PULL_SKIP=1 \
    child-validator
```

{% endtab %}

{% tab title="Using Your Own Deployed Contracts" %}

```
cargo make --makefile infra/fendermint/Makefile.toml \
   -e NODE_NAME=validator-1 \
   -e PRIVATE_KEY_PATH=<PLEASE PUT FULL PATH TO validator_1.sk> \
   -e SUBNET_ID=<PLEASE PUT SUBNET ID> \
   -e CMT_P2P_HOST_PORT=26656 \
   -e CMT_RPC_HOST_PORT=26657 \
   -e ETHAPI_HOST_PORT=8545 \
   -e RESOLVER_HOST_PORT=26655 \
   -e PARENT_GATEWAY=<YOUR_GATEWAY_DIAMOND_ADDRESS> \
   -e PARENT_REGISTRY=<YOUR_SUBNET_REGISTRY_DIAMOND_ADDRESS> \
   -e FM_PULL_SKIP=1 \
   child-validator
```

Replace `<YOUR_GATEWAY_DIAMOND_ADDRESS>` and `<YOUR_SUBNET_REGISTRY_DIAMOND_ADDRESS>` with the addresses you got from the `make deploy-stack` command in Step 2.
{% endtab %}
{% endtabs %}

Once the first validator is up and running, it will print out the relative information for this validator.

{% hint style="info" %}
TIP: Highly recommend documenting that information which will be useful to bootstrap other validators, connect to the IPC subnet on MetaMask, etc.
{% endhint %}

```
#################################
#                               #
# Subnet node ready! 🚀         #
#                               #
#################################

Subnet ID:
 /r314159/t410f6b2qto756ox3qfoonq4ii6pdrylxwyretgpixuy

Eth API:
 http://0.0.0.0:8545

Chain ID:
 3684170297508395

Fendermint API:
 http://localhost:26658

CometBFT API:
 http://0.0.0.0:26657

CometBFT node ID:
 ca644ac3194d39a2834f5d98e141d682772c149b

CometBFT P2P:
 http://0.0.0.0:26656

IPLD Resolver Multiaddress:
 /ip4/0.0.0.0/tcp/26655/p2p/16Uiu2HAkwhrWn9hYFQMR2QmW5Ky7HJKSGVkT8xKnQr1oUGCkqWms
```

You'll need the final component of the `IPLD Resolver Multiaddress` (the `peer ID`) and the `CometBFT node ID` for the next nodes to start.

* ***BOOTSTRAPS***: \<CometBFT node ID for validator1>@validator-1-cometbft:26656

  ```
  // An example
  ca644ac3194d39a2834f5d98e141d682772c149b@validator-1-cometbft:26656
  ```
* ***RESOLVER\_BOOTSTRAPS***: /dns/validator-1-fendermint/tcp/26655/p2p/\<Peer ID in IPLD Resolver Multiaddress>

  <pre><code>// An example
  <strong>/dns/validator-1-fendermint/tcp/26655/p2p/16Uiu2HAkwhrWn9hYFQMR2QmW5Ky7HJKSGVkT8xKnQr1oUGCkqWms
  </strong></code></pre>

Now, run the 2nd validator in a separate terminal.

{% tabs %}
{% tab title="Using Public Shared Contracts" %}

```
cargo make --makefile infra/fendermint/Makefile.toml \
    -e NODE_NAME=validator-2 \
    -e PRIVATE_KEY_PATH=<PLEASE PUT FULL PATH TO validator_2.sk> \
    -e SUBNET_ID=<PLEASE PUT SUBNET ID> \
    -e CMT_P2P_HOST_PORT=26756 \
    -e CMT_RPC_HOST_PORT=26757 \
    -e ETHAPI_HOST_PORT=8645 \
    -e RESOLVER_HOST_PORT=26755 \
    -e BOOTSTRAPS=<PLEASE PUT COMETBFT NODE ID of VALIDATOR-1>@validator-1-cometbft:26656 \
    -e RESOLVER_BOOTSTRAPS=/dns/validator-1-fendermint/tcp/26655/p2p/<PLEASE PUT PEER_ID of VALIDATOR-1> \
    -e PARENT_GATEWAY=`curl -s https://raw.githubusercontent.com/consensus-shipyard/ipc/cd/contracts/deployments/r314159.json | jq -r '.gateway_addr'` \
    -e PARENT_REGISTRY=`curl -s https://raw.githubusercontent.com/consensus-shipyard/ipc/cd/contracts/deployments/r314159.json | jq -r '.registry_addr'` \
    child-validator
```

{% endtab %}

{% tab title="Using Your Own Deployed Contracts" %}

```
cargo make --makefile infra/fendermint/Makefile.toml \
    -e NODE_NAME=validator-2 \
    -e PRIVATE_KEY_PATH=<PLEASE PUT FULL PATH TO validator_2.sk> \
    -e SUBNET_ID=<PLEASE PUT SUBNET ID> \
    -e CMT_P2P_HOST_PORT=26756 \
    -e CMT_RPC_HOST_PORT=26757 \
    -e ETHAPI_HOST_PORT=8645 \
    -e RESOLVER_HOST_PORT=26755 \
    -e BOOTSTRAPS=<PLEASE PUT COMETBFT NODE ID of VALIDATOR-1>@validator-1-cometbft:26656 \
    -e RESOLVER_BOOTSTRAPS=/dns/validator-1-fendermint/tcp/26655/p2p/<PLEASE PUT PEER_ID of VALIDATOR-1> \
    -e PARENT_GATEWAY=<YOUR_GATEWAY_DIAMOND_ADDRESS> \
    -e PARENT_REGISTRY=<YOUR_SUBNET_REGISTRY_DIAMOND_ADDRESS> \
    child-validator
```

{% endtab %}
{% endtabs %}

Now, the 3rd:

{% tabs %}
{% tab title="Using Public Shared Contracts" %}

```
cargo make --makefile infra/fendermint/Makefile.toml \
    -e NODE_NAME=validator-3 \
    -e PRIVATE_KEY_PATH=<PLEASE PUT FULL PATH TO validator_3.sk> \
    -e SUBNET_ID=<PLEASE PUT SUBNET ID> \
    -e CMT_P2P_HOST_PORT=26856 \
    -e CMT_RPC_HOST_PORT=26857 \
    -e ETHAPI_HOST_PORT=8745 \
    -e RESOLVER_HOST_PORT=26855 \
    -e BOOTSTRAPS=<PLEASE PUT COMETBFT NODE ID of VALIDATOR-1>@validator-1-cometbft:26656 \
    -e RESOLVER_BOOTSTRAPS=/dns/validator-1-fendermint/tcp/26655/p2p/<PLEASE PUT PEER_ID of VALIDATOR-1> \
    -e PARENT_GATEWAY=`curl -s https://raw.githubusercontent.com/consensus-shipyard/ipc/cd/contracts/deployments/r314159.json | jq -r '.gateway_addr'` \
    -e PARENT_REGISTRY=`curl -s https://raw.githubusercontent.com/consensus-shipyard/ipc/cd/contracts/deployments/r314159.json | jq -r '.registry_addr'` \
    child-validator
```

{% endtab %}

{% tab title="Using Your Own Deployed Contracts" %}

```
cargo make --makefile infra/fendermint/Makefile.toml \
    -e NODE_NAME=validator-3 \
    -e PRIVATE_KEY_PATH=<PLEASE PUT FULL PATH TO validator_3.sk> \
    -e SUBNET_ID=<PLEASE PUT SUBNET ID> \
    -e CMT_P2P_HOST_PORT=26856 \
    -e CMT_RPC_HOST_PORT=26857 \
    -e ETHAPI_HOST_PORT=8745 \
    -e RESOLVER_HOST_PORT=26855 \
    -e BOOTSTRAPS=<PLEASE PUT COMETBFT NODE ID of VALIDATOR-1>@validator-1-cometbft:26656 \
    -e RESOLVER_BOOTSTRAPS=/dns/validator-1-fendermint/tcp/26655/p2p/<PLEASE PUT PEER_ID of VALIDATOR-1> \
    -e PARENT_GATEWAY=<YOUR_GATEWAY_DIAMOND_ADDRESS> \
    -e PARENT_REGISTRY=<YOUR_SUBNET_REGISTRY_DIAMOND_ADDRESS> \
    child-validator
```

{% endtab %}
{% endtabs %}

And finally, the 4th:

{% tabs %}
{% tab title="Using Public Shared Contracts" %}

```
cargo make --makefile infra/fendermint/Makefile.toml \
    -e NODE_NAME=validator-4 \
    -e PRIVATE_KEY_PATH=<PLEASE PUT FULL PATH TO validator_4.sk> \
    -e SUBNET_ID=<PLEASE PUT SUBNET ID> \
    -e CMT_P2P_HOST_PORT=26956 \
    -e CMT_RPC_HOST_PORT=26957 \
    -e ETHAPI_HOST_PORT=8845 \
    -e RESOLVER_HOST_PORT=26955 \
    -e BOOTSTRAPS=<PLEASE PUT COMETBFT NODE ID of VALIDATOR-1>@validator-1-cometbft:26656 \
    -e RESOLVER_BOOTSTRAPS=/dns/validator-1-fendermint/tcp/26655/p2p/<PLEASE PUT PEER_ID of VALIDATOR-1> \
    -e PARENT_GATEWAY=`curl -s https://raw.githubusercontent.com/consensus-shipyard/ipc/cd/contracts/deployments/r314159.json | jq -r '.gateway_addr'` \
    -e PARENT_REGISTRY=`curl -s https://raw.githubusercontent.com/consensus-shipyard/ipc/cd/contracts/deployments/r314159.json | jq -r '.registry_addr'` \
    child-validator
```

{% endtab %}

{% tab title="Using Your Own Deployed Contracts" %}

```
cargo make --makefile infra/fendermint/Makefile.toml \
    -e NODE_NAME=validator-4 \
    -e PRIVATE_KEY_PATH=<PLEASE PUT FULL PATH TO validator_4.sk> \
    -e SUBNET_ID=<PLEASE PUT SUBNET ID> \
    -e CMT_P2P_HOST_PORT=26956 \
    -e CMT_RPC_HOST_PORT=26957 \
    -e ETHAPI_HOST_PORT=8845 \
    -e RESOLVER_HOST_PORT=26955 \
    -e BOOTSTRAPS=<PLEASE PUT COMETBFT NODE ID of VALIDATOR-1>@validator-1-cometbft:26656 \
    -e RESOLVER_BOOTSTRAPS=/dns/validator-1-fendermint/tcp/26655/p2p/<PLEASE PUT PEER_ID of VALIDATOR-1> \
    -e PARENT_GATEWAY=<YOUR_GATEWAY_DIAMOND_ADDRESS> \
    -e PARENT_REGISTRY=<YOUR_SUBNET_REGISTRY_DIAMOND_ADDRESS> \
    child-validator
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
NOTE:

* Use full path to PRIVATE\_KEY\_PATH, don't path with "\~"
* Do not change values of any port from the ones provided unless you have to
* If you are deploying all validators on a single server, ports will need to be different, as shown in above examples. If you are deploying them from different servers, the ports can be similar.
  {% endhint %}

### Step 7: Interact with your subnet using the IPC CLI

* Make sure `~/.ipc/config.toml` contains the configuration of your subnet in the "Subnet template" section. Uncomment the section and populate the corresponding fields

```
# Subnet template - uncomment and adjust before using
[[subnets]]
id = <PUT YOUR SUBNET ID>

[subnets.config]
network_type = "fevm"
provider_http = "http://localhost:8545/"
gateway_addr = "0x77aa40b105843728088c0132e43fc44348881da8"
registry_addr = "0x74539671a1d2f1c8f200826baba665179f53a1b7"
```

{% hint style="info" %}
NOTE: The ETH addresses for `gateway_addr` and `registry_addr` used when they are deployed in genesis in a child subnet by Fendermint are `0x77aa40b105843728088c0132e43fc44348881da8` and `0x74539671a1d2f1c8f200826baba665179f53a1b7`, respectively, so no need to change them.
{% endhint %}

* Fetch the balances of your wallets using the following command. The result should show the initial balance that you have included for your validator address in genesis:

```
ipc-cli wallet balances --wallet-type evm --subnet=<SUBNET_ID>
```

### Step 8: Run a relayer

IPC relies on the role of a specific type of peer on the network called the relayers that are responsible for submitting bottom-up checkpoints that have been finalized in a child subnet to its parent.

This process is key for the commitment of child subnet checkpoints in the parent, and the execution of bottom-up cross-net messages. Without relayers, cross-net messages will only flow from top levels of the hierarchy to the bottom, but not the other way around.

* Run the relayer process passing the 0x address of the submitter account:

```
ipc-cli checkpoint relayer --subnet <SUBNET_ID> --submitter <RELAYER_ADDR>
```


# Subnets

## Definition

A subnet is a new subsystem that a user can spawn from a parent subnet in a permissionless and on-demand way, depending on scalability requirements. Subnets have separate consensus algorithms and cryptoeconomic rules from their parent subnet. Subnets are firewalled from the parent network.

## Hierarchy trees

Subnets begin with a chosen "rootnet". Rootnets refer to a layer 1 blockchain, such as Filecoin or Ethereum. Child subnets are spawned from the rootnet and the rootnet becomes the parent subnet.

Each subnet can have any number of child subnets, while each child subnet only has one parent subnet. Subnets can scale infinitely, to layer 2 and beyond. A single hierarchy tree begins at the chosen rootnet.

Subnets within a single hierarchy tree have native communication protocols and are able to transfer assets and state without a custom bridge.

## Lifecycle

The lifecycle of a subnet begins when it’s established and ends when the subnet is closed.

At the time of subnet creation, a minimum collateral requirement is set by the subnet creator. A standard fee for the transaction on the parent network will be paid for the transaction that establishes the subnet.

Conditions for closing a subnet include:

* A child subnet cannot be killed until its circulating supply is zero, which can be achieved when all users send their funds back to a parent.
* If all validators leave a subnet even when their are still users of the subnet, the users will have to either run their own validator or wait for a validator to return to the subnet.
* If a bug causes the subnet to fail, there is no way to recover funds in the subnet without a valid checkpoint signed by the latest validator committee.

## Staking

It’s likely that many IPC subnets will be subnets of proof-of-stake chains, or the subnets themselves will be proof of stake chains to other types of chains. For this reason, IPC has native functionality that is intended to handle staking with respect to subnets. These native functionalities include staking and releasing collateral associated with subnet validators and slashing collateral associated with a provably misbehaving subnet validator.

## Fees

### Establishing a subnet

There are a number of fees that are paid when a subnet is established:

* At the time of subnet creation, a minimum collateral requirement is set by the subnet creator.
* A standard fee for the transaction on the parent network will be paid for the transaction that establishes the subnet.
* When a participant or validator (other than the subnet creator) joins the subnet, initial funds for their participation in the subnet should be moved from their respective account in the parent by using the `join` command. This also enables the signing of checkpoint transactions.

### Closing a Subnet

The conditions for closing a subnet are as follows:

* A child subnet cannot be killed untill its circulating supply is zero, which can be achieved when all users send their funds back to a parent.
* If all validators leave a subnet even when their are still users of the subnet, the users will have to either run their own validator or wait for a validator to return to the subnet.
* If a bug causes the subnet to fail, there is no way to recover funds in the subnet without a valid checkpoint signed by the latest validator committee.


# Parent-child interactions

The interaction between two subnets in a parent-child relation, is the basic building block of the recursive IPC hierarchy.

There are following parent-child interactions available in IPC

1\. Creating child subnets in the IPC hierarchy.

2\. Depositing funds from an account in a subnet to an account in its child.

3\. Withdrawing funds from an account in a subnet to an account in its parent.

4\. Checkpointing a subnet’s replicated state in the replicated state of its parent.

5\. Invoking actor functions across subnets.

6\. Removing child subnets from the IPC hierarchy.

## Checkpointing

Checkpointing is a method for a parent subnet to keep a record of the evolution of its child subnet’s state by including snapshots of the child’s state (called checkpoints) in the parent’s state. If, for some reason, the child subnet misbehaves as a whole, agreement can be reached in the parent subnet about how to proceed.

Checkpointed history of a child subnet cannot be reverted as long as a parent subnet operates as expected. Checkpoints are propagated in a recursive way all the way to the rootnet (L1), which makes child subnets benefit from security of their ancestor subnets.

In case of subnet failure, checkpointing enables participants (e.g., former users of the failed subnet) to agree on picking up an older version of the child subnet’s state from before the occurrence of the failure and, say, use that version as the initial state of a new, more robust subnet.\\

### Checkpointing fees

There are a number of fees that are paid during checkpointing:

* When a subnet checkpoints its state to a parent, this is the equivalent of a transaction on the parent. The usual transaction fees of the parent are paid to accomplish this.
* In order for a subnet to be considered *anchored* to the parent, relayers must have sufficient funds in their respective wallets in the parent to be able to pay for a checkpointed transaction.
* When a cross-net transaction is included in a subnet's checkpoint to a parent, the fees for that transaction are distributed as a reward equally among all the relayers that have submitted an instance of that checkpoint.
* Relayers are allowed to submit a checkpoint and eligible for rewards from the commitment of the first checkpoint in, e.g. epoch \`h\`, to the first submission of a checkpoint of epoch \`h+1\`. From this point on, no new valid submissions for checkpoint \`h\` will be accepted.

## Parent finality

Parent finality is a mechanism for proving that a subnet irreversibly reached a certain state.

It is achieved in the following way

* Validators in the child subnet periodically listen to new blocks from the parent. On the implementation level it is performed by Fendermint node of subnet validator subscribing to events via parent's ETH RPC.
* As part of the consensus algorithm in the child, the leader of the consensus proposes the height and hash of the parent’s block that they currently consider final.
* The rest of the validators agree or reject this finality from the parent as part of the process of voting the validity of a block (i.e. validators implicitly agree on the finality of the parent through the block validation).
* As part of the execution of the block, validators implicitly commit the finality seen in the parent. This triggers the execution of all top-down messages from the latest finality to the one committed, as well as any changes on the validator set or collateral that may need to be propagated down.
* When a user performs a top-down message, it is added to a queue in the parent chain. Top-down messages are indexed in the parent by the height where they were committed. This index is used by child subnet validators to determine when a top-down message commitment can be considered as final and their execution can be triggered in the child (when their corresponding finality has been triggered).


# Circulating supply

## Supply sources

At the moment, there are 2 supply sources possible in IPC: *native* and *ERC20*.

The native source means that funds in parent's native coin are moved to the subnet.

The ERC20 source means that there is an ERC20 contracts on the parent, which is configured as subnet's supply source. When funds are to be moved to the subnet, they are locked in ERC20 contract and then minted on the parent.

Other than this, the flows for depositing and withdrawing funds are equivalent.

## Depositing funds

The flow of depositing funds from the address on the parent to the address on the subnet consists of the following steps:

1. One of fund functions is called on the *parent's* Gateway Actor
2. Funds are locked on the parent's subnet.
3. A top-down message is enqueued in the parent to move funds.
4. The subnet's validator fetches top-down messages and executes them, including the message related to depositing funds.
5. As part of the message execution funds are credited to the specified address in the subnet.

## Withdrawing funds

The flow of withdrawing funds from the address on the subnet to the address on the parent consists of the following steps:

1. A release function is called on the *subnet's* Gateway Actor.
2. The release function burns the funds which are going to be withdrawn.
3. A top-down message is enqueued in the subnet to move funds.
4. The relayer, which is responsible for submitting checkpoints containing bottom-up messages, propagates the messages to the parent, including the message related to withdrawing funds.
5. As part of the message execution funds are credited to the specified address in the parent.


# Performing transactions in a subnet

The below is a tutorial on how to issue a transaction from a wallet in a subnet.

### Connecting to a subnet

Assuming the process of launching a custom IPC Subnet with at least one validator node is complete, the custom IPC Subnet is now available for a user (defined here as a wallet on the subnet that is does not necessarily represent a validator) to perform transactions.

* A user would begin by launching Metamask, and manually adding the custom IPC Network to their Metamask networks list using “Add a network manually.”

\
![](https://github.com/consensus-shipyard/docs/blob/main/assets/add_network_manually.png?raw=true)

* Name the network, include the local New RPC URL and relevant ChainID (both of which are provided when you successfully launch the validator node), and name the currency symbol (for Filecoin subnets, tFIL is used).

\
![](https://github.com/consensus-shipyard/docs/blob/main/assets/ipc_local_subnet.png?raw=true)

* Once, the network is added successfully, "Switch to IPC Local Subnet," or whatever you named your subnet.

![](https://github.com/consensus-shipyard/docs/blob/main/assets/network_added.png?raw=true)

* Now, an account on this custom IPC Subnet can be imported to MetaMask. Select “+ Add account or hardware wallet.”

\
![](https://github.com/consensus-shipyard/docs/blob/main/assets/add_account.png?raw=true)

* Select "Import account".

![](https://github.com/consensus-shipyard/docs/blob/main/assets/import_account.png?raw=true)

* Enter the private key for the account, which is generated when establishing the subnet, and select “import.”

\
![](https://github.com/consensus-shipyard/docs/blob/main/assets/import.png?raw=true)

Note: The private key can be retrieved on the command line by the user that deployed the subnet using the following command:

```
ipc % cat ~/.ipc/validator_1.sk
```

The user now has a wallet address available for transactions in the custom IPC subnet.

### Deploying Smart Contracts in a Subnet

Consider the deployment of a simple smart contract that issues ERC20 tokens in the IPC subnet --these tokens will be acquired by a user's IPC subnet MetaMask wallet. The steps for achieving this are as follows:

* Open Remix and a simple contract .sol file for issuing ERC20 tokens.

![](https://github.com/consensus-shipyard/docs/blob/main/assets/token.png?raw=true)

* For the environment, connect the IPC Subnet using the “Injected provider - Metamask” custom network. Connect the relevant account on the IPC Subnet.

![](https://github.com/consensus-shipyard/docs/blob/main/assets/injected_provider.png?raw=true)

* Select the relevant account and set the gas limit.

![](https://github.com/consensus-shipyard/docs/blob/main/assets/injected_provider.png?raw=true)

* Compile and deploy the contract. Mint new tokens by specifying the address you want the tokens to go to, and the number of tokens you wish to mint. Click “Transact.”

![](https://github.com/consensus-shipyard/docs/blob/main/assets/deploy_run.png?raw=true)

* Copy the token address and import the token to the MetaMask wallet on the custom IPC subnet.

![](https://github.com/consensus-shipyard/docs/blob/main/assets/deploy_metamask.png?raw=true)

* The funds of the new token should be displayed, indicating a successful transaction was performed on the IPC subnet.

### Block Explorers

Currently, browsing a custom subnet using a block explorer is not possible. However, a block explorer that can browse custom subnets should be released soon.<br>


# Customizing a subnet

Tutorial to add a new syscall to Filecoin Virtual Machine and create a new built-in actor in IPC to activate the syscall

IPC is uniquely hyper customizable as a scalability framework. Subnets are highly customizable and can be temporal, allowing subnet operators to spin up customized subnets for various needs, including modular consensus, gas option, configurable chain primitives, customized features vis pluggable syscalls and build-in actor.

In this tutorial, we will focus on how to extend features to your IPC subnet by customizing syscalls that are 'pluggable' as needed.

IPC uses the [Filecoin Virtual Machine (FVM)](https://docs.filecoin.io/smart-contracts/fundamentals/the-fvm) as its execution layer, which is a WASM-based polyglot VM. FVM exposes all system features and information through syscalls, as part of its SDK. The FVM SDK is designed to be pluggable to enable user-defined custom features with syscalls, while implementing default features from FVM kernel. Use cases includes:

* Extending chain-specific syscalls once IPC supports more root chains. Because other chains may have their own special syscalls different as Filecoin (proof validation, etc.).
* Extending features to support better development tools. E.g. adding special debugging syscalls, adding randomness syscalls, and supporting more ECC curve, etc.

## Pre-requisite knowledge for tutorial

* [ref-fvm](https://github.com/filecoin-project/ref-fvm)
* [fvm syscall APIs](https://docs.rs/fvm_sdk/latest/fvm_sdk/sys/index.html)
* [IPC/fendermint](https://github.com/consensus-shipyard/ipc) implementation
* Rust
* [Ambassador](https://crates.io/crates/ambassador)

## Instructions

These instructions describe the steps required to create a new kernel which implements a new syscall along with an example built-in actor that shows how you would call that syscall. Full example [here](https://github.com/consensus-shipyard/ipc/pull/630).

{% hint style="info" %}
TIP: For clarity, the instructions may have skipped certain files (like long `Cargo.toml` files) so make sure to refer to the above full example, if you want to follow along and get this compiling on your machine.
{% endhint %}

### **1. Define the custom syscall**

* In this example, we will be creating a simple syscall which accesses the filesystem. Inside syscalls, you can run external processes, link to rust libraries, access network, call other syscalls, etc.
* We’ll call this new syscall `my_custom_syscall`and its defined as follows:

  ```rust
  pub trait CustomKernel: Kernel {
      fn my_custom_syscall(&self) -> Result<u64>;
  }
  ```

  [fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L23](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L23)
* Define a struct `CustomKernelImpl` which extends `DefaultKernel` . We use the `ambassador` crate to automatically delegate calls which reduces the boilerplate code we need to write. Here we simply delegate all calls to existing syscall to the `DefaultKernel`.

  ```rust
  #[derive(Delegate)]
  #[delegate(IpldBlockOps, where = "C: CallManager")]
  #[delegate(ActorOps, where = "C: CallManager")]
  #[delegate(CryptoOps, where = "C: CallManager")]
  #[delegate(DebugOps, where = "C: CallManager")]
  #[delegate(EventOps, where = "C: CallManager")]
  #[delegate(MessageOps, where = "C: CallManager")]
  #[delegate(NetworkOps, where = "C: CallManager")]
  #[delegate(RandomnessOps, where = "C: CallManager")]
  #[delegate(SelfOps, where = "C: CallManager")]
  #[delegate(SendOps<K>, generics = "K", where = "K: CustomKernel")]
  #[delegate(UpgradeOps<K>, generics = "K", where = "K: CustomKernel")]
  pub struct CustomKernelImpl<C>(pub DefaultKernel<C>);
  ```

  [fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L27](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L27)

### **2. Implementing all necessary functions for the syscall**

* Implement `my_custom_syscall`

  Here is where we implement our custom syscall:

```rust
impl<C> CustomKernel for CustomKernelImpl<C>
where
    C: CallManager,
    CustomKernelImpl<C>: Kernel,
{
    fn my_custom_syscall(&self) -> Result<u64> {
        // Here we have access to the Kernel structure and can call
        // any of its methods, send messages, etc.

        // We can also run an external program, link to any rust library
        // access the network, etc.

        // In this example, lets access the file system and return
        // the number of paths in /
        let paths = fs::read_dir("/").unwrap();
        Ok(paths.count() as u64)
    }
}
```

[fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L42](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L42)

* Next we need to implement the `Kernel` trait for the new `CustomKernelImpl`. You can treat this as boilerplate code and you can just copy it as is:

```rust
impl<C> Kernel for CustomKernelImpl<C>
where
    C: CallManager,
{
    type CallManager = C;
    type Limiter = <DefaultKernel<C> as Kernel>::Limiter;

    fn into_inner(self) -> (Self::CallManager, BlockRegistry)
    where
        Self: Sized,
    {
        self.0.into_inner()
    }

    fn new(
        mgr: C,
        blocks: BlockRegistry,
        caller: ActorID,
        actor_id: ActorID,
        method: MethodNum,
        value_received: TokenAmount,
        read_only: bool,
    ) -> Self {
        CustomKernelImpl(DefaultKernel::new(
            mgr,
            blocks,
            caller,
            actor_id,
            method,
            value_received,
            read_only,
        ))
    }

    fn machine(&self) -> &<Self::CallManager as CallManager>::Machine {
        self.0.machine()
    }

    fn limiter_mut(&mut self) -> &mut Self::Limiter {
        self.0.limiter_mut()
    }

    fn gas_available(&self) -> Gas {
        self.0.gas_available()
    }

    fn charge_gas(&self, name: &str, compute: Gas) -> Result<GasTimer> {
        self.0.charge_gas(name, compute)
    }
}
```

[fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L61](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L61)

### **3. Link syscalls to the kernel**

* Next we need to implement the `SyscallHandler` trait for the `CustomKernelImpl` and link all the syscalls to that kernel. We need to explicitly list each of the syscall traits (ActorOps, SendOps, etc) manually here in addition to the `CustomKernel` trait. Then inside the `link_syscalls` method we plug in the actor invocation to the kernel function that should process that syscall. We can link all the existing syscalls using the `link_syscalls` on the `DefaultKernel` and then link our custom syscall.

<pre class="language-rust"><code class="lang-rust"><strong>impl&#x3C;K> SyscallHandler&#x3C;K> for CustomKernelImpl&#x3C;K::CallManager>
</strong>where
    K: CustomKernel
        + ActorOps
        + SendOps
        + UpgradeOps
        + IpldBlockOps
        + CryptoOps
        + DebugOps
        + EventOps
        + MessageOps
        + NetworkOps
        + RandomnessOps
        + SelfOps,
{
 fn link_syscalls(linker: &#x26;mut Linker&#x3C;K>) -> anyhow::Result&#x3C;()> {
        DefaultKernel::&#x3C;K::CallManager>::link_syscalls(linker)?;

        linker.link_syscall("my_custom_kernel", "my_custom_syscall", my_custom_syscall)?;

        Ok(())
    }
}
</code></pre>

[fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L112](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L112)

### **4. Expose the customized syscall**

* Once this function is linked to a syscall and exposed publicly, we can use this syscall by calling `my_custom_kernel.my_custom_syscall`

```rust
pub fn my_custom_syscall(
    context: fvm::syscalls::Context<'_, impl CustomKernel>
) -> Result<u64> {
    context.kernel.my_custom_syscall()
} 
```

[fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L136](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/examples/mycustomkernel.rs#L136)

### **5. Replace existing IPC kernel with new custom kernel**

* Since the customized syscall is implemented in a `CustomKernelImpl` which extends and implements all the behaviors for `DefaultKernel`, we can plug it into IPC instead of `DefaultKernel`\\
* To use this kernel in fendermint code, replace `DefaultKernel` with `CustomKernelImpl` for the `executor` declaration in `fendermint/vm/interpreter/src/fvm/state/exec.rs`

```rust

use crate::fvm::examples::mycustomkernel::CustomKernelImpl;

executor: DefaultExecutor<CustomKernelImpl<DefaultCallManager<DefaultMachine<DB, FendermintExterns>>>,
```

[fendermint/vm/interpreter/src/fvm/state/exec.rs#L86](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/state/exec.rs#L86)

### **6. Use syscall in your IPC subnet**

* Now, we are all set to use the custom syscall in the IPC subnet. The custom syscall can be called in IPC actors to utilize the extended feature. For this tutorial, we can create a simple actor to demonstrate how to import and call the custom syscall and then confirm that its working correctly.
* Let’s create a `customsyscall` folder in `ipc/fendermint/actors/` and then create a file called [actor.rs](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/actors/customsyscall/src/actor.rs) in that new folder. Here we want to create a very simple actor, which when invoked (received a message on its Invoke method) will call the new syscall and return its value:

```rust
fvm_sdk::sys::fvm_syscalls! {
    module = "my_custom_kernel";
    pub fn my_custom_syscall() -> Result<u64>;
}

pub struct Actor;
impl Actor {
    fn invoke(rt: &impl Runtime) -> Result<u64, ActorError> {
        rt.validate_immediate_caller_is(std::iter::once(&SYSTEM_ACTOR_ADDR))?;

        unsafe {
            let value = my_custom_syscall().unwrap();
            Ok(value)
        }
    }
}

impl ActorCode for Actor {
    type Methods = Method;

    fn name() -> &'static str {
        CUSTOMSYSCALL_ACTOR_NAME
    }

    actor_dispatch! {
        Invoke => invoke,
    }
}
```

[fendermint/actors/customsyscall/src/actor.rs#L14](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/actors/customsyscall/src/actor.rs#L14)

* Even though this is Rust code, IPC will compile it as a Wasm target and then run the compiled Wasm code inside FVM as an actor. However, we want to share some of the code between Wasm and IPC, such as the actor name `CUSTOMSYSCALL_ACTOR_NAME` and the `Invoke` method enum. We will define these in a separate file called [`shared.rs`](http://shared.rs) as follows:

```rust
use num_derive::FromPrimitive;

pub const CUSTOMSYSCALL_ACTOR_NAME: &str = "customsyscall";

#[derive(FromPrimitive)]
#[repr(u64)]
pub enum Method {
    Invoke = frc42_dispatch::method_hash!("Invoke"),
}
```

[fendermint/actors/customsyscall/src/shared.rs](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/actors/customsyscall/src/shared.rs#L1)

* We next need to write a [`lib.rs`](http://lib.rs) file which exports the shared code and only compiles `actor.rs` if we are building the Wasm actor.

```rust
#[cfg(feature = "fil-actor")]
mod actor;
mod shared;

pub use shared::*;
```

[fendermint/actors/customsyscall/src/lib.rs](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/actors/customsyscall/src/lib.rs#L1)

{% hint style="info" %}
NOTE: There are several other files you need to change to compile this actor and package it with the other actors that IPC uses. Please refer to the full example [here](https://github.com/consensus-shipyard/ipc/pull/630) for the following other files you need to change:

* `fendermint/actors/customsyscall/Cargo.toml`: The package for your new actor and all its dependencies
* `fendermint/actors/Cargo.toml`: Add your new actor as a Wasm target
* `fendermint/actors/build.rs`: Include your new actor in the `ACTORS` array so it will get included in the bundle.
* `fendermint/actors/src/manifest.rs`: Add your new actor in the `REQUIRED_ACTORS` array so we can confirm it was correctly bundled on IPC startup
* `fendermint/vm/actor_interface/src/customsyscall.rs`: A macro which assigns an ID to your new actor and declares constants for accessing it by ID and Address
* `fendermint/vm/actor_interface/src/lib.rs`: export the constants to IPC
  {% endhint %}

### **7. Load and deploy actor at genesis**

We have so far created a new kernel and syscall, switched IPC to use that kernel and created an actor which calls the new syscall. However, in order to call this actor in IPC, we must load it from the custom\_actors\_bundle.

* To do this open `fendermint/vm/interpreter/src/fvm/genesis.rs` file and in the `init` function add our customsyscall actor right after creating the `chainmetadata` actor:

```rust
// Initialize the customsyscall actor which gives an example of calling a custom syscall
state
    .create_custom_actor(
        fendermint_actor_customsyscall::CUSTOMSYSCALL_ACTOR_NAME,
        customsyscall::CUSTOMSYSCALL_ACTOR_ID,
        &EMPTY_ARR,
        TokenAmount::zero(),
        None,
    )
    .context("failed to create customsyscall actor")?;
```

[fendermint/vm/interpreter/src/fvm/genesis.rs#L251](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/genesis.rs#L251)

Your actor has now been deployed and we should be able to send it messages!

### **8. Invoke the actor**

In the last step in this tutorial we will send our customsyscall actor messages which will cause it to run its Invoke method and execute the custom syscall. Here, we will simply call it for every new block height. Go to `fendermint/vm/interpreter/src/fvm/exec.rs` and inside the `begin` function add the following code:

```rust
let msg = FvmMessage {
    from: system::SYSTEM_ACTOR_ADDR,
    to: customsyscall::CUSTOMSYSCALL_ACTOR_ADDR,
    sequence: height as u64,
    gas_limit,
    method_num: fendermint_actor_customsyscall::Method::Invoke as u64,
    params: Default::default(),
    value: Default::default(),
    version: Default::default(),
    gas_fee_cap: Default::default(),
    gas_premium: Default::default(),
};

let (apply_ret, _) = state.execute_implicit(msg)?;

if let Some(err) = apply_ret.failure_info {
    anyhow::bail!("failed to apply customsyscall message: {}", err);
}

let val: u64 = apply_ret.msg_receipt.return_data.deserialize().unwrap();
println!("customsyscall actor returned: {}", val);
```

[fendermint/vm/interpreter/src/fvm/exec.rs#L115](https://github.com/consensus-shipyard/ipc/blob/98497363a10e08236325e6d5c52755b9fcd52958/fendermint/vm/interpreter/src/fvm/exec.rs#L115)

This code sends a message to the `customsyscall` actor and parses it output after it has been executed. We print out the return value from the actor, which will be the return value of our custom syscall.

### **9. Test your actor**

In order to see this working end to end in IPC, you can run one of our integration tests. These tests run IPC in docker containers so make sure to have docker installed on your machine if you are following along.

We must first need to build a new docker container for the fendermint image which will contain all the code you have added so for. To do this run:

```bash
cd fendermint
make docker-build
```

After the fendermint docker image has been built, you can run one of the integration tests

```bash
cd fendermint/testing/smoke-test
# creates the docker containers
cargo make setup
# runs the integration test
cargo make test
```

View fendermint logs and see the output generated by calling the `customsyscall` actor in each epoch:

```bash
docker ps
CONTAINER ID   IMAGE                       COMMAND
8da423d8bb1e   fendermint:latest           "fendermint --networ…"
...
```

View the docker logs:

```bash
docker logs 8da423d8bb1e
...
customsyscall actor returned: 21
```

You can now run `cargo make teardown` to stop the containers.


# Upgrading a subnet

## IPC Upgrades

As IPC evolves, upgrades are necessary to introduce new features, enhance security, improve scalability, and address new challenges. However, it is critical that all upgrades seamlessly transition without causing forks or consensus failures. This ensures that all nodes in the network remain synchronized.

In this document, we describe how IPC achieves full upgradability of all its components and ensures seamless progression of the blockchain network without compromising consensus.

## IPC Actor upgrades

The on-chain logic for IPC is implemented in two contracts, the IPC gateway and the subnet actor, as well as an auxiliary contract use for UX purposes, the registry.

* The gateway is a singleton actor that lives in every IPC subnet and implements the common logic for IPC. It is responsible for managing the collateral of subnet, enforcing the firewall requirement, and the cross-net interactions.
* The subnet actor is a user-defined actor that implements the specific logic of a subnet. This contract is deployed in the parent from which the child subnet wants to be deployed. There is one subnet-actor for each child subnet in the parent.
* The subnet registry behaves as a subnet actor factory that offers users a convenient way of deploying instances of the reference implementation of the subnet actor in a network.

The IPC actors are implemented using the diamond pattern to make it easier to upgrade them. The following sections describes how to upgrade each of these actors

### Upgrading the Subnet actor

Once you have you [deployed your own subnet](https://docs.ipc.space/quickstarts/deploy-a-subnet) you might want to make changes to the subnet actor, for example make custom logic changes, add new functionality, fix bugs, etc.

The code for the subnet actor is located in `contracts/src/subnet` which you can directly edit to make your changes. Once ready, you run the following steps to upgrade your already deployed subnet actor:

1. First, you must know your subnet ID which was returned when you created a child subnet (using `ipc-cli subnet create`). Lets say when you created your subnet that your subnet ID was: `/r314159/t410fkp4r67rks3ok4bvbn2rjojhkbig2rwvp4nmor5q`
2. We must convert the subnet ID to an ETH address. To do this, visit the [Beryx address converter](https://beryx.zondax.ch/address_converter) and input the `t410` address in the Filecoin address input, in this example this would be `t410fkp4r67rks3ok4bvbn2rjojhkbig2rwvp4nmor5q`. Click `Convert to ETH` and it will compute the Ethereum address as `0x53f91f7e2a96dcae06a16ea29724ea0a0da8daaf`.
3. You must set the `RPC_URL` and `PRIVATE_KEY` environmental variables to point to your network provider and the private key of the address you want to use for the deployment, respectively
4. Now you should be able to upgrade the subnet actor by running the following command (change NETWORK if you have deployed your subnet on another network)

```
> cd contracts/
> make upgrade-sa-diamond SUBNET_ACTOR_ADDRESS=0x53f91f7e2a96dcae06a16ea29724ea0a0da8daaf NETWORK=calibrationnet
```

A successfull output looks like:

```
Successfully generated 84 typings!
Compiled 14 Solidity files successfully (evm target: paris).

Facet Bytecode Not Found:
---------------------------------
Facet Name: SubnetActorGetterFacet
Libraries:
  - SubnetIDHelper: 0x1CaA8B5a18d0401f2089aC82063AaecA08142744
Address: 0xcb4EE21d081e0cBdd3D82D94114bEAa497f11C4a


Diamond Facet Upgrade:
-----------------------------------
Diamond Address: 0x53f91f7e2a96dcae06a16ea29724ea0a0da8daaf
Replacement Facet Name: SubnetActorGetterFacet


Deployment Status:
-------------------------
New replacement facet (SubnetActorGetterFacet) deployed.


Facet Bytecode Not Found:
---------------------------------
Facet Name: SubnetActorCheckpointingFacet
Libraries:

Address: 0xd5C97F52030DFE7595b6cD92868EcE5dC175C224


Diamond Facet Upgrade:
-----------------------------------
Diamond Address: 0x53f91f7e2a96dcae06a16ea29724ea0a0da8daaf
Replacement Facet Name: SubnetActorCheckpointingFacet


Deployment Status:
-------------------------
New replacement facet (SubnetActorCheckpointingFacet) deployed.
```

If you run the `make upgrade-sa-diamond` command again, you should see no output which would also confirm that the upgrade has succeeded.

## Fendermint upgrades

Fendermint incorporates a builtin `UpgradeScheduler`, enabling the execution of hardcoded `Upgrade` migrations at predetermined block heights to advance the on-chain state. Fendermint also supports functionality to halt at predetermined `halt_height` in order to switch binary versions.

### Upgrade scheduler

At the start of processing each block, Fendermint checks the `UpgradeScheduler` whether it contains an `Upgrade` for that block height. Only a single `Upgrade` can be scheduled for each block height. If an upgrade is found to be scheduled, it will execute the `migration` function that is associated with that `Upgrade`. If the upgrade migration returns an error it will crash the node.

The migration has access to the `state_tree` and `blockstore` which allows the user to write migrations such as patching state, sending messages, deploying solidity contracts, deploying new WASM actors, etc.

An `Upgrade` is defined as follows:

```rust
/// a function type for migration
pub type MigrationFunc = fn(state: &mut FvmExecState) -> anyhow::Result<()>;

pub struct Upgrade {
    chain_id: ChainID,
    block_height: BlockHeight,
    new_app_version: Option<u64>,
    migration: MigrationFunc,
}
```

Fields:

* `chain_id`: The chain id the upgrade should be applied to.
* `block_height`: The block height where the upgrade should be applied to and the `migration` function executed.
* `new_app_version`: Indicates a new application protocol version if the upgrade introduces backward-incompatible changes.
* `migration`: The migration function that is executed when applying the upgrade. The migration function is passed the `FvmExecState` which gives access to the `state_tree`, `block_store`, ability to send messages and more.

Fendermint supports scheduling multiple upgrades through its `UpgradeScheduler` API which is defined as follows:

```rust
struct UpgradeKey(ChainID, BlockHeight);

pub struct UpgradeScheduler {
  upgrades: BTreeMap<UpgradeKey, Upgrade>,
}

impl UpgradeScheduler {
  // add an upgrade to the upgrade schedule
  fn add(&mut self, upgrade: Upgrade<DB>) -> Result<()> {
    // implementation left out
  }

  // check if the current chain_id has a scheduled upgrade at the given height
  fn get(&self, chain_id: ChainID, height: BlockHeight) -> Option<&Upgrade<DB>> {
    // implementation left out
  }
}
```

As an example, if we want to create an `UpgradeScheduler` with a single `Upgrade`, we could write something like:

```rust
// create a new upgrade scheduler
let mut scheduler = UpgradeScheduler::new();

// add a new upgrade at predetermined block height
let upgrade_func = |state: &mut FvmExecState<DB>| {
  // implement our upgrade logic here
  Ok()
}
let upgrade = Upgrade::new(chain_name, block_height, app_version, upgrade_func);
scheduler.add(upgrade);

// when initializing the FvmMessageInterpreter, specify the upgrade schedule
let interpreter = FvmMessageInterpreter::<DB, _>::new(
  ...
  scheduler,
);
```

We have written guides showing several examples of using the UpgradeScheduler API, which you'll find on the sidebar.

### Halting at predetermined height

Although the `UpgradeScheduler` supports multiple different types of upgrades, it can not support non-state related changes such as upgrading Fendermint dependencies (such as FVM), adding new syscalls, etc.

To support these use cases, Fendermint includes a `halt_height` config which when set (is non zero), will halt and exit Fendermint with a specific exit code (`2`) once it reaches that block height.

This enables node operators to replace the Fendermint version at a predetermined future block height with a new Fendermint version containing the new upgrade. Optionally, the new version can specify a new `Upgrade` at the specified `halt_height` in case a state migration is also required.

This ensures that all nodes run the same Fendermint version for every block height which prevents accidental forking or consensus failures.

#### Instructions

These instructions detail the steps for utilizing the halting mechanism to synchronize Fendermint application switching with newer versions.

{% hint style="info" %}
NOTE: Node operators should always check the fendermint exit code. If fendermint exits with code 2, which indicates reaching the halting height, disable automatic restarts. Instead, follow steps 4 and 5 in this section.
{% endhint %}

**1. Determine the future halt\_height**

Node operators collaborate on which future block height should be used as `halt_height`. Selecting the halting height is totally up to the operators, but let say for example purposes that the halt\_height was agreed to be set to `10000`.

**2. Update the halt\_height config**

Now, each operator needs to edit the Fendermint config (`.fendermint/config/default.toml`) and make sure it contains the following line

```bash
> cat .fendermint/config/default.toml | grep halt_height
halt_height = 10000
```

Note that Fendermint needs to be restarted before the config changes take effect.

**3. Wait until halt\_height**

Once Fendermint reaches the `halt_height`, it will exit and log it reached the halting height.

```bash
...
2024-03-19T13:57:52.904824Z  INFO fendermint/app/src/app.rs:677: event=ProposalProcessed is_accepted=true block_height=9999 block_hash="55E10F55140A995CEE4B540776EF5916CBC5C9D81137FED6788C82F2ACBEA28D" num_txs=0 proposer="BA9ADDB0F298912DB80757EA743C3672AE918FF8"
2024-03-19T13:57:53.006128Z  INFO fendermint/app/src/app.rs:786: event=NewBlock block_height=449
2024-03-19T13:57:53.980298Z  INFO fendermint/app/src/app.rs:677: event=ProposalProcessed is_accepted=true block_height=10000 block_hash="7B5830D4F138D4329909B64AE0A71296271AD65C83158DDED59DEB3C01DDC650" num_txs=0 proposer="BA9ADDB0F298912DB80757EA743C3672AE918FF8"
2024-03-19T13:57:54.036789Z  INFO fendermint/app/src/app.rs:701: Stopping node due to reaching halt height height=10000
```

The node operator can check if Fendermint exited due to reaching halting height by checking if its exit code is `2`.

```bash
> echo $?
2
```

**4. Reset the halt\_height**

We must change the Fendermint `halt_height` to 0 (or some future block height if there is another upgrade planned).

```bash
> cat .fendermint/config/default.toml | grep halt_height
halt_height = 0
```

**5. Start the new Fendermint version**

We can now start the new version of Fendermint which contains the upgrade we need.


# Example: Patching actor state

In this example, we show how you can write an upgrade migration which patches the `ActorState` of an existing actor stored on chain.

More specifically, in this example we want to patch the state of the `chainmetadata` actor which was deployed at genesis. This actor is used to store blockhashes of the previous blocks on chain. Internally, this actor has the following state:

```rust
// in fendermint/actors/chainmetadata/src/shared.rs
pub struct State {
    // the AMT root cid of blockhashes
    pub blockhashes: Cid,
    // the maximum size of blockhashes before removing the oldest epoch
    pub lookback_len: u64,
}
```

At genesis, this actor was deployed with `lookback_len` of 256. In this migration, we want to change the `lookback_len` to 512 to extend the lookback history.

Inside this migration function, we need to retrieve the actor state associated with the `chainmetadata` actor, update its `lookback_len` to 512, save the new state to the block store and then update the actor state in the state tree.

Our migration function is defined as follows:

```rust
pub fn patch_actor_state_func(state: &mut FvmExecState<NamespaceBlockstore>) -> anyhow::Result<()> {
    let state_tree = state.state_tree_mut();

    // get the ActorState from the state tree
    //
    let actor_state = match state_tree.get_actor(CHAINMETADATA_ACTOR_ID)? {
        Some(actor) => actor,
        None => {
            return Err(anyhow!("chainmetadata actor not found"));
        }
    };
    println!(
        "chainmetadata code_cid: {:?}, state_cid: {:?}",
        actor_state.code, actor_state.state
    );

    // retrieve the chainmetadata actor state from the blockstore
    //
    let mut chainmetadata_state: State = match state_tree.store().get_cbor(&actor_state.state)? {
        Some(v) => v,
        None => return Err(anyhow!("chain metadata actor state not found")),
    };
    println!(
        "chainmetadata lookback length: {}",
        chainmetadata_state.lookback_len
    );

    // lets patch the state, here we increase the lookback_len from the default (256) to 512
    //
    chainmetadata_state.lookback_len = 512;

    // store the updated state back to the blockstore and get the new state cid
    //
    let new_state_cid = state_tree
        .store()
        .put_cbor(&chainmetadata_state, Code::Blake2b256)
        .map_err(|e| anyhow!("failed to put chain metadata actor state: {}", e))?;
    println!("new chainmetadata state_cid: {:?}", new_state_cid);

    // next we update the actor state in the state tree
    //
    state_tree.set_actor(
        CHAINMETADATA_ACTOR_ID,
        ActorState {
            code: actor_state.code,
            state: new_state_cid,
            sequence: actor_state.sequence,
            balance: actor_state.balance,
            delegated_address: actor_state.delegated_address,
        },
    );

    Ok(())
}
```

Once we have finished writing our `Upgrade` migration, we can add it to the `UpgradeScheduler`:

```rust
let mut scheduler = UpgradeScheduler::new();
let upgrade = Upgrade::new(chain_name, block_height, app_version, patch_actor_state_func);
scheduler.add(upgrade);

// when initializing the FvmMessageInterpreter, specify the upgrade schedule
let interpreter = FvmMessageInterpreter::<DB, _>::new(
  ...
  scheduler,
);
```


# Example: Upgrading Wasm actor

In this example, we show how you can write an upgrade migration which upgrades the code of an existing WASM actor that has been deployed on chain.

More specifically, in this example we want to replace the code of the `chainmetadata` actor which was deployed at genesis. This actor is used to store blockhashes of the previous blocks on chain. For instance, suppose we intend to enhance this actor to store additional information beyond block hashes. The specifics of the new version's functionality are irrelevant; we focus solely on replacing the actor's code.

Inside the migration function, we first must have access to the WASM binary of the new actor. Here, we simply copied the source code of the `chainmetadata` actor to a new location, made relevant changes to the source code of that new actor and compiled it to the `fendermint_actor_chainmetadata_v2.wasm` target.

To replace the existing `chainmetadata` actor that we deployed at genesis with this new v2 version, we need store the new WASM code in the blockstore, then update `code` of the actor state associated with the `chainmetadata` actor with `code_cid` of the new WASM actor.

Our migration function is defined as follows:

```rust
// The WASM binary of the new version of the chainmetadata actor.
static WASM_BIN: &[u8] = include_bytes!("../output/fendermint_actor_chainmetadata_v2.wasm");

pub fn upgrade_wasm_actor_func(
    state: &mut FvmExecState<NamespaceBlockstore>,
) -> anyhow::Result<()> {
    let state_tree = state.state_tree_mut();

    // get the ActorState from the state tree
    //
    let actor_state = match state_tree.get_actor(CHAINMETADATA_ACTOR_ID)? {
        Some(actor) => actor,
        None => {
            return Err(anyhow!("chainmetadata actor not found"));
        }
    };
    println!(
        "chainmetadata code_cid: {:?}, state_cid: {:?}",
        actor_state.code, actor_state.state
    );

    // store the new wasm code in the blockstore and get the new code cid
    //
    let new_code_cid = state_tree.store().put(
        Code::Blake2b256,
        &Block {
            codec: IPLD_RAW,
            data: WASM_BIN,
        },
    )?;
    println!("new chainmetadata code_cid: {:?}", new_code_cid);

    // next we update the actor state in the state tree
    //
    state_tree.set_actor(
        CHAINMETADATA_ACTOR_ID,
        ActorState {
            code: new_code_cid,
            state: actor_state.state,
            sequence: actor_state.sequence,
            balance: actor_state.balance,
            delegated_address: actor_state.delegated_address,
        },
    );

    Ok(())
}
```

Once we have finished writing our `Upgrade` migration, we can add it to the `UpgradeScheduler`:

```rust
let mut scheduler = UpgradeScheduler::new();
let upgrade = Upgrade::new(chain_name, block_height, app_version, upgrade_wasm_actor_func);
scheduler.add(upgrade);

// when initializing the FvmMessageInterpreter, specify the upgrade schedule
let interpreter = FvmMessageInterpreter::<DB, _>::new(
  ...
  scheduler,
);
```


# Deploying an explorer

Before delving into this tutorial, you should have [deployed a local subnet](https://github.com/consensus-shipyard/ipc/blob/main/docs-gitbook/developer-guides/deploy-a-subnet/README.md). If you're connecting to an existing remote subnet and not following the guide, make sure that you have a local docker installation.

These are instructions for deploying a basic, non-customised, local subnet explorer. The resulting instance will not provide all Blockscout features not be appropriate for production use.

1. Get Blockscout

```
git clone https://github.com/blockscout/blockscout
cd ./blockscout/docker-compose
```

2. Edit `./envs/common-blockscout.env` and set

```
INDEXER_DISABLE_PENDING_TRANSACTIONS_FETCHER=true           
INDEXER_DISABLE_INTERNAL_TRANSACTIONS_FETCHER=true
```

The default setup assumes a local subnet with the Ethereum RPC on `localhost:8545`. If you're connecting to a remote RPC, remember to also set `ETHEREUM_JSONRPC_HTTP_URL` accordingly.

Some frontend calls use hardcoded absolute URLs. Unless you're only accessing the Blockscout interface on `localhost`, make sure to review the environmental variables in the different files under `./envs/` and adjust addresses (e.g. `BLOCKSCOUT_HOST` in `envs/common-blockscout.env` and `NEXT_PUBLIC_STATS_API_HOST` in `envs/common-frontend.env`. You'll also need to make sure the required ports are accessible (at least 80, 8080, and 8081).

3. Start Blockscout

```
docker compose -f docker-compose-no-build-geth.yml up -d
```

The web interface will be available at `http://localhost/`

If you need to take down the setup, run `docker compose -f docker-compose-no-build-geth.yml down`. Note that the data store is mounted externally and will be reused on redeployment. If you want to clear the database, run `rm -rf ./services/blockscout-db-data`.


# Addressing

## IPC Actors addressing

### Filecoin adressing schemes

When the contracts described below (Gateway, Registry, Subnet) are deployed to a Filecoin network (e.g. Mainnet, Calibration), `f0`, `f2` and `f410` addresses will be assigned to them. The latter is equivalent to an Ethereum hex address. Detailed explanation of Filecoin addressing scheme is available in [the documentation](https://docs.filecoin.io/smart-contracts/filecoin-evm-runtime/address-types).

This also applies to IPC L2+ networks since IPC inherits the addressing model, the Init actor mechanics, and the EVM runtime and the Ethereum Address Manager from Filecoin.

### Gateway Actor and Registry Actor addressing

Gateway and Registry actors are EVM smart contracts, so they inherit the addresses specified above.

### Subnet Actor addressing

Subnet Actor is an EVM smart contract deployed separately for each child Subnet and registered in the parent's Gateway. It gets assigned the address types stated above. The creation of Subnet Actor is performed via the Registry actor (which acts like a factory). The Ethereum address acquired by the Subnet Actor is determined by the semantics of the EVM `CREATE` opcode. It's worth noting that the caller address seen by `CREATE` will be that of the Registry diamond contract. One can use this fact to counterfactually predict subnet actor deployment addresses ahead of time, although being aware of the non-determinism present in us using `CREATE` instead of `CREATE2` for the time being.

## Subnet address

IPC subnets are uniquely identified by their [SubnetId](https://github.com/consensus-shipyard/ipc/blob/main/contracts/src/structs/Subnet.sol#L9) which consists of

* `uint64 root` - a [Chain ID](https://chainlist.org/?search=filecoin\&testnets=true) of the root subnet. Eg. all subnets anchored to Filecoin Mainnet have `root` equal to `314`
* `address[] route`- the array of addresses down the IPC hierarchy.

[`SubnetIDHelper`](https://github.com/consensus-shipyard/ipc/blob/main/contracts/src/lib/SubnetIDHelper.sol) contains utility functions to create and operate on `SubnetID`s

### String representation

The string representation of subnet address equals:

* prefix `/r` indicating root chainID directly followed by the value of `root` Chain ID (eg. `/r314` for Filecoin Mainnet).
* concatenated with the f410 addresses of the subnet actors top-to-bottom in the hierarchy, separated by `/` as a divider.

Example 1: the string representation of the Filecoin Mainnet itself is `/r314` as this is a root, not anchored to any parent.

Example 2: the string representation of an L3 subnet anchored to Filecoin Calibration (`chainID` equal `314159`) could be `/r314159/t410fgalav7yo342zbem3kkqhx4l5d43d3iyswlpwkby/t410fixm5mqenkfm2g6msjt2chs36cxaa7ka745xo2jq`

where `t410fgalav7yo342zbem3kkqhx4l5d43d3iyswlpwkby` and `t410fixm5mqenkfm2g6msjt2chs36cxaa7ka745xo2jq` are addresses of L2 and L3 subnet respectively.

> Note that the `t` prefix denotes a test network. `f` denotes mainnet.

### Binary representation

`SubnetID` is serialised using `keccak256(abi.encode(SubnetID))` ([utility function](https://github.com/consensus-shipyard/ipc/blob/main/contracts/src/lib/SubnetIDHelper.sol#L58)).

This hash is used as the way to verify equality of 2 `SubnetID` s ([utility function](https://github.com/consensus-shipyard/ipc/blob/main/contracts/src/lib/SubnetIDHelper.sol#L89)), store mapping of hash to the Subnet object etc.

## IPC Address

[IPCAddress](https://github.com/consensus-shipyard/ipc/blob/main/contracts/src/structs/Subnet.sol#L149) contains both `SubnetID` and `FvmAddress` to uniquely identify an actor (EOA or smart contract) existing within the IPC hierarchy.


# CometBFT

## IPC Spec - CometBFT

This document explains the relationship between Fendermint and CometBFT.

## What is CometBFT?

[CometBFT](https://cometbft.com/) is a *generic* blockchain engine using the Tendermint consensus. What this means for us has been argued in the original [solution concept](https://docs.google.com/document/d/1cFoTdoRuYgxmWJia6K-b5vmEj-4MvyHCNvShZpyconU) that introduced it (still as Tendermint Core at the time) for use for IPC.

By *generic* we mean that CometBFT doesn’t say anything about the format and content of the *transactions* and the *ledger* of the replicated state machine; that is completely within the jurisdiction of the *application*, which is Fendermint.

The two communicate over the Application BlockChain Interface (ABCI). We are currently using [version 0.37](https://docs.cometbft.com/v0.37/) with [ABCI v1](https://docs.cometbft.com/v0.37/spec/abci/) ; version 0.38 moves to [ABCI v2](https://docs.cometbft.com/v0.38/spec/abci/abci++_methods), which will unlock new capabilities.

To get an idea of how little CometBFT says about what the transactions and the ledger should look like, have a look at the [kvstore example](https://docs.cometbft.com/v0.37/guides/go-built-in) in the guide; there are no signatures, the transactions are literally just strings formatted as `"key=value"`. Check out [these steps](https://github.com/consensus-shipyard/ipc/blob/main/docs/fendermint/tendermint.md) to see how to get started with CometBFT by running the built-in `kvstore` application.

A really good overview of the Tendermint consensus lifecycle in CometBFT is provided by the following diagram:

![https://docs.tendermint.com/v0.34/assets/img/tm-transaction-flow.258ca020.png](https://docs.tendermint.com/v0.34/assets/img/tm-transaction-flow.258ca020.png)

## What is Fendermint?

Fendermint implements a ledger using the ABCI interface, driven by CometBFT. It contains the definition of transactions and queries, and uses the FVM to execute messages and store the data as IPLD.

See the [IPS Spec - Executions](https://www.notion.so/IPS-Spec-Executions-ebf13d833d6845ec9c11b59bd514fcda?pvs=21) and [IPC Spec - Snapshots](https://www.notion.so/IPC-Spec-Snapshots-edcbd2716d4e48e3bd3fe8d39a22fdb2?pvs=21) to get a detailed overview of how the ABCI methods are implemented, and the [Architecture diagram](https://github.com/consensus-shipyard/ipc/blob/main/docs/fendermint/architecture.md) for the component layout.

### Common Misconceptions

There were some questions we should clarify, as they are not immediately clear for someone not familiar with how CometBFT works.

#### How many CometBFT instances per Fendermint?

There is a 1-to-1 relationship between CometBFT and Fendermint. CometBFT feeds the blocks to Fendermint for execution. Fendermint maintains the application state with deterministic execution of transactions. Fendermint is not a standalone application, with CometBFT as some kind of a front-end, like the Ethereum API facade. It is just an execution engine.

#### Can Fendermint instances collude and lie to CometBFT?

This assumes that Fendermint instances communicate between each other in a Byzantine way, present their version of truth to CometBFT, and then the question is what happens if Fenderrmint isn’t majority honest. No, it doesn’t work that way.

Fendermint instances do not run consensus between themselves, that is precisely the job of CometBFT. Each CometBFT is associated with a single dedicated and trusted Fendermint instance that it owns, and communicates with other CometBFT instances about the results of their respective transaction executions.

#### How many subnets does Fendermint run?

Each CometBFT+Fendermint chain is a specific subnet. To run a subnet, the operators must launch a separate blockchain.

#### Does Fendermint call CometBFT through ABCI?

No, the ABCI application does not normally make RPC calls to CometBFT, the communication is strictly initiated by CometBFT, the application acts as the server reacting to events happening in consensus.

Fendermint itself calls CometBFT via JSON-RPC to enquire about the history of the power table, and the status of the chain syncing, but that has to happen outside the ABCI cycle.

## Process Model

CometBFT is written in Go, while Fendermint is written in Rust. Had Fendermint been written in Go, we would have the opportunity to compile a `fendermint` binary that runs CometBFT with an in-process implementation of the Fendermint ABCI ledger. This is how [Cosmos SDK](https://docs.cosmos.network/) applications work. In our case, however, we have to run two distinct processes: `cometbft` and `fendermint`, with the communication happening over TCP. They can be deployed on the same machine, or separately.

In Docker they are two distinct containers, with `cometbft` being one of the official images. This has the advantage that there is no need to audit it, because we haven’t changed anything.

💡 Sometimes the question comes up whether the `fendermint` process should somehow own `cometbft` and try to hide the fact that it even exists to simplify its operation and deployment - in this regard it would make it more similar to the experience of native Cosmos SDK apps written in Go, which are executed in the CometBFT process itself.

It is worth keeping in mind in this context that [securely deploying](https://docs.cometbft.com/v0.37/core/validators#setting-up-a-validator) CometBFT in production is a task left for engineering. Fendermint doesn’t contain any tools or documentation to do this for the user.

For this reason it is worth highlighting the need for operators to familiarise themselves with CometBFT. Trying to hide CometBFT as an implementation detail would risk obscuring it, and making it more difficult for them to find expert advice on operating their blockchain.

When in need for inspiration, one can look at the [Penumbra deployments](https://github.com/penumbra-zone/penumbra/tree/v0.74.0/deployments), which have a similar split between a Rust application and the CometBFT engine.


# IPLD Resolver

## IPC Spec - IPLD Resolver

The IPLD Resolver can facilitate IPC in multiple ways:

* connecting all participants of an IPC hierarchy into a P2P network
* advertising via GossipSub which subnets a particular node can serve data for (the primary motivation being sending bottom-up checkpoints containing CIDs, which the nodes on the parent subnet procure with the resolver)
* gossiping votes about application specific things in subnet-specific topics, e.g. evidence, observations, attestations
* resolving CIDs into content via BitSwap
* pre-emptively push data to the parent subnet to circulate it via GossipSub, instead of waiting until the request arrives via BitSwap (e.g. the contents of a bottom-up checkpoint)

The resolver used to be a [standalone library](https://github.com/consensus-shipyard/ipc-ipld-resolver) before it was [migrated](https://github.com/consensus-shipyard/ipc/tree/main/ipld/resolver) to the IPC monorepo and upgraded to use a newer version of [`libp2p`](https://github.com/libp2p/rust-libp2p) . Since then the BitSwap unit tests show that there is a [bug](https://github.com/consensus-shipyard/ipc/issues/537) with larger data structures, which we haven’t had time to investigate. This function isn’t used at the moment, but if it were, the problem had to be fixed first.

## Use Cases

The [docs](https://github.com/consensus-shipyard/ipc/tree/main/ipld/resolver/docs) have a fairly good overview of what this component does, so here we’ll just concentrate on how it is used in the context of Fendermint:

* gossiping votes about which blocks are final on the parent subnet
* resolving bottom-up checkpoints from the child subnet (not used at the moment)

The resolver is instantiated in the [`run`](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/app/src/cmd/run.rs#L165-L233) CLI command if the node is configured with both:

* an IPC subnet (can be root), and
* a multiaddress where it will listen to incoming requests

If enabled, the application will be started with:

* checkpoint resolver pool
* a finality vote publisher
* a finality vote subscriber
* the IPLD resolver service itself, which discovers peers, manages subscriptions, publishes memberships, etc.

### Parent Finality Vote Gossip

The [`voting`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/vm/topdown/src/voting.rs) module in the `topdown` crate has a generic [STM](https://crates.io/crates/async-stm) enabled `VoteTally` component which has the following components:

* `chain` contains contains block hashes that our node sees as final on the parent subnet at each block height
* `votes` contains votes that any particular block hash received at any height from validators
* `power_table` contains the public keys of the validators who are currently eligible to vote

With these the `VoteTally` can be used to register votes coming in over a gossip channel, and to look for a finalized block height that our node knows of where there is also a quorum, treating a vote on a block as an implicit vote on all its known ancestors as well.

The `VoteTally` is part of the `ChainEnv` and consulted by the `ChainMessageInterpreter` during block proposals. The goal is that we only make proposals on parent subnet finalities when the tally indicates that there is already a quorum. Since the voters are the same validators who will vote about the proposal, the presence of the quorum should be enough for the proposal to pass as well, preventing any liveness issues with the consensus.

The votes are being fed to the tally by the [`dispatch_resolver_events`](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/app/src/cmd/run.rs#L501) function.

### BottomUp Checkpoint Resolution

The [`resolver`](https://github.com/consensus-shipyard/ipc/tree/specs/fendermint/vm/resolver) crate under `vm` is a generic component which consists of two parts:

* The [`pool`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/vm/resolver/src/pool.rs) module contains the `ResolvePool` which is an [STM](https://crates.io/crates/async-stm) enabled component where we can submit items to be resolved, and monitor their status, collecting. The pool is generic in the items it can resolve, as long as they can be mapped to a `Cid` and a `SubnetId`. The pool is the shared memory which is used by the interpreters to add items and inquire about their status during the block execution.
* The [`ipld`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/vm/resolver/src/ipld.rs) module contains the `IpldResolver` which is runs in the background to execute tasks sent to the `ResolvePool` by sending them to actual IPLD `Service`.

Currently the `ChainEnv` requires a pool working with [`CheckpointPoolItem`](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/interpreter/src/chain.rs#L51).

💡 Ultimately this is not currently in use because checkpoint submissions ended up containing all the bottom-up messages.


# Materializer

## IPC Spec - Materializer

This document gives an introduction and the rationale for the Materializer, which is a tool that helps provision entire IPC subnet hierarchies, mainly for testing purposes.

## Hierarchical Deployment

Deploying IPC subnets has traditionally been a laborious process which involved running scripts manually and copying values from the output of one and passing it as input to the next. The steps involved are described in the [usage](https://github.com/consensus-shipyard/ipc/blob/specs/docs/ipc/usage.md) document.

What complicates matters is that variables aren’t known up front, for example we have to create a subnet to learn its ID which we have to use in all subsequent commands, and continue recursively until the entire hierarchy is established. We have to fund accounts on subnets, join with validators, wait for the subnet to activate, then start subnet nodes.

An early attempt to help with the creation of [test environments](https://github.com/consensus-shipyard/ipc-libs/blob/main/ipc/testing/e2e/README.md#test-environment) was the creation of topologies such as this [example](https://github.com/consensus-shipyard/ipc-libs/blob/main/ipc/testing/e2e/topologies/example.yaml) which were compiled to a [series of steps](https://github.com/consensus-shipyard/ipc-libs/blob/main/ipc/testing/e2e/topologies/example.sh) in a script to provision a hierarchy. The steps were executed with `make` and left behind enough artifacts to ensure that repeated execution was idempotent.

An evolution of this idea is the Fendermint [`materializer`](https://github.com/consensus-shipyard/ipc/tree/specs/fendermint/testing/materializer), which is available as a [CLI command](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/app/options/src/materializer.rs) under the main `fendermint` entrypoint.

## Manifests

The materializer uses the [`manifest`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/testing/materializer/src/manifest.rs) to describe entire hierarchies. This is typically a nested, recursive YAML or JSON document with some examples used in tests available [here](https://github.com/consensus-shipyard/ipc/tree/specs/fendermint/testing/materializer/tests/manifests).

The entities in the manifest are generally organised into maps, so that they have a unique logical ID within their parent context. By logical ID we mean that it’s just a static string label, e.g. `alice`, rather than something that we’ll only know once we deployed something.

At the root of the manifest we have the following properties:

* `accounts` is technically a map, but really is just a set of logical account identifiers such as `alice` and `bob`. These IDs can be used throughout the manifest whenever we refer to an account, instead of public keys or FVM addresses. That is so we can provide a static description of the subnet, and create cryptographic keys later, which can get unpredictable FVM actor IDs when we use them on a chain.
* `rootnet` describes the L1, which can have one of the following types:
  * `New` means we will provision the L1 chain ourselves, in which case we have to define:
    * `validators` containing a mapping from account ID to collateral (voting power)
    * `balances` listing the initial token balance for every account on the L1 (assumed to be Ethereum accounts because that’s how we interact with IPC).
    * `nodes` is the list of physical nodes to create to run the L1
    * `env` contains custom environment variables passed to all nodes
  * `External` means we will use an existing L1 such as Calibration net:
    * `chain_id` is the numerical ID of the L1 chain
    * `deployment` describes how we’ll get the IPC stack:
      * `New` means we have to deploy it from scratch using the given `deployer` account, which must have balance on the L1
      * `Existing` means it’s already deployed, and we just have to give the Ethereum addresses of the `gateway` and `registry` actors
    * `urls` contains a list of RPC addresses of nodes on the L1 where we can send queries and transactions
* `subnets` is a map which contains the IPC subnets to be created on the L1 (we currently do not support connecting to an existing subnet):
  * `creator` is the logical ID of the account to be used to create the subnet, ie. to send the transaction invoking the `registry` contract on the L1
  * `validators` is a map with the collateral each validator in this subnet will `join` with (for this they must have the balance on the L1; currently the materializer only supports PoS, not PoA)
  * `balances` is a map with the amount of tokens that accounts will `fund` the subnet with
  * `nodes` lists the physical machines that will run this subnet
  * `relayers` defines the how information flows bottom-up:
    * `submitter` is the account that sends the checkpoint as a transaction to the parent network
    * `follow_node` is the ID of the node which the relayer will query for changes
    * `submit_node` is either a URL or the ID of the parent node to submit transactions to; URLs are used when we have an external rootnet (in which case there are no nodes in the manifest), while IDs work with new rootnets that are run by nodes we defined
  * `bottom_up_checkpoint.period` defines the frequency of checkpoints on the subnet
  * `env` is a list of custom environment variables all nodes get
  * `subnets` recursively defines nested subnets

The `nodes` in the manifest have the following properties:

* `mode` defines whether we have a `Validator` or a `Full` node. The former has a `validator` which is must be of the accounts, the latter just follows others without producing blocks.
* `ethapi` indicates whether the Ethereum API facade should run on this node
* `seed_nodes` is a list of node IDs *in this subnet* to bootstrap from - nodes can mutually seed from each other
* `parent_node` is only used on subnets, not on the rootnet; it can be a URL or a node ID depending on whether the parent subnet is an external rootnet, or consists of nodes defined in the manifest. It defines which node the top-down sync follows.

### Transitive Balances

The balances and collaterals in the manifest are given in *atto*, which is is 1/10\*\*18 of a FIL token.

The subnet balances describe the desired state after the whole hierarchy has been provisioned, not when a particular subnet is created. That means that the balances of nested subnets are not subtracted from the balances of ancestor subnets, but rather brought in recursively from the rootnet balance. For example if we say we want `alice` to be a validator in subnet `/root/foo/bar` with 100 collateral, then we don’t have to list `alice` with a balance of 100 in `/root/foo` ; we just have to make sure `alice` has the necessary starting balance in `/root` and the necessary `fund` transactions will be issued to move the funds from `/root` to `/root/foo` and then `join` the `/root/foo/bar` subnet there.

## Materializers

The materializer figures out the steps necessary to provision the subnet hierarchy and execute them while leaving behind a trail of artifacts on the file system that allows it to be idempotent. For example:

* every account has their own directory where the generated secret/public key, FVM and EVM addresses are stored in various formats; if the directory exists creating an account is skipped
* every subnet has their own directory where the subnet ID is written to a file; if it exists, we don’t have to create the subnet again
* every node has their own directory where the database files and logs are mounted

The logic of which steps to execute is contained in the [`testnet`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/testing/materializer/src/testnet.rs) module, and depends purely on the contents of the manifest. Where we *can* have different strategies is how to physically materialize the nodes and subnets:

* We can provision everything as local docker containers
* We could remotely provision instances on cloud infrastructure
* We could run multiple instances in-memory on different ports and override their behaviour in tests

The only currently available implementation is [using docker](https://github.com/consensus-shipyard/ipc/tree/specs/fendermint/testing/materializer/src/docker) to provision local containers, potentially connecting to an external rootnet. This is used by the `setup` command.

We can also implement materializers that do not actually provision resources:

* Visit the manifest and validate that accounts exist, that balances are feasible
* Log actions before forwarding them to another materializer
* Print CLI commands that the operator could execute in the terminal

The combination of the first two is how the `validate` command works.

## Use Cases

### Integration testing

The integration tests [use a docker materializer](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/testing/materializer/tests/docker.rs) to instantiate testnets on the fly and run assertions against specific nodes. The tests are organised into [modules](https://github.com/consensus-shipyard/ipc/tree/specs/fendermint/testing/materializer/tests/docker_tests), according to which [test manifest](https://github.com/consensus-shipyard/ipc/tree/specs/fendermint/testing/materializer/tests/manifests) they use.

The machinery available in the tests try to make it easy to connect to specific nodes, so a test can look for example like this:

```rust
let node_id = testnet.root().node("node-2");
let node = testnet.node(&node_id)?;

let provider = node
    .ethapi_http_provider()?
    .ok_or_else(|| anyhow!("node-2 has ethapi enabled"))?;

let bn = provider.get_block_number().await?;

if bn <= U64::one() {
    bail!("expected node-2 to sync with node-1");
}
```

### Connecting to Calibration

For debugging purposes it is possible to use the materializer to provision a subnet on Calibration net and then stop the local containers, and use the artifacts left behind to run a node through the IDE with a debugger.

The materializer CLI has a `fendermint materializer import-key` command to import some secret key that already exists on an external L1 (funded through the faucet) to be associated with a logical account ID in the manifest.

For example the following manifest was used to create a local stand-alone node to run a subnet we create on Calibration net:

* External Standalone Manifest

  ```jsx
  accounts:
    fridrik: {}

  rootnet:
    type: External
    chain_id: 314159
    deployment:
      type: Existing
      gateway: 0x6d25fbFac9e6215E03C687E54F7c74f489949EaF
      registry: 0xc938B2B862d4Ef9896E641b3f1269DabFB2D2103
    urls:
      - https://api.calibration.node.glif.io/rpc/v1

  subnets:
    iceland:
      creator: fridrik
      validators:
        fridrik: '1000000000'
      balances:
        fridrik: '20000000000000000000'
      bottom_up_checkpoint:
        period: 100
      relayers:
      nodes:
        moso:
          mode:
            type: Validator
            validator: fridrik
          parent_node: "https://api.calibration.node.glif.io/rpc/v1"
          seed_nodes: []
          ethapi: true
  ```

The following commands provision the subnet locally:

```jsx
cargo run -q -p fendermint_app --release -- \
  materializer --data-dir $PWD/testing/materializer/tests/docker-materializer-data \
  import-key --manifest-file ./testing/materializer/tests/manifests/external.yaml \
  --account-id fridrik --secret-key ~/.ipc/validator_0.sk

cargo run -q -p fendermint_app --release -- \
  materializer --data-dir $PWD/testing/materializer/tests/docker-materializer-data \
  setup --manifest-file ./testing/materializer/tests/manifests/external.yaml
```


# Top-down Finality

## Topdown Finality

The topdown finality helps propagating the important states in the parent subnet to the child blockchain such as fund transfer and validator updates.

The following data is passed:

* Parent block height + block hash
* Transactions from the parent to the child subnet
* Validator changes from parent to the child subnet

The finality is represented in contract in the form of:

```solidity
// Link: https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/contracts/src/structs/CrossNet.sol#L11
struct ParentFinality {
    uint256 height;
    bytes32 blockHash;
}
```

The validator changes are represented as an array of Changes while topdown messages are an array of [IPCEnvelope](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/contracts/src/structs/CrossNet.sol#L63).

Once enabled, the topdown finality runs in the background. From a high level point of view, the topdown finality works as follows:

* Topdown finality will first fetch the last committed `ParentFinality`. If there is no previous committed parent finality, the genesis block is [used](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/sync/mod.rs#L36).
* There is a [ParentSyncer](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/sync/syncer.rs#L24C19-L24C36) that constantly polls the parent states through RPC calls, either fetching events emitted or through getters. The `ParentSyncer` stores the pulled data in cache and publishes a vote on the latest block seen.
* Once a quorum is formed on the blocks seen, a topdown finality proposal will be [added](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/interpreter/src/chain.rs#L132) to the cometbft proposal. The proposal is just the `ParentFinality` struct shown above, i.e. a block height and the corresponding block hash.
* Once a topdown proposal is received by each node, it will be validated against the topdown syncer cache and the RPC node if there is a cache miss. If the checks do not pass, the proposal will be rejected. The checks include:
  * The height proposed is present in the cache or exists in the RPC node
  * The hash matches the corresponding height’s block hash
* Once the proposal is accepted, it will be executed. This means the topdown messages will be executed and validator changes will be stored.

All nodes must agree on the parent state, even though they may be using different RPC endpoints. Those RPC endpoints may have different views of the parent state. At the moment the nodes don't have any way to verify the correctness of `ParentFinality` received from the RPC endpoints. That's why the nodes have to reach consensus on the parent finality through a voting mechanism.

## Parent Syncer

The parent syncer polls the parent to obtain the “finalized” state in the parent. The “finalized” view in the parent is by delaying a fixed number of blocks from the chain head. This ensures the probability of a parent reorg is small.

There are currently two implementations of parent syncer, [LotusParentSyncer](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/sync/syncer.rs#L24) and [TendermintAwareSyncer](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/sync/tendermint.rs#L16C15-L16C36). The `TendermintAwareSyncer` is a wrapper of `LotusParentSyncer` , which stops the polling if the node is currently syncing with its peers, because if it is, then it won’t be proposing anything until it’s finished, and it will receive the parent finalities from the CometBFT blocks instead.

The `LotusParentSyncer` will actually constantly poll the next height. The next height is either the last committed parent finality height plus 1 or the latest height seen in cache plus 1. If the next height is not more than the delayed chain head, parent syncer will skip this round and sleep for some fixed time.

Once the data is pulled, it will be committed to a in-memory cache. This cache is indexed by block height. It forces sequential insertion the blocks inserted are [sequential](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/cache.rs#L27) in block height.

At the same time, the observed parent block will be added to the `[VoteTally](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/vm/topdown/src/voting.rs)`.

In Lotus, there is a concept of null block. When this happens, there is no data in the block. In the parent syncer, this block will be skipped, i.e. a None is inserted.

To ensure the parent blocks are chained, parent syncer also makes sure the block hash of the parent block [matches](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/sync/syncer.rs#L227) the latest non-null parent hash. Otherwise a warning will be logged.

Do note that the current [implementation](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/proxy.rs#L65) of parent syncer relies heavily on solidity events. Make sure the RPC node used gives full access to historical events, otherwise not all events will be correctly returned and result in data loss.

## Vote Tally

The [`VoteTally`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/vm/topdown/src/voting.rs) is a component used to keep track of votes cast on parent finalities by the current child validators, and supports looking for the highest parent block height and hash which received a quorum. The votes are published over a GossipSub channel, once a node observes a finalized block through the syncer described above.

A quorum detected by the `VoteTally` is used a pre-condition for finality proposals being added to the CometBFT block proposals, to avoid any liveness issues which could arise if the other validators were to reject the proposal. If a premature finality causes the block propsal to fail, it means in that round CometBFT cannot make progress, it cannot finalize a block, potentially causing the subnet blockchain to stall. By requiring a quorum, we avoid this issue by only proposing when we have high confidence that the proposal will be accepted.

See `IPC Spec - IPLD Resolver` for a more detailed discussion of the `VoteTally`.

## Topdown Finality Proposal

Once the `VoteTally` has detected a quorum, a parent finality proposal will be made. The entrypoint is this [link](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/finality/null.rs#L84). The high level idea for proposal creation is that it will first get the latest height in cache. However, do note that some fendermint nodes might be syncing faster than other fendermint nodes, if the proposed height is too large, other nodes will reject the proposal because they have yet to “see” it. As such, there is a `MaxProposalRange` parameter that controls how far the proposed height can be greater than the last committed block height. Once the height is set, the corresponding block hash is added to the proposal. The proposed topdown finality is added to the list of transactions sending to the cometbft.

## Topdown Finality Proposal Validation

Once a topdown finality proposal is received by a node, it’s [checked](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/interpreter/src/chain.rs#L209) to ensure its validity. The checking performs:

* The proposed height either exists in topdown syncer
* The block hash of the proposed height is correct.

If the above two checks are valid, the proposal is accepted.

## Topdown Finality Proposal Execution

Once a proposal is accepted, it will be [executed](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/interpreter/src/chain.rs#L290) together with the batch of transactions. The execution of the proposal happens in three stages:

* Commit new topdown finality
* Fetch validator changes and store
* Fetch topdown messages and execute

The validator changes and topdown messages are first fetched from the parent syncer cache. If they are not present, then it will fallback to the RPC node. There is a simple exponential backoff [retry](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/topdown/src/finality/fetch.rs#L174). If the retry fails, which means the node cannot be reached to obtain the validator changes or topdown messages, an error is thrown. The reason for this is that the cluster has decided on the topdown finality, if one cannot obtain the corresponding changes and message, then it should stop.

As topdown finality relies heavily on RPC node for querying, make sure the RPC node is reliable and returns enough historical data.

See also `IPS Spec - Executions` for a description of how proposals and executions are implemented.


# Bottom Up Checkpoint

## Bottom Up Interactions

This document takes a closer look in the IPC mechanics involved in information flowing from the child to the parent subnet, a.k.a. bottom-up.

## Interactions

There are two user initiated interactions in IPC that result in bottom-up messages being sent:

* `release` sends tokens from a user account on the child subnet to another on the parent subnet
* general cross-net messages sent to, or through, the parent

The mechanism for propagating information from the child to the parent is through *checkpoints*.

## Checkpoints

The epic for implementing checkpointing is [here](https://github.com/consensus-shipyard/ipc/issues/211). The end-to-end workflow can be followed on this [diagram](https://github.com/consensus-shipyard/ipc/blob/main/docs/fendermint/checkpointing.md). The IPLD Resolver docs also present a [use case](https://github.com/consensus-shipyard/ipc/blob/specs/ipld/resolver/docs/README.md#checkpointing) for checkpointing.

### Contents

The original idea for a checkpoint was to contain the following information:

* `subnet_id`: to identify to the parent subnet which child the checkpoint is for, and to prevent any replay attacks across subnets run by the same validators
* `block_height` : the height of the child subnet blockchain at the time of checkpoint creation
* `block_hash`: the hash of the block whose execution results in a checkpoint being added to the ledger, to prevent long range attacks on the subnet by anchoring it to the parent chain
* `next_configuration_number`: this is the identifier of the validator set which is going to sign the *next* checkpoint; the current checkpoint is always going to be signed by the *current* validator set

To these would be added fields to carry the contents of the checkpoint, which would be either:

* `messages`, which would be a list of bottom-up cross-net messages, or
* `messages_cid`, which would be the CID of the messages, but not the payload itself, which would be procured by the [IPC Spec - IPLD Resolver](https://www.notion.so/IPC-Spec-IPLD-Resolver-7b4290a0d60c40cdba98cd6d3e66648b?pvs=21)

The CID based approach would only work with Fendermint, not Lotus running on rootnet, but in general a commitment based approach can work with Lotus too.

### Triggers

Originally checkpoints were supposed to be submitted at regular intervals, which was governed by the parameters with which the child subnet contract was created on the parent subnet, and made part of the `IpcParams` in [`genesis`](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/fendermint/vm/genesis/src/lib.rs#L227). However, this presented a problem with the number of messages that could be included in a checkpoint, which is why later the triggers for checkpoint creation were amended to be any of the following conditions:

* a fixed period in terms of block height
* the number of enqueued bottom-up messages being over a [limit](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/contracts/src/GatewayDiamond.sol#L68C37-L68C55)
* the number of enqueued bottom-up messages being under the immediate checkpoint limit, but having waited some maximum amount of time already

If any of these conditions are met, a checkpoint is added to the ledger.

One side effect of adding extra conditions is that it makes transactions irregular in their appearance, and so the parent cannot verify which is the next checkpoint to admit. For this reason, it is planned to add a new `prev_checkpoint_height`, so that checkpoints can form a chain, and then the parent subnet will only accept the next checkpoint if it points at the last submitted one as its predecessor.

When the number of enqueued bottom-up messages exceeds the limit, a new message batch is created and committed at the current epoch. This will trigger a new checkpoint to be [created](https://github.com/consensus-shipyard/ipc/blob/7af25c4c860f5ab828e8177927a0f8b6b7a7cc74/contracts/src/lib/LibGateway.sol#L272).

### Creation

Where checkpoint creation fits into the process is explained in [IPS Spec - Executions](https://www.notion.so/IPS-Spec-Executions-ebf13d833d6845ec9c11b59bd514fcda?pvs=21).

Creating a checkpoint in the ledger is performed deterministically by every full node; they simply call the [`gateway`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/vm/interpreter/src/fvm/state/ipc.rs) contract with the following inputs:

* an unsigned `BottomUpCheckpoint`
* the root hash of a Merkle tree which consists of the current power table, ie. the public keys and powers of the current validator set
* the total power of the validators

### Signatures

After the checkpoint has been added to the ledger *and committed in a block*, those nodes which are currently validators broadcast transactions which add their signatures using the `broadcast_signature` function of the [`checkpoint`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/vm/interpreter/src/fvm/checkpoint.rs) module.

The reason we wait for the the change to be committed is so that the transactions that add the signatures don’t get rejected by `check_tx` because they are referring to a non-existing checkpoint.

The signing and sending of transactions happens in the [`broadcast`](https://github.com/consensus-shipyard/ipc/blob/specs/fendermint/vm/interpreter/src/fvm/broadcast.rs) module which fetches the current nonce of the validator, estimates the gas, performs retries, etc. Because it fetches the nonce for each submission, it cannot be used in parallel.

💡 To submit transactions the validators of the subnet need to have an Ethereum account with sufficient tokens to cover the gas cost. They can use \`fund\` in order bring in tokens from the parent subnet.

The fact that validators have to pay to submit transactions to the subnet they validate may seem harsh; intuitively these should be free! But remember that a subnet can contain Byzantine validators who might abuse their privileges of free transaction submission. However, there is no reason why Fendermint couldn’t contain extra logic to compensate successful submissions for their costs.

The signature transactions are sent to the child ledger, where they accumulate until a quorum is reached, that is, more that 2/3 of the total power of validators have signed the checkpoint.

Note that by this time the CometBFT validators could be different, which is why signature submissions contain a Merkle proof that shows that the submitter was indeed part of the committee responsible for signing a past checkpoint.

### Relayers

Once a checkpoint has gathered a quorum of signatures in the child ledger, it can be picked up by a *relayer* and submitted in the form of a transaction to the parent subnet.

Ideally we would like some redundancy in the number of validators, so there is no single point of failure.

Relayers generally should be rewarded for their service. There are numerous opinions on how to implement rewards, each with their drawback:

* *Only the first submitter gets a reward.* To achieve redundancy the reward would have to be multiple of the cost. Depending on how likely it is to beat the fastest relayer (a parent validator might insert their own transaction to steal the rewards), it might make it unprofitable for multiple validators to operate.
* *The first N submitters get rewards.* It is easy for any relayer to submit N transactions in a Sybil attack to reap all rewards, hampering redundancy.
* *All submitters in a fixed time period get equal share of a fixed reward.* This takes out the competition aspect, and discourages Sybil attacks because the reward doesn’t grow. It should lead to a dynamic equilibrium of the number of relayers. However if the fixed time window is too wide, it encourages freeloaders who just repeat the first submission, which would make it look like there is redundancy where there isn’t.

💡 Fendermint has a naive implementation of the fixed reward scheme divided between all relayers, however due to a vulnerability this was removed for now and relayers get no rewards.

### Validation

The parent subnet contains a smart contract specific to the child subnet which can validate the contents of the checkpoint. Currently the checkpoint submission will contain the multisig of the validators as proof of quorum.

The parent subnet is the source of the validator power distribution, but it’s the child subnet that communicates through checkpointing how far ahead it has synchronised the changes in validator powers; this is signalled by the *configuration ID*, with different power tables being different configurations.

The parent knows what the last committed configuration ID of the subnet is, and it expects these validators to be the ones who sign the checkpoint, with sufficient weight to form a quorum.

Once the quorum has been verified, the `next_configuration_id` informs the parent how far ahead it can apply the pending validator updates on the active validator set, and thus know who to expect the *next* checkpoint to be signed by.

### Execution

When the checkpoint is submitted to the Lotus rootnet, it is currently expected to either contain all bottom-up messages or that they accompany the checkpoint in a different way, and only a commitment is in the checkpoint. However in both cases the bottom-up messages would be executed and their gas cost paid for by the relayer.

With a Fendermint parent network, the same thing works if that’s how the smart contracts are implemented. There was another way laid out it in the epic above, which involved the IPLD Resolver procuring the checkpoint payload from the subnet based on a CID, and executing the messages implicitly when the validators decide that they all have the data available. In this case the relayer would have only paid for the *validation* of the checkpoint, not its *execution*.

Both of these schemes suffer from the fact that the gas limit of the messages included in the checkpoint is unknown when the checkpoint is made (and it *cannot* be known, as by definition the gas cost depends on where the message will be executed):

* In the case where a relayer executes the messages, they first have to estimate the gas cost, so at least it is known before the checkpoint-bearing transaction is included in a rootnet block, and thus the block gas limit can be observed. However, if the gas spent by the checkpoint would exceed the block gas limit, the checkpoint will never be included, but at the same time no other checkpoint can be produced by the subnet, and thus checkpointing stalls.
* In the case where the validators execute messages implicitly, they can choose whether to include it in a block or not, but to do so they would need to estimate the cost at some point, and again it might exceed the limits. Implicit execution also makes it difficult to deal with errors, in particular there is no room for retries.

To overcome this issue, ideally cross messages would *not* be executed in the block where the checkpoint is included. Instead either just a commitment would be stored, or messages would be parked in inboxes (e.g. organised by sender account). The senders could come later and initiate their own transactions to kick off the execution of the messages delivered as part of the IPC consensus mechanism, at which point they can pay for the gas and retry as many times as they see fit if they run out of gas.

💡 Because of the untractable nature or cross-net message gas limits, currently only \`fund\` and \`release\` messages are allowed.


# Networks

Test networks are available for deployment of IPC subnets.

### Filecoin Calibration Network

The [Filecoin Calibration network](https://docs.filecoin.io/networks/calibration) is the most realistic testnet simulation of the Filecoin mainnet. As the largest decentralized storage network, applications that require data storage and retrieval can leverage Filecoin's storage capabilities by connecting an IPC subnet to the Calibration net as the parent network.

The params to connect to Filecoin Calibration are:

* subnet\_id : `/r314159`
* provider\_http : `https://api.calibration.node.glif.io/rpc/v1`
* gateway\_addr : [![Gateway Address](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Fconsensus-shipyard%2Fipc%2Fcd%2Fcontracts%2Fdeployments%2Fr314159.json\&query=%24.gateway_addr\&label=Gateway%20Address)](https://github.com/consensus-shipyard/ipc/blob/cd/contracts/deployments/r314159.json)
* registry\_addr : [![Registry Address](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Fconsensus-shipyard%2Fipc%2Fcd%2Fcontracts%2Fdeployments%2Fr314159.json\&query=%24.registry_addr\&label=Registry%20Address)](https://github.com/consensus-shipyard/ipc/blob/cd/contracts/deployments/r314159.json)

The Chain ID for Filecoin Calibration is `314159`. A [faucet](https://faucet.calibration.fildev.network/) and [explorer](https://calibration.filfox.info/en) are available.


# IPC CLI

This page lists all the ipc-cli commands to interact with IPC subnets.

{% hint style="info" %}
For background and setup information, make sure to start with the [README](https://github.com/consensus-shipyard/ipc/blob/main/README.md) of IPC on GitHub.
{% endhint %}

If you have IPC installed on your machine, you should be able to run `ipc-cli --help` to check all the available commands.

```sh
ipc-cli --help

The IPC agent command line tool

Usage: ipc-cli [OPTIONS] [COMMAND]

Commands:
  config      config related commands
  subnet      subnet related commands such as create, join and etc
  wallet      wallet related commands
  cross-msg   cross network messages related commands
  checkpoint  checkpoint related commands
  util        util commands
  help        Print this message or the help of the given subcommand(s)

```

### Configuration

#### IPC initialization

```sh
ipc-cli config init
```

This command will initialize a new empty config file under `~/.ipc` with all parameters required to connect to the IPC rootnet network.

### Subnet Management

#### List active subnets

```
ipc-cli subnet list --subnet=<PARENT_SUBBNET_ID>
```

You can check all the active child subnets on a specific parent subnet.

This command only shows subnets that have been registered to the gateway, i.e. that have provided enough collateral to participate in the IPC protocol and haven't been killed. It is not an exhaustive list of all of the subnet actors deployed over the network.

<pre class="language-sh"><code class="lang-sh"># Example execution
<strong>$ ipc-cli subnet list --subnet /r314159
</strong>/r314159/t410fmdbc3kcv4gody6drgztmgwnzs2ryzwiazjzu5pq - status: Active, collateral: 5.0 FIL, circ.supply: 21.0 FIL, genesis: 1069882
</code></pre>

#### Create a child subnet

{% code overflow="wrap" %}

```
ipc-cli subnet create
    --parent <parent-subnet-id>
    --min-validator <MIN_VALIDATORS>
    --min-validator-stake <MIN_VALIDATOR_STAK>
    --bottomup-check-period <BOTTOMUP_CHECK_PERIO>
```

{% endcode %}

This command will create a subnet and create a corresponding contract based on the parameters specified with it. Make a note of the subnet-id for the subnet just created.

```sh
# Example execution
$ ipc-cli subnet create --parent /r314159 --min-validators 3 --min-validator-stake 10 --bottomup-check-period 30
[ipc_cli::commands::subnet::create] created subnet actor with id: /r314159/t410fylyzufn7lfg3q6zxpt6cdvq4yyiwm4tkaged2oy
```

#### Join a subnet as a validator

{% code overflow="wrap" %}

```sh
ipc-cli subnet join
    --subnet <subnet-id>
    --collateral <collateral_amount>
    --public-key <public_key_validator_addr>
    --initial-balance <genesis-balance>
```

{% endcode %}

This command specifies the subnet to join, the amount of collateral to provide, and the public key of the `--from` address that is joining as a validator.

<pre class="language-sh"><code class="lang-sh"># Example execution
<strong>$ ipc-cli subnet join --subnet /r314159/t410fh4ywg4wvxcjzz4vsja3uh4f53johc2lf5bpjo6i --collateral 10 --public-key 043385c3b9ab8a697cd7bec6ca623cbdd0fea1293e8b464df825b104eb58a44cc8efacc6a3482b866b85ecdf734b5d4ef5495737deb348625ce6a35536142d2955
</strong></code></pre>

To join a subnet and also include some initial balance for the validator in the subnet, you can add the `--initial-balance` flag with the balance to be included in the genesis.

```sh
# Example execution
$ ipc-cli subnet join --subnet=/r314159/t410fh4ywg4wvxcjzz4vsja3uh4f53johc2lf5bpjo6i --collateral=1 --public-key=043385c3b9ab8a697cd7bec6ca623cbdd0fea1293e8b464df825b104eb58a44cc8efacc6a3482b866b85ecdf734b5d4ef5495737deb348625ce6a35536142d2955 --initial-balance 0.5
```

#### Leave a subnet

```sh
ipc-cli subnet leave --subnet <subnet-id>
```

Leaving a subnet will release the collateral for the validator and remove all the validation rights from its account. This means that if you have a validator running in that subnet, its validation process will immediately terminate.

<pre class="language-sh"><code class="lang-sh"><strong># Example execution
</strong>$ ipc-cli subnet leave --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq
</code></pre>

#### Stake more collateral as a validator

```
ipc-cli subnet stake --subnet <subnet-id> --collateral <collateral_amount>
```

After initially joining a subnet with specified collateral, a validator can stake more collaterals to the subnet.

```sh
# Example execution
$ ipc-cli subnet stake --subnet=/r314159/t410fh4ywg4wvxcjzz4vsja3uh4f53johc2lf5bpjo6i --collateral=10
ipc_provider::manager::evm::manager] interacting with evm subnet contract: 0x60c2…d900 with collateral: 10000000000000000000
```

#### Unstake collateral from a subnet

```sh
ipc-cli subnet unstake --subnet <subnet-id> --collateral <collateral_amount>

# Example execution
$ ipc-cli subnet unstake --subnet /r314159/t410fmdbc3kcv4gody6drgztmgwnzs2ryzwiazjzu5pq --collateral 2
```

#### Claim the unstaked collateral

```
ipc-cli subnet claim --subnet <subnet-id>
```

Validators need to claim their collateral after they reduce collateral in the subnet through `unstake`

```sh
# Example execution
$ ipc-cli subnet claim --subnet=/r314159/t410fh4ywg4wvxcjzz4vsja3uh4f53johc2lf5bpjo6i
```

{% hint style="info" %}
Changes in collateral and the power table are not reflected immediately in the parent. They need to be confirmed in the execution of the next bottom-up checkpoint, so until this happens, even if there has been a change in collateral, you may not be the change immediately when running `ipc-cli subnet list`. This impacts any change to the collateral of validators, i.e. `stake`, `unstake` and `leave` commands.

To inspect the changes to the power table that have been performed between two epochs you can use the following command:

```
ipc-cli checkpoint list-validator-changes --from-epoch=<START_EPOCH> --to-epoch=<END_EPOCH>
```

{% endhint %}

#### Transfer tokens within a subnet

{% code overflow="wrap" %}

```sh
ipc-cli subnet send-value
    --subnet <subnet-id>
    [--from <from-addr>]
    --to <to-addr>
    <value>
```

{% endcode %}

You can use this command to send tokens between addresses of the same subnet. If `--from` is not specified, `ipc-cli` will send tokens from the default wallet address.

```sh
# Example execution
$ ipc-cli subnet send-value --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq --to 0x406a7a1d002b71ece175cc7e067620ae5b58e9ec 10
```

### Wallet Key Management

The `ipc-cli` has an EVM-compatible wallet that it uses to sign transactions and interact with IPC on behalf of specific addresses. This wallet type can also be used with FEVM.

#### Create new address

```
ipc-cli wallet new -w <wallet-type>
```

This command will create a wallet, and store the key information in `~/.ipc/evm_keystore.json`. You can create an EVM-compatible wallet which can also used with FVM (Filecoin Virtual Machine).

```sh
# Sample execution
$ ipc-cli wallet new --wallet-type evm
"0x406a7a1d002b71ece175cc7e067620ae5b58e9ec"
```

#### Check wallet balance

```sh
ipc-cli wallet balances --wallet-type <wallet-type> --subnet <subnet-id>
```

You can check the token balance in your wallet addresses for any active subnet configured in the `ipc-cli`.

<pre class="language-sh"><code class="lang-sh"># Sample execution
<strong>$ ipc-cli wallet balances --wallet-type evm --subnet /r314159
</strong>0x406a7a1d002b71ece175cc7e067620ae5b58e9ec - Balance: 100
</code></pre>

#### Set default wallet address

```
ipc-cli wallet set-default --wallet-type <wallet-type> --address <EVM-ADDRESS>
```

You can set a default address for your wallet so it is always the one used when the `--from` flag is not explicitly set.

```sh
# Sample execution
$ ipc-cli wallet set-default --wallet-type evm --address 0x406a7a1d002b71ece175cc7e067620ae5b58e9ec
```

#### Get the default wallet address

```sh
ipc-cli wallet get-default --wallet-type <wallet-type>

# Sample execution
$ ipc-cli wallet set-default --wallet-type evm
"0x406a7a1d002b71ece175cc7e067620ae5b58e9ec"
```

#### Export a wallet key

```sh
ipc-cli wallet export --wallet-type <wallet-type> --address <EVM-ADDRESS> > <OUTPUT_FILE>
```

This command will exporte a wallet private key which is stored in the `ipc-cli` keystore `~/.ipc/evm_keystore.json`.

```sh
# Sample execution
$ ipc-cli wallet export --wallet-type evm --address 0x406a7a1d002b71ece175cc7e067620ae5b58e9ec > /tmp/priv.key
exported new wallet with address 0x406a7a1d002b71ece175cc7e067620ae5b58e9ec in file "/tmp/priv.key"
```

* Export key encoded in based64 for Fendermint

  ```sh
  ipc-cli wallet export --wallet-type evm --address <EVM-ADDRESS> --fendermint > <OUTPUT_FILE>
  ```
* Export key in HEX

  ```sh
  ipc-cli wallet export --wallet-type evm --address <EVM-ADDRESS> --hex > <OUTPUT_FILE>
  ```

#### Import a wallet

```
ipc-cli wallet import --wallet-type evm --path <INPUT_FILE_WITH_KEY> --private-key <PRIVATE_KEY>
```

This command will import a wallet from an EVM key file with this format `{“address”:,“private_key”:<PRIVATE_KEY>}`.

```sh
# Sample execution
$ ipc-cli wallet import --wallet-type evm --path=~/tmp/wallet.key
imported wallet with address "0x406a7a1d002b71ece175cc7e067620ae5b58e9ec"
```

Import a wallet from the private key.

```sh
# Sample execution
$ ipc-cli wallet import --wallet-type evm --private-key=0x405f50458008edd6e2eb2efc3bf34846db1d6689b89fe1a9f9ccfe7f6e301d8d
imported wallet with address "0x406a7a1d002b71ece175cc7e067620ae5b58e9ec"
```

### Cross subnet messages

At the moment, the `ipc-cli` only expose commands to perform the basic IPC interoperability primitives for cross-net communication, which is the exchange of FIL (the native token for IPC) between the same address of a subnet. Mainly:

* `fund`, which sends native tokens from one public key address to the address in the child subnet.
* `release` that moves native tokens from one account in a child subnet to its counterpart in the parent.

#### Fund tokens in a child subnet

```sh
ipc-cli cross-msg fund
    --subnet <subnet-id>
    [--from <from-addr>]
    [--to <to-addr>]
    <amount>
```

This command includes the cross-net message into the next top-down proof-of-finality. Once the top-down finality is committed in the child, the message will be executed and you should see the funds in your account of the child subnet. If the `--to` is not set explicitly, the funds are sent to the address of the `--from` in the subnet.

```sh
# Example execution
$ ipc-cli cross-msg fund --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq 100
fund performed in epoch 1030279
```

Alternatively, we can pass an additional parameter to send the funds to a specific address in the child subnet.

```sh
# Example execution
$ ipc-cli cross-msg fund --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq --to=0x406a7a1d002b71ece175cc7e067620ae5b58e9ec 100
fund performed in epoch 1030279
```

#### Pre-fund subnet address in genesis

```
ipc-cli cross-msg pre-fund
    --subnet <subnet-id>
    [--from <from-addr>]
    she<amount>
```

To fund your address in a child subnet genesis before it is bootstrapped, and include some funds on your address in the subnet in genesis, you can use the `pre-fund` command. This command can only be used before the subnet is bootstrapped and started.

```sh
# Example execution
$ ./bin/ipc-cli cross-msg pre-fund --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq 1
```

#### Release funds from a subnet

```sh
ipc-cli cross-msg release
    --subnet <subnet-id>
    [--from <from-addr>]
    [--to <to-addr>]
    <amount>
```

This command will release funds to the parent subnet from its subnet. To release funds from a subnet, your account must hold enough funds inside this subnet.

This command includes the cross-net message into a bottom-up checkpoint after the current epoch. Once the bottom-up checkpoint is committed in the parent, you should see the funds in your account in the parent.

```sh
# Example execution
$ ipc-cli cross-msg release --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq 100
release performed in epoch 1023
```

Alternatively, we can pass an additional parameter to release the funds to a specific address in the parent subnet by setting `--to` address.

```sh
# Example execution
$ ipc-cli cross-msg release --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq --to 0x406a7a1d002b71ece175cc7e067620ae5b58e9ec 100
release performed in epoch 1030
```

#### Release initial subnet funds

```sh
ipc-cli cross-msg pre-release
    --subnet <subnet-id>
    [--from <from-addr>]
    <amount>
```

This command will recover some (or all) of the funds that were sent to a subnet through `pre-fund` to be included as a genesis balance for your address.

```sh
# Example execution
$ ipc-cli cross-msg pre-release --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq 0.1
```

#### Check parent subnet finality

```
ipc-cli cross-msg parent-finality --subnet <SUBNET_ID>
```

The epoch in which the message is performed can give you a sense of the time the message will take to be propagated. You can check the current finality in a subnet and wait for the finality height that includes your message to be committed.

```sh
# Example execution
$ ipc-cli cross-msg parent-finality --subnet /r314159/t410fmdbc3kcv4gody6drgztmgwnzs2ryzwiazjzu5pq
ipc_provider::manager::evm::manager] querying latest parent finality
1070541
```

#### List top-down messages

```sh
ipc-cli cross-msg list-topdown-msgs --subnet=<SUBNET_ID> --epoch=<EPOCH>
```

This command will list the top-down messages sent for a subnet from a parent network for a specific epoch.

```sh
# Example execution
$ ipc-cli cross-msg list-topdown-msgs --subnet /r314159/t410fmdbc3kcv4gody6drgztmgwnzs2ryzwiazjzu5pq --epoch 100450
```

### CheckPoint

#### List checkpoints for a subnet

```sh
ipc-cli checkpoint list-bottomup
    --from-epoch <range-start>
    --to-epoch <range-end>
    --subnet <subnet-id>
```

Subnets are periodically committing checkpoints to their parent every `bottomup-check-period` (parameter defined when creating the subnet). You can use this command to inspect the information of a range of bottom-up checkpoints committed in the parent for a subnet.

```sh
# Example execution
$ ipc-cli checkpoint list-bottomup --from-epoch 0 --to-epoch 100 --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq
epoch 0 - prev_check={"/":"bafy2bzacedkoa623kvi5gfis2yks7xxjl73vg7xwbojz4tpq63dd5jpfz757i"}, cross_msgs=null, child_checks=null
epoch 10 - prev_check={"/":"bafy2bzacecsatvda6lodrorh7y7foxjt3a2dexxx5jiyvtl7gimrrvywb7l5m"}, cross_msgs=null, child_checks=null
epoch 30 - prev_check={"/":"bafy2bzaceauzdx22hna4e4cqf55jqmd64a4fx72sxprzj72qhrwuxhdl7zexu"}, cross_msgs=null, child_checks=null
```

#### Check quorum-reached bottom-up checkpoints

```
ipc-cli checkpoint quorum-reached-events
    --from-epoch <range-start>
    --to-epoch <range-end>
    --subnet <subnet-id>
```

As with bottom-up messages, you can get a sense of the time that your message will take to get to the parent by looking at the epoch in which your bottom-up message was triggered (the output of the command) and listing the latest bottom-up checkpoints to see how far it is from being propagated.

This command will list the bottom-up checkpoints populated, signed, and agreed on their validity by a majority of validators in the child subnet.

```sh
# Sample execution
$ ipc-cli checkpoint quorum-reached-events --from-epoch 600 --to-epoch 680 --subnet /r314159/t410ffumhfeppdjixhkxtgagowxkdu77j7xz5aaa52vy
```

#### Check if bottom-up checkpoints are submitted

<pre><code><strong>ipc-cli checkpoint has-submitted-bottomup-height
</strong>    --subnet &#x3C;SUBNET_ID>
    --submitter &#x3C;RELAYER_ADDR>
</code></pre>

This command can be used to check the state of the checkpoints submitted from a subnet relayer. Once subnet validators have agreed on the bottom-up checkpoint to be submitted in the parent for a specific epoch, relayers need to pick up the checkpoint and submit it in the parent.

```sh
# Sample execution
$ ipc-cli checkpoint has-submitted-bottomup-height --subnet /r314159/t410ffumhfeppdjixhkxtgagowxkdu77j7xz5aaa52vy --submitter  0x406a7a1d002b71ece175cc7e067620ae5b58e9ec
```

#### List submitted bottom-up checkpoints

```
ipc-cli checkpoint list-bottomup-bundle
    --subnet <SUBNET>
    --from-epoch <FROM_EPOCH>
    --to-epoch <TO_EPOCH>
```

This command can be used to check the list of the bundle of bottom-up checkpoints and signatures populated and already signed by a child subnet for their submission to the parent on a window of heights.

```sh
# Sample execution
$ ipc-cli checkpoint list-bottomup-bundle --subnet /r314159/t410ffumhfeppdjixhkxtgagowxkdu77j7xz5aaa52vy --from-epoch 600 --to-epoch 680
```

#### Run a relayer

```
ipc-cli checkpoint relayer --subnet <SUBNET_ID> --submitter <RELAYER_ADDR>
```

IPC relies on the role of a specific type of peer on the network called the **relayers** that are responsible for submitting bottom-up checkpoints that have been finalized in a child subnet to its parent. Without relayers, cross-net messages will only flow from the top levels of the hierarchy to the bottom, but not the other way around.

```sh
# Example execution
$ ipc-cli checkpoint relayer --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq
```

To run the relayer from a different address you can use the `--submitted` flag.

<pre class="language-sh"><code class="lang-sh"># Example execution
<strong>$ ipc-cli checkpoint relayer --subnet /r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq --submitter 0x406a7a1d002b71ece175cc7e067620ae5b58e9ec
</strong></code></pre>

Relayers are rewarded through cross-net message fees for the timely submission of bottom-up checkpoints to the parent. Relayers can claim the checkpointing rewards collected for a subnet.

```sh
# Example execution
$ ipc-cli subnet claim --subnet=/r31415926/t4xwzbdu7z5sam6hc57xxwkctciuaz7oe5omipwbq --reward
```


# Troubleshooting

## Troubleshooting Cargo build issues

This project has a large set of dependencies and they are all bundled together in a root Cargo.lock file. This means that sometimes, when upgrading some of our dependencies, Cargo will do something unexpected which causes build errors which can be very time consuming to figure out.

### Example: failed to select a version for `xyz`

```
error: failed to select a version for `fvm_sdk`.
    ... required by package `frc42_dispatch v5.0.0`
    ... which satisfies dependency `frc42_dispatch = "^5.0.0"` of package `fil_actor_account v12.0.0 (/home/fridrik/workspace4/builtin-actors/actors/account)`
    ... which satisfies path dependency `fil_actor_account` (locked to 12.0.0) of package `fil_actor_miner v12.0.0 (/home/fridrik/workspace4/builtin-actors/actors/miner)`
    ... which satisfies path dependency `fil_actor_miner` (locked to 12.0.0) of package `fil_actors_integration_tests v1.0.0 (/home/fridrik/workspace4/builtin-actors/integration_tests)`
    ... which satisfies path dependency `fil_actors_integration_tests` (locked to 1.0.0) of package `test_vm v12.0.0 (/home/fridrik/workspace4/builtin-actors/test_vm)`
versions that meet the requirements `~4.0` are: 4.0.0

all possible versions conflict with previously selected packages.

  previously selected package `fvm_sdk v4.1.1`
    ... which satisfies dependency `fvm_sdk = "^4.1.0"` (locked to 4.1.1) of package `fil_actors_runtime v12.0.0 (/home/fridrik/workspace4/builtin-actors/runtime)`
    ... which satisfies path dependency `fil_actors_runtime` (locked to 12.0.0) of package `fil_actor_account v12.0.0 (/home/fridrik/workspace4/builtin-actors/actors/account)`
    ... which satisfies path dependency `fil_actor_account` (locked to 12.0.0) of package `fil_actor_miner v12.0.0 (/home/fridrik/workspace4/builtin-actors/actors/miner)`
    ... which satisfies path dependency `fil_actor_miner` (locked to 12.0.0) of package `fil_actors_integration_tests v1.0.0 (/home/fridrik/workspace4/builtin-actors/integration_tests)`
    ... which satisfies path dependency `fil_actors_integration_tests` (locked to 1.0.0) of package `test_vm v12.0.0 (/home/fridrik/workspace4/builtin-actors/test_vm)`
```

If you get this error, then it means that Rust could not find a version of the `xyz` crate which fulfills the requirements of the package and other packages that depend on it. To debug this, look what dependencies of `xyz` package are, and check if they need to be updated.

This error can happen for example when upgrading to a new major/minor FVM versions without upgrading also other dependencies like `frc_dispatch` which requires fvm as well. In that case we must upgrade the `frc_dispatch` package to use the same FVM version as we are using.

### Unexplained transitive dependencies in wasm32 target after upgrading FVM version

When upgrading FVM dependency (from 4.0 to 4.1) it resulted in our `fendermint/actors/build.rs` script to fail due Cargo including multiple new dependencies in the `wasm32` target which did not occur before and caused build errors since these new dependencies did not support Wasm target.

By running `cargo tree` we saw that these dependencies were pulled in from the `filecoin-proofs-api` required by `fvm_shared`. This dependency is pulled in when requiring `fvm_shared` with the `crypto` feature. Looking at our different Cargo.toml files, we noticed that `contract-bindings/Cargo.toml` file was the only one setting that feature. We needed to remove the `crypto` feature, compile, and then add it back in for the wasm build (and tests) to succeed.

## Troubleshooting Misc Cargo related issues

### Unexplained behaviour due to local changes in \~/.cargo/registry

<details>

<summary>Example: Failing integration tests due to local changes in ~/.cargo/registry</summary>

</details>

If you are seeing weird unexplained behaviour that you kind of can't wrap your head around, then you may want to delete your `~/.cargo/registry` and run `cargo build`. Here is why, you *might* have accidentally changed some of the crates's source files that cargo is using in your project. There is no way to know if you had made any local changes to any of these crates as \`Cargo\`\` does not maintain hash of these dependencies and there is no git repo available to compare against.


# FAQ

### Roadmap

**Q: What’s the roadmap of IPC, and will IPC have a beta version go live?**

The IPC team is working on delivering IPC features incrementally based on the roadmap planning in milestones. We expect to deliver a market-ready GA version by Milestone 3. The rough roadmap of IPC is:

* :white\_check\_mark: ***Milestone 1:** The testnet-ready version to deploy a Lotus-based IPC subnet.*
* :white\_check\_mark: ***Milestone2.5:** IPC preview.*
  * This milestone migrates away from the Eudico/Lotus stack to the Fendermint stack.
  * Developers can spin up new IPC subnets anchored on Filecoin's Calibration network for general purposes with the fast block time and finality, not customization at this stage.
* ***\[WIP] Milestone 3:** Production-grade manually-created, customizable L2+ networks*

  Audited and tested a version of the Fendermint stack that is safe to deploy production apps and move customers’ funds into (well-designed) subnets.
* ***Milestone 4**: Support for user-deployed Wasm actors + multi-subnet apps + QoL improvements.*
  * Developments from the FVM team make it safe to support user-deployed Wasm actors and runtimes.
  * Streamlines the operation of multi-subnet/federated apps by introducing multi-hop networks and subnet-aware wallets.
  * Network-wide and subnet explorers, plus better infra recipes are introduced at this stage.

**Q: IPC is currently compatible with Filecoin and Ethereum. Will more L1 chains be supported in the future?**

IPC is implemented to be compatible with Filecoin and EVM-compatible chains. The current focus is on improving the features, like general message passing, subnet customization, atomic upgrades, native multi-coin support, etc.

Once the IPC stack achieves stability and maturity to meet the requirements of general web3 scalability, detailed roadmaps will be established to support additional Layer 1 (L1) chains such as Bitcoin, Solana, etc.

***

### Design & Architecture

**Q: What are the best use cases of IPC?**

As a multi-chain scalability framework, IPC offers hyper customization, lower fees, and faster transactions for decentralized technology. It offers enhanced performance for decentralized ecosystems and dApps e.g. decentralized computation, AI/ML, and metaverse/gaming use cases, etc. Read more about different use cases of IPC [here](/overview/use-cases).

**Q: What is the difference between IPC and other Ethereum L2 solutions (e.g. Polygon, Arbitrum, and Optimism) How does it compare to rollups**?

IPC is a flexible scalability framework that can be configured to resemble rollups, side chains, and more. Using IPC, developers can build customized scaling L2 or L3 solutions (operating as independent side chains) on-demand while anchoring their trust to the root chains, like Filecoin, Ethereum, or even BTC in the future.

IPC achieves horizontal scalability by using a hierarchical subsystem of subnets. This allows for flexible scaling throughout the network and is not limited to the rootnets on Layer 2 solutions.

**Q: What is the advantage of using IPC for scalability?**

* Highly customizable on consensus, runtime, gas fee, throughput, and membership.
* Inherit security and other unique features from the rootnet.
* Multi-chain interoperability via FVM WASM-based polyglot execution environment.
* Efficient chain storage with [IPLD](https://spec.filecoin.io/libraries/ipld/)

Read more in [How IPC Compares](https://docs.ipc.space/overview/how-ipc-compares).

**Q: Is deploying an IPC subnet more convenient for accessing the Filecoin storage network?**

Supporting data storage and retrieval on Filecoin, natively in an IPC subnet, requires specific system calls enabled from Filecoin to IPC and is in the roadmap.

**Q: How does the checkpoint work?**

To learn more about checkpointing [here](/concepts/subnets/parent-child-interactions#checkpointing).

***

### Subnet customization

**Q: Is it possible to launch an IPC subnet without a pledge required and block rewards?**

Yes, it is possible since pledging and block rewards are configurable for an IPC subnet.

When you launch a subnet, the default configuration is to use FIL as the native token and not include block rewards. But you can configure the subnet to use any ERC20 token as the native token for the utility, fee, and block rewards in that subnet.

For the pledge, you have a couple of different subnet modes to choose from when you set up a subnet.

* **Collateral**: Validator power is determined by the collateral staked, and then you need an arbitrary positive pledge.
* **Federate**: Validator power is assigned by the owner of the subnet, with no need for a pledge.
* **Static**: Validators need to provide a fixed initial pledge to join the subnet when the subnet is set up. After that, no validators can join the subnet.

**Q: Can we modify chain parameters like the message size limits and gas price within the subnet?**

Currently, IPC utilizes the default values from Filecoin Mainnet, including the gas model and block limits.

However, a core feature of IPC is its high customizability. As part of the future roadmap, there are plans to introduce mechanisms that allow users to configure these chain parameters for a subnet.

**Q: Can we configure the logic for selecting subnet validators based on the geography or physical location of nodes?**

The concept behind the subnet actor in IPC's parent architecture is to allow for the inclusion of specific logic and requirements for individual subnets. While anyone can become a user of a subnet, the subnet administrators can restrict the pool of validators and define the capabilities of validators that can participate in their subnet.

The development plan is to incrementally add interfaces and "levels" to help subnet admins configure their subnet capabilities. This allows subnet admins to have a greater degree of control over the validators and capabilities of their subnets, enabling them to tailor the geography and physical node selection to align with their intended use cases and objectives.

***

### Deploying & joining a subnet

**Q: Can a node join the IPC subnet as a validator? Does it require staking or depositing enough funds?**

When a subnet is created by its maintainer, the subnet permission mode will be configured for this subnet. The available options are collateral, federal, and static. So whether or not a node can join an IPC subnet depends on its permission mode configuration.

When creating a subnet in IPC, the subnet permission mode can be configured by its maintainer. There are three available options for the permission mode: collateral, federal, and static. The permission mode determines how a node can join the subnet based on its configuration.

* **Collateral**: validators are required to provide collateral or meet the criteria to join the subnet.
* **Federate**: validators are assigned by the owner of the subnet. Not any node can join it as a validator.
* **Static**: Once a static subnet is created, a node cannot join that IPC subnet anymore.

**Q: How to leave a subnet and release the funds properly?**

We can use the following command to leave a subnet as a validator.

```sh
ipc-cli subnet leave --subnet <subnet-id>
```

Leaving a subnet will release the collateral for the validator and remove all the validation rights from its account. This means that if you have a validator running in that subnet, its validation process will immediately terminate.

If an IPC account holds enough funds inside this subnet, it can release funds to the parent subnet from its subnet with `ipc-cli cross-msg release` command. Refer to [Release funds from a subnet](/reference/ipc-cli-usage#release-funds-from-a-subnet) for more details.

**Q: Who is responsible for generating and signing a checkpoint transaction for the parent subnet?**

Subnets are periodically committing checkpoints to their parent every `bottomup-check-period` (parameter defined when creating the subnet) by the subnet validators. A bottom-up checkpoint will be submitted after it is populated, signed, and agreed on their validity by a majority of validators in the child subnet.

Once subnet validators have agreed on the bottom-up checkpoint to be submitted in the parent for a specific epoch, relayers need to pick up the checkpoint and submit it in the parent. Then relayers are rewarded through cross-net message fees for the timely submission of bottom-up checkpoints to the parent.

**Q: What is the difference between ISA and IGA?**

**ISA (a.k.a subnet actor)** is subnet-specific and user-defined, which means that subnet operators are free to implement their own. They define the specific logic for the subnet such as the collateral policy, minimum number of validators, or any other subnet-specific logic that needs to be implemented. This contract lives in the parent of the subnet.

**The IGA (a.k.a gateway actor)** is the contract that implements the logic for the IPC protocol. There is one per subnet and depending on if the network is behaving as a root network, a parent, or a child, it triggers different logic to propagate messages, enforce collateral requirements, etc.

Read more at [IPC Actors](/overview/architecture#ipc-actors).

***

### Developing on Subnet

**Q: Can we deploy smart contracts within a subnet?**

IPC used FVM (Filecoin virtual machine) as the runtime environment for the IPC subnet. Since FVM is built as a polyglot VM to enable on-chain programmability, IPC is initially compatible with Filecoin and Ethereum.

FVM is the transaction execution layer of the IPC subnet, allowing builders to deploy their smart contracts on top of subnets to build any use cases. Since FVM is EVM-compatible, builders can use EVM-compatible toolings to develop smart contracts for a subnet, such as solidity libraries, Hardhat, foundry, MetaMask, etc.

**Q: How to send a transaction between a subnet and its parent subnet?**

At the moment, the `ipc-cli` only expose commands to perform the basic IPC interoperability primitives for cross-net communication, which is the exchange of FIL (the native token for IPC) between the same address of a subnet. Mainly:

* `fund`, which sends native tokens from one public key address to the address in the child subnet.
* `release` that moves native tokens from one account in a child subnet to its counterpart in the parent.

To learn how to use those `ipc-cli` commands for [cross-subnet messages](/reference/ipc-cli-usage#cross-subnet-messages).

The smart contract interaction between subnets is achieved by using GMP (General Massage Passing), you can learn more about GMP here.

***

### Cost & Performance

**Q: Are the gas usage costs the same as the Filecoin mainnet (independent from gas price)?**

Currently, IPC adopts the default values from the Filecoin mainnet, including the gas model and block limits. However, as part of the roadmap, there are plans to introduce mechanisms that allow for the configuration of gas-related parameters within IPC.

**Q: What is the current maximum TPS the subnet can achieve? And how far is it from the target?**

IPC client (Fendermint) uses Tendermint under the hood which potentially can reach [10k tps](https://github.com/tendermint/tendermint/wiki/Benchmarks) in single-node lab conditions as a theoretical limit.

In practice, the actual TPS achieved with Tendermint can vary depending on various factors, including the number of validators in the network. As the number of validators increases, the TPS tends to decrease. Real-world deployments of Tendermint have been known to achieve TPS numbers around 1,000 to 4,000.

Those numbers are theoretical and might not be possible if we benchmark an IPC subnet since we've been focusing on adding features instead of performance optimizations.