Data storage is only useful if the data can be reliably recovered over many years without the data owner’s involvement. Most decentralized storage solutions have not yet overcome the challenge of offering contracts for more than a few months. This is because short storage periods help make sure that data can be retrieved reliably. PIPE allows safe retrieval over a much longer time span, because the network itself can fix issues without the user getting involved.

Long-term reliability is hard to achieve in decentralized networks, because it’s hard to get nodes to commit while still giving them the freedom that comes with a decentralized network. It’s one of the reasons why decentralized storage protocols typically offer storage periods of three months or less. PIPE solves this with a different, more flexible storage model that allows nodes to transfer responsibility for chunk storage to other nodes.

A significant advantage for PIPE users is that they do not have to actively monitor their chunks across the network, which is not easy to do. The overhead of these searches would severely limit the scalability and affordability of the network.

Besides this solution for long-term storage, PIPE also offers various concepts for better security, data retrieval, network resilience, and fair treatment of nodes’ interests. Because a happy node is a node that stays.

What ensures data retrieval?

In a decentralized network, you can’t predict what can go wrong. Therefore, PIPE introduces incentives for nodes and users to remain honest, while also having financial and reputational penalties for misbehavior. Incentives and penalties are balanced in a way that introduces as little friction as possible for honest users.

  • Nodes are paid to store, and punished if they don’t store correctly
    Paid storage chunks are guaranteed to be retrievable through a collateral deposit. The node’s stake will be slashed in case proof of storage or delivery can’t be satisfied, similar to FileCoin/SiaCoin/Swarm. If the node proves it stores its chunks correctly, it gets rewarded from the storage credit connected to the chunks.
  • Chunk storage doesn’t rely on a node staying in the network forever

PIPE’s proximity based storage algorithm means that a chunk doesn’t have a fixed location, only an approximate one. This allows a node to transfer the responsibility of storing a chunk, without losing its stake. It can simply ask a neighbor to accept the commitment to store it, along with the remaining Storage Credit attached to the chunk. This all happens without the user getting involved.

  • Nodes build a reputation that is hard to gain and easy to lose

PIPE’s reputation system allows nodes to charge a premium for reliably storing chunks, as proven by their history. This incentive contributes to making sure a node remains its previous uptime and reliable storage in the future, since it ensures the likelihood of receiving new, high value chunks. 

  • PIPE only requires the minimum redundancy for retrieval
    Developers can choose to store extra sets to ensure sufficient backups, depending on the use case. This of course multiplies the cost of storage, but it’s not a linear equation. We can see storage requirements get creative with reputation demands, storing one set on a highly reputable node, and a secondary set on a cheaper, lower reputation node.
  • You can choose to store the data on a node you trust
    Pinning is a special API function that allows a node to store all transactions from a particular address, for instance your own node or a node service that is paid to take care of this. Pinning data to this node gives a separate backup, with provable immutability under more controllable conditions. 
  • You can store on your own network
    When relying on the PIPE disc is unwanted or non-compliant, your own private PIPE disc can be set up in a trusted setting. This removes some benefits like resilience and censorship resistance, but it is up to the data owner to decide the best practice for their use case. 

Network characteristics that make PIPE reliable

PIPE removes as many single points of failure as possible. These features together create a resilient and fair network for both storage nodes and the users who store data on the network.

  • The network topology incentivizes nodes to be good neighbors
    Nodes are connected based on the proximity to their address. As a result, you build your reputation with your neighbors and can’t take it elsewhere. This gives a strong incentive to treat neighbors fairly – and not spam them – because there aren’t that many nodes you can naturally connect with. 
  • Data doesn’t have a fixed location that can be lost over time.

Chunks of files have a predetermined approximate storage location, based on a hash of the chunk. The node address closest to this hash gets to store the file, if they want. This makes chunks retrievable even if the make-up of the network changes over time.

  • Built-in error correction and a self-healing network
    Erasure coding provides a first layer of redundancy from forward error correction. A future feature will introduce self-restoring properties, by enabling storers of remaining chunks to recreate missing chunks – without the data owner having to interfere. This is incentivized through the slashed stake that the original node lost.
  • PIPE offers permissionless participation, meaning users can’t be censored

Anyone with the PIPE token can not only store data, but also request its retrieval. The way data flows through the network is by nodes asking their neighbors if they have a certain chunk, or if they know where to find it. Nodes do not know if the request comes from an end-user, or if it is relayed from another node. This creates plausible deniability and, in practice, makes sure the network is accessible for anyone. There is no central entity keeping a central index on what is stored where, or who joins the network. 

Balancing uptime and accidents

PIPE strives to be fair to all users – both nodes and end-users – by balancing the needs of a reliable network with the reality of parts of a system failing. Users can be sure that their data can be retrieved, while node runners can rely on a fair business model.

  • Node uptime requirements are strong, but not absolute

Nodes must be quasi-permanent. This means that they should be online 24/7, but small accidents can happen without grave consequences. 

  • Long term storage does not depend on long term node participation

With the commitment to storing comes the risk of slashing the stake when a node violates terms of storage. However, if a node wants to or has to stop storing, it can transfer chunk storage to a willing node. This resolves their commitment to storing in the smart contract, and opens up long term storage contracts for users, since chunks can be exchanged with neighbors close to the chunk hash.

  • Reputation is not linear, but has some leniency built in

The exact algorithm is still to be determined, but “quasi-permanence” gives away the intention to not punish nodes the second they go offline. The envisioned solution requires a game-theoretical analysis, to confirm it directs behavior as intended, without introducing dangerous attack vectors.

Conclusion

PIPE offers long-term decentralized storage with reliable retrieval capacity. The PIPE network is secure, incentivizes nodes to stay honest and online, but also has the capacity to deal with unforeseen events. Offering the ability to transfer the commitment for storing a chunk to another node is an important feature for being able to offer long-term reliable storage. The network also has self-healing properties that can restore lost chunks of data. Its general distributed design removes single points of failure and adds to the resilience of the network.

These are game-changing features for user-friendliness in decentralized storage. This is because they allow the network to deal with the effects of nodes dropping out for whatever reason without the data owner having to spring into action.

Join us!

Are you building web2.5 or web3 apps that require long-term, reliable data storage? Subscribe to newsletter or join our Discord! Can’t wait to test PIPE? Sign up for our Beta test.