A common problem with blockchain and decentralized storage is that once it starts to become popular, it tends to be hard for it to stay cheap and fast. This is not a unique insight, but it is difficult to fix. The way PIPE has solved it for decentralized data storage and data transfers is good for the overall adoption of IoT, DID and NFT use cases.

PIPE decentralized storage can grow in a cost-effective way because it does not have proof of work (PoW) overhead. It also stores its data on a subset of nodes in the network. Overhead like PoW and storage on every node become impossible to keep up in a network of thousands or even millions of nodes.

PIPE can also maintain stable prices because, unlike blockchains with PoW consensus, there is no bottleneck that can cause a price surge. Another way to control costs at scale is by only storing and transferring data that someone is willing to pay for in some way.

How does PIPE store data so effectively?

PIPE solves the blockchain scalability problem by eliminating redundancy and optimizing for data storage in its lightweight design:

  • Unlike traditional blockchains, PIPE is designed for data storage
    PIPE runs a heavily optimized data structure on L2. It’s a data first tangle, with network consensus removed. The consensus part takes place on a L1 network instead – IOTA in this case – whenever needed. This enables PIPE to scale for data storage, while still keeping the functions that require consensus available.
  • The data is stored subset, not on every node in the network
    The algorithm dictates an approximate location, independent of node count. The closest available node to the chunk hash is expected to store the file and collect rewards for storage.
  • PIPE distributes the data in chunks over the network
    Through Erasure Code (EC), PIPE slices data and scatters it uniformly over the network. The chunk location is a predetermined approximate storage location, based on a hash of the chunk.
    PIPE uses a DISC (Distributed Immutable Store for Chunks) to store data. The PIPE DISC is a modified Kademlia distributed hash table to accommodate differently spec’d nodes. Different specifications can, for instance, be SSD or HDD. Both types of storage have different latency characteristics and cost models, but which one is better depends on the use case. PIPE doesn’t dictate what is optimal; instead it gives freedom of choice.
  • Data does not have to be stored forever
    PIPE’s data channels are optimized through the Freighter protocol. Freighter is snapshot resistant. This means it can have empty (or removed) messages, while leaving the data trail intact. Most decentralized storage and blockchain networks can’t do this and will require ever-growing data storage to keep working, even for data that has lost its economic incentive.
  • Freighter makes decentralized searches seem as fast as normal downloads
    Freighter can download blockchain messages in parallel instead of downloading the oldest message first and then progressively looking for the next one. It can quickly search through hundreds or even thousands of messages in a channel to find the relevant data using a binary search. Relevant in this case can for instance mean the most recent message, or the oldest still present message in a pruned ledger.
    Better data design can make it even easier to find the start of a data set. This however is outside the protocol’s scope and should be done case-by-case.

Increasing scalability by lowering payment costs

Blockchains usually need a very high level of security and agreement because they solve problems of different importance and priority with the same mechanism. This increases costs and congestion. PIPE’s design separates these functions and has a more tailored approach to each problem from the perspective of data storage. It reduces congestion and the costs of usage, while maintaining security and consensus standards.

  • Service-for-service node traffic eliminates on-chain overhead
    Neighbors keep track of each other’s traffic through a Debt Meter. Traffic moving the meter too much in one direction means the other party has to pay for the disbalance. The Debt Meter allows nodes to barter the majority of their interactions, resulting in less network overhead.
  • Files are only stored as long as the prepaid credit lasts
    PIPE offers data persistence by requiring files to be stored for as long as somebody prepays for its storage. After credits run out, it is up to the node to decide if it wants to discontinue storing it, without risking a penalty.
  • All data can be uploaded to the network, but only paid data must be stored
    All files are uploaded into the cache. For paid storage, a storage credit is added. Its value is gradually released over the storage time. This rewards the node for correctly storing a chunk of data. After the stamp runs out, files are returned to the cache. Files in the cache may be purged, but a node can also wait for a new storage credit to be added. It will simply overwrite the data when it could instead store a paid chunk or a chunk with a higher chance of receiving a reward.
  • Buy and settle storage credit in bulk to reduce smart contract fees
    Storage is paid through a smart contract, but not every bit of storage requires a smart contract interaction, because this would make storage unaffordable from smart contract fees alone. Instead this credit decays over time, and the node receiving it can choose when it settles the smart contract. This gives both users and nodes control over how often they use a smart contract and have to pay the associated fees.
  • Apply credit to a data channel instead of individual data chunks
    Storage credit can be applied to pay for all chunks sent to a data channel by whoever has permission to write. Such a design reduces overhead and complexity, because each user/device/private key doesn’t have individually own its own tokens. Instead, storage on many data channels can be prepaid for by a different entity. This still keeps data channels decentralized, because only the owner can write there, but it allows for flexibility in how data storage is paid for. Having this flexibility enables different forms of payment and regulatory compliance, because the owner of the data channel does not hold cryptocurrency. 
  • PIPE only stores data as long as it’s deemed worthwhile
    One big advantage of how the PIPE token works is that it enables selective storage duration. By having users decide what data to pay for, there is no need for a central decision maker to determine which data needs to be kept and which data can be discarded. By only storing relevant data, the network size and cost are kept in check.

Conclusion

Once a blockchain becomes popular, it can become expensive and slow to use. PIPE solves this with a lightweight design that removes standard cryptocurrency redundancy and instead optimizes towards data storage. This makes decentralized data storage and transfers a feasible option in IoT, DID, and NFT use cases. The way it does this lets storage grow with demand, without giving way to unpredictable costs.

CTA

Do you need scalable, decentralized storage that will remain affordable as your data volume grows? To stay up to date on our progress, sign up for our newsletter or join us on our Discord server. Can’t wait to test PIPE? Sign up for our Beta test.