Creating and Sharing Hypercores

This walkthrough will go through the basics of using Hypercore as a standalone module.

Introduction

Hypercore is the bread-and-butter of the Hypercore Protocol. It is a lightweight, secure append-only log with several powerful properties:

1. Secure: Hypercore builds a Merkle tree out of its blocks, so readers can always verify that the log hasn't been tampered with.
2. Easy Replication: The replicate method returns a Duplex stream that can be piped over an arbitrary transport stream.
3. On-Demand Downloading: With "sparse mode", readers will download blocks from peers when they are first requested.
4. Caching: Once peer has downloaded a block, it will be cached locally. The download will only happen once.
5. Bandwidth Sharing: As with BitTorrent, readers download blocks from many connected peers in parallel.
6. Live Updating: Peers can be notified whenever a Hypercore has grown.

A Hypercore can only have a single writer on a single machine; the creator of the Hypercore is the only person who can modify to it, because they're the only one with the private key. That said, the writer can replicate to many readers, in a manner similar to BitTorrent.

Unlike with BitTorrent, a Hypercore can be modified after its initial creation, and peers can receive live notifications whenever the writer adds new blocks.

In this walkthrough, we'll create two Hypercores: one writable and one that's a read-only clone of the first, which will simulate a remote peer. We'll then show you how to initiate replication between the two cores, and have the reader download and display live updates.

Walkthrough

If you want to follow along with the code, setup the walkthrough repo:

git clone https://github.com/hypercore-protocol/walkthroughs.git
cd walkthroughs/hypercore
npm install

This walkthrough only has two dependencies: hypercore and hypercore-promisifier, which gives you a Promises interface for Hypercore.

Step 1: Create a Writable Hypercore

The Hypercore constructor has two main parameters, storage and key, as well as a handful of options, several of which we'll work with in the next steps.

Let's create a new core with UTF-8 encoded blocks, stored in the ./main directory:

const core = toPromises(hypercore('./main', {
  valueEncoding: 'utf-8' // The blocks will be UTF-8 strings.
}))

Hypercore currently only has a callback API, so we use the hypercore-promisifier module to wrap it in a Promises API. This keeps things more concise.

We can append two new blocks to our core using the append method, which accepts either a single block or an Array of blocks:

await core.append(['hello', 'world'])

Run this step (full code):

node step-1.js

Now that our main core has two blocks in it, let's create a clone and start replicating.

Step 2: Create a Read-Only Clone

Let's simulate a remote peer by creating a read-only clone of the Hypercore from the previous step.

const clone = toPromises(hypercore('./clone', core.key, {
  valueEncoding: 'utf-8',
  sparse: true, // When replicating, don't eagerly download all blocks.
}))

This time, we pass a key parameter into the constructor to indicate that we're loading an existing hypercore rather than creating a new one. Since this clone is going to be downloading data from the original core, we've specified sparse: true, described below.

The replicate method can be used to create a Duplex stream that implements the Hypercore Wire Protocol. We can pipe together two replication streams, one from the original core and one from the clone, in order to start exchanging blocks:

const firstStream = core.replicate(true, { live: true })
const cloneStream = clone.replicate(false, { live: true })

// Pipe the stream together to begin replicating.
firstStream.pipe(cloneStream).pipe(firstStream)

These replication streams are end-to-end encrypted using the NOISE protocol. The first argument (true or false) indicates which peer initiated the replication process. Both replication streams are "live", meaning they'll continue replicating indefinitely.

Now that the clone has a connected peer, we can start requesting blocks. These blocks will be lazily downloaded, because the clone was constructed in sparse mode:

console.log('First clone block:', await clone.get(0)) // 'hello'
console.log('Second clone block:', await clone.get(1)) // 'world'

Let's append 100 more blocks, then request the last block from the clone. When this is done, the clone will have blocks 0, 1, and 101 available locally.

for (let i = 0; i < 100; i++) {
  await core.append(`New Block ${i}`)
}
await clone.update()
await clone.get(clone.length - 1) // New Block 99

Run this step (full code):

node step-2.js

Next Steps

In the Hyperswarm Walkthrough, we'll see how to replicate Hypercores to remote peers over a P2P network.