Storage - Workertown

Most Workertown services require some kind of persistent storage, i.e. data that lives between requests. Following the "sensible defaults" paradigm, each of these services provides a default storage adapter built upon Cloudflare's D1 database but there are other options available, and also the ability to bring your own storage adapter entirely.

How it works

Workertown services that require storage expose a runtime configuration option that allows you to specify a storage adapter to use.

import { search } from "@workertown/search";
import { SqliteStorageAdapter } from "@workertown/search/storage/sqlite";

export default search({
  runtime: {
    storage: new SqliteStorageAdapter({
      // Options go here...
    }),
    // Other options go here...
  }
});

Built-in storage adapters

Every Workertown service with storage support has a set of supported first-party storage adapters provided within the package.

D1 (default)

The Cloudflare D1 storage adapter is the default storage adapter for all Workertown services, and so any Workertown service that requires storage will require a bound D1 database to run by default.

The D1StorageAdapter is exposed from each package, but in reality you should never have to manually instantiate this adapter.

import { search } from "@workertown/search";
import { D1StorageAdapter } from "@workertown/search/storage/d1";

export default search({
  runtime: (options, env) => ({
    storage: new D1StorageAdapter({ d1: env.D1 }),
    // Other options go here...
  }),
});

[[d1_databases]]
binding = "DB"
database_name = "example_db"
database_id = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
preview_database_id = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

DynamoDB

DynamoDB is AWS's NoSQL database service that provides a REST API for interacting with databases.

import { search } from "@workertown/search";
import { DynamoDBStorageAdapter } from "@workertown/search/storage/dynamodb";

export default search({
  runtime: {
    storage: new DynamoDBStorageAdapter({
      credentials: {
        accessKeyId: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", // The AWS access key ID to use for authentication
        secretAccessKey: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", // The AWS secret access key to use for authentication
      },
      region: "...", // The AWS region the DynamoDB table is located in
      table: "example_table", // Optionally, the name of the DynamoDB table - otherwise each service has a default
      options: {
        billingMode: "PROVISIONED",
        readCapacityUnits: 1,
        writeCapacityUnits: 1,
      }, // Optionally, additional options about the table's billing mode and capacities - otherwise it will set the billing mode to PAY_PER_REQUEST
    }),
    // Other options go here...
  },
});

The DynamodDBStorageAdapters provide migration support, which will provision the table if it does not already exist, using the table and options provided to set the table name and billing mode/capacities. If you would like to manage the table yourself, you can omit options and simply provide the table name for the adapter to use.

Each DynamodDBStorageAdapter has a dependency on the @aws-sdk/client-dynamodb and @aws-sdk/lib-dynamodb packages, and therefore you must install them as dependencies of your project.

npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb

Planetscale

Planetscale is a hosted MySQL database that provides SQL (Vitess) databases that are accessible via a REST API - meaning they are edge runtime compatible.

import { search } from "@workertown/search";
import { PlanetscaleStorageAdapter } from "@workertown/search/storage/planetscale";

export default search({
  runtime: {
    storage: new PlanetscaleStorageAdapter({
      url: "...", // The URL of the Planetscale database
      username: "username", // The Planetscale username to use for authentication
      password: "password", // The Planetscale password to use for authentication
    }),
    // Other options go here...
  },
});

Each PlanetscaleStorageAdapter has a dependency on the @planetscale/database package, and therefore you must install this package as a dependency of your project.

npm install @planetscale/database

Sqlite

If you are running your Workertown service in a NodeJS environment, you can use Sqlite as your storage adapter to persist data directly on the disk in your environment.

import { search } from "@workertown/search";
import { SqliteStorageAdapter } from "@workertown/search/storage/sqlite";

export default search({
  runtime: {
    storage: new SqliteStorageAdapter({
      db: "db.sqlite", // The path to the Sqlite database file
    }),
    // Other options go here...
  },
});

Each SqliteStorageAdapter has a dependency on the better-sqlite3 package, and therefore you must install this package as a dependency of your project.

npm install better-sqlite3

Turso

Turso is a hosted Sqlite (libsql) database provider that provides a REST API for interacting with Sqlite databases.

import { search } from "@workertown/search";
import { TursoStorageAdapter } from "@workertown/search/storage/turso";

export default search({
  runtime: {
    storage: new TursoStorageAdapter({
      url: "...", // The URL of the Turso database
      authToken: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", // The Turso auth token to use for authentication
    }),
    // Other options go here...
  },
});

Memory (dev only)

For situations where you would like to only simulate storage (such as when running tests), you can use the MemoryStorageAdapter to store data in memory.

This will not persist any data to disk, and therefore any values stored are lost when the process is terminated.

import { search } from "@workertown/search";
import { MemoryStorageAdapter } from "@workertown/search/storage/memory";

export default search({
  runtime: {
    storage: new MemoryStorageAdapter(),
    // Other options go here...
  },
});

For development use only

The memory storage adapter is not intended for production use of any kind, and should only be used in development/test environments.

Bring your own storage adapter

Each Workertown service with storage support exposes a StorageAdapter class that can be extended upon to create your own storage adapter.

import { search } from "@workertown/search";
import { StorageAdapter } from "@workertown/search/storage";

class CustomStorageAdapter extends StorageAdapter {
  // Implement the required methods here...
}

export default search({
  runtime:{
    storage: new CustomStorageAdapter(),
    // Other options go here...
  },
});

Each StorageAdapter class exposes a different set of methods that must be implemented for full storage support in a particular Workertown service.

Migrations

For storage adapters built upon relational databases (or DynamoDB), each Workertown service exposes an endpoint to run the migrations required to run the service. The endpoint takes a POST request with an empty JSON body to run the migrations.

By default, this endpoint is exposed at /v1/admin/migrate, but this can be configured via the endpoints.v1.admin configuration option.

import { search } from "@workertown/search";

export default search({
  endpoints: {
    v1: {
      admin: "custom-admin", // Migrations can now be run from /custom-admin/migrate
    }
  }
});

Run this first

When first deploying a Workertown service with storage support, you must run the migrations via the /v1/admin/migrate endpoint before the service will work correctly.