ShardMap
The intros Sessions beat — a key lives on someones node; get forwards to the owner via Hyperlink.peers — is a pattern every multi-droplet app reinvents. ShardMap is that pattern as a Hyperlink factory: declare key / value, distribute across app/Droplet* nodes, and every routed get / put / delete finds the owner. Leaf *Local ops stay on this shard. Fleet folds report sizes. An unreachable owner degrades to a miss — never a silent write on the wrong droplet.
Declare the map
Schemas on the Tag. keyOf extracts the partition key from a value (routed put). Partition strategy is a runtime option on serve / layer (default: ShardMap.consistentHash).
class class DropletEastDropletEast extends import NodeNode.Tag<class DropletEastDropletEast>()("app/DropletEast") {}
class class DropletWestDropletWest extends import NodeNode.Tag<class DropletWestDropletWest>()("app/DropletWest") {}
class class DropletCentralDropletCentral extends import NodeNode.Tag<class DropletCentralDropletCentral>()("app/DropletCentral") {}
const const SessionId: Schema.StringSessionId = import SchemaSchema.const String: Schema.StringType-level representation of
String
.
Schema for string values. Validates that the input is typeof "string".
@categorymodels@since4.0.0@categoryschemas@since4.0.0String
const const Session: Schema.Struct<{
readonly id: Schema.String;
readonly userId: Schema.String;
readonly seat: Schema.optionalKey<Schema.String>;
}>
import SchemaSchema.function Struct<{
readonly id: Schema.String;
readonly userId: Schema.String;
readonly seat: Schema.optionalKey<Schema.String>;
}>(fields: {
readonly id: Schema.String;
readonly userId: Schema.String;
readonly seat: Schema.optionalKey<Schema.String>;
}): Schema.Struct<{
readonly id: Schema.String;
readonly userId: Schema.String;
readonly seat: Schema.optionalKey<Schema.String>;
}>
Defines a struct schema from a map of field schemas.
Details
Each field value is a schema. Use
optionalKey
or
optional
to
mark fields as optional, and
mutableKey
to mark them as mutable.
The resulting schema's Type is a readonly object type with the fields'
decoded types. The Encoded form mirrors the field schemas' encoded types.
Example (Defining a basic struct)
import { Schema } from "effect"
const Person = Schema.Struct({
name: Schema.String,
age: Schema.Number,
email: Schema.optionalKey(Schema.String)
})
// { readonly name: string; readonly age: number; readonly email?: string }
type Person = typeof Person.Type
const alice = Schema.decodeUnknownSync(Person)({ name: "Alice", age: 30 })
console.log(alice)
// { name: 'Alice', age: 30 }
@categoryconstructors@since3.10.0Struct({
id: Schema.Stringid: const SessionId: Schema.StringSessionId,
userId: Schema.StringuserId: import SchemaSchema.const String: Schema.StringType-level representation of
String
.
Schema for string values. Validates that the input is typeof "string".
@categorymodels@since4.0.0@categoryschemas@since4.0.0String,
seat: Schema.optionalKey<Schema.String>seat: import SchemaSchema.const optionalKey: optionalKeyLambda
<Schema.String>(self: Schema.String) => Schema.optionalKey<Schema.String>
Type-level representation returned by
optionalKey
.
Creates an exact optional key schema for struct fields. Unlike optional,
this creates exact optional properties (not | undefined) that can be
completely omitted from the object.
Example (Creating a struct with optional key)
import { Schema } from "effect"
const schema = Schema.Struct({
name: Schema.String,
age: Schema.optionalKey(Schema.Number)
})
// Type: { readonly name: string; readonly age?: number }
type Person = typeof schema["Type"]
@categorymodels@since4.0.0@categorycombinators@since4.0.0optionalKey(import SchemaSchema.const String: Schema.StringType-level representation of
String
.
Schema for string values. Validates that the input is typeof "string".
@categorymodels@since4.0.0@categoryschemas@since4.0.0String),
})
class class SessionsSessions extends import ShardMapShardMap.Tag<class SessionsSessions>()("app/Sessions", {
key: Schema.Stringkey: const SessionId: Schema.StringSessionId,
value: Schema.Struct<{
readonly id: Schema.String;
readonly userId: Schema.String;
readonly seat: Schema.optionalKey<Schema.String>;
}>
const Session: Schema.Struct<{
readonly id: Schema.String;
readonly userId: Schema.String;
readonly seat: Schema.optionalKey<Schema.String>;
}>
keyOf: (s: any) => anykeyOf: (s: anys) => s: anys.id,
}).pipe(
import HyperlinkHyperlink.distributed([class DropletEastDropletEast, class DropletWestDropletWest, class DropletCentralDropletCentral]),
) {}Bring a droplet online
One materialization — local shard + RPC handlers + peer clients. Swap DropletEast for West / Central on the other machines; the callers program does not change.
const const east: anyeast = import ShardMapShardMap.serve(class SessionsSessions).pipe(
import LayerLayer.const provide: <any>(that: any) => <A, E, R>(self: Layer.Layer<A, E, R>) => Layer.Layer<A, any, any> (+3 overloads)Feeds the output services of the dependency layer into the requirements of
this layer, returning a layer that only provides the services from this layer.
When to use
Use when you need to hide an implementation dependency layer from callers.
Details
In serviceLayer.pipe(Layer.provide(dependencyLayer)), the dependency layer is
built first and is used to satisfy the requirements of serviceLayer.
Example (Providing layer dependencies)
import { Context, Effect, Layer } from "effect"
class Database extends Context.Service<Database, {
readonly query: (sql: string) => Effect.Effect<string>
}>()("Database") {}
class UserService extends Context.Service<UserService, {
readonly getUser: (id: string) => Effect.Effect<{
id: string
name: string
}>
}>()("UserService") {}
class Logger extends Context.Service<Logger, {
readonly log: (msg: string) => Effect.Effect<void>
}>()("Logger") {}
// Create dependency layers
const databaseLayer = Layer.succeed(Database, {
query: Effect.fn("Database.query")((sql: string) => Effect.succeed(`DB: ${sql}`))
})
const loggerLayer = Layer.succeed(Logger, {
log: Effect.fn("Logger.log")((msg: string) => Effect.sync(() => console.log(`[LOG] ${msg}`)))
})
// UserService depends on Database and Logger
const userServiceLayer = Layer.effect(UserService, Effect.gen(function*() {
const database = yield* Database
const logger = yield* Logger
return {
getUser: Effect.fn("UserService.getUser")(function*(id: string) {
yield* logger.log(`Looking up user ${id}`)
const result = yield* database.query(
`SELECT * FROM users WHERE id = ${id}`
)
return { id, name: result }
})
}
}))
// Provide dependencies to UserService layer
const userServiceWithDependencies = userServiceLayer.pipe(
Layer.provide(Layer.mergeAll(databaseLayer, loggerLayer))
)
// Now UserService layer has no dependencies
const program = Effect.gen(function*() {
const userService = yield* UserService
return yield* userService.getUser("123")
}).pipe(
Effect.provide(userServiceWithDependencies)
)
@seeprovideMerge for retaining the dependency services@categoryproviding services@since2.0.0provide(import HyperlinkHyperlink.peersLayer(class SessionsSessions, class DropletEastDropletEast)),
const nodeServer: (port: number) => <A, E, R>(resource: Layer.Layer<A, E, R>) => anynodeServer(3001),
)
// east: Layer — this droplet owns its shard and forwards the rest through peersPut and get from anywhere
From Easts HTTP edge or Wests poller — same handle. Ownership + the hop stay inside the Hyperlink.
const const sessions: anyconst sessions: {
get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
put: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
putLocal: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
}
class Sessionsclass Sessions {
key: Identifier;
Service: {
get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
put: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
putLocal: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
};
groupId: string;
description: string | undefined;
of: (this: void, self: { readonly get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>; readonly put: (payload: { id: string; userId: string; seat?:…;
context: (self: { readonly get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>; readonly put: (payload: { id: string; userId: string; seat?: string | un…;
use: (f: (service: { readonly get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>; readonly put: (payload: { id: string; userId: string; seat?: stri…;
useSync: (f: (service: { readonly get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>; readonly put: (payload: { id: string; userId: string; seat?: stri…;
Identifier: Identifier;
stack: string | undefined;
pipe: { <A>(this: A): A; <A, B = never>(this: A, ab: (_: A) => B): B; <A, B = never, C = never>(this: A, ab: (_: A) => B, bc: (_: B) => C): C; <A, B = never, C = never, D = never>(this: A, ab: (_: A) => B, bc: (_: B) => C, cd: (_: C) => D): D; <…;
toString: () => string;
toJSON: () => unknown;
}
const wrote: anywrote = yield* const sessions: anyconst sessions: {
get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
put: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
putLocal: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
}
id: stringid: "fan-90210",
userId: stringuserId: "u_nik",
seat: stringseat: "124-A",
})
// wrote: boolean — true when the owning node accepted the write
const const session: anysession = yield* const sessions: anyconst sessions: {
get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
put: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
putLocal: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
}
const dropped: anydropped = yield* const sessions: anyconst sessions: {
get: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
put: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<{ readonly id: string; readonly userId: string; readonly seat?: string | undefined }>, never, never>;
putLocal: (payload: { id: string; userId: string; seat?: string | undefined }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
}
Leaf ops (getLocal / putLocal / deleteLocal / sizeLocal) stay on this shard — that is what peers fold and what routed ops forward to.
Fleet sizes
Ops across the pack without inventing a second dashboard tag:
const const sessions: anyconst sessions: {
get: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>;
put: (payload: { id: string; userId: string }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>;
putLocal: (payload: { id: string; userId: string }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
}
class Sessionsclass Sessions {
key: Identifier;
Service: {
get: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>;
put: (payload: { id: string; userId: string }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>;
putLocal: (payload: { id: string; userId: string }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
};
groupId: string;
description: string | undefined;
of: (this: void, self: { readonly get: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>; readonly put: (payload: { id: string; userId: …;
context: (self: { readonly get: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>; readonly put: (payload: { id: string; userId: string }) =>…;
use: (f: (service: { readonly get: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>; readonly put: (payload: { id: string; userId: strin…;
useSync: (f: (service: { readonly get: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>; readonly put: (payload: { id: string; userId: strin…;
Identifier: Identifier;
stack: string | undefined;
pipe: { <A>(this: A): A; <A, B = never>(this: A, ab: (_: A) => B): B; <A, B = never, C = never>(this: A, ab: (_: A) => B, bc: (_: B) => C): C; <A, B = never, C = never, D = never>(this: A, ab: (_: A) => B, bc: (_: B) => C, cd: (_: C) => D): D; <…;
toString: () => string;
toJSON: () => unknown;
}
const shards: anyshards = yield* const sessions: anyconst sessions: {
get: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>;
put: (payload: { id: string; userId: string }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>;
putLocal: (payload: { id: string; userId: string }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
}
const fleet: anyfleet = yield* const sessions: anyconst sessions: {
get: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>;
put: (payload: { id: string; userId: string }) => Effect.Effect<boolean, never, never>;
delete: (payload: string) => Effect.Effect<boolean, never, never>;
getLocal: (payload: string) => Effect.Effect<Option<Schema.Struct.ReadonlySide<{ readonly id: Schema.String; readonly userId: Schema.String }, 'Type'>>, never, never>;
putLocal: (payload: { id: string; userId: string }) => Effect.Effect<void, never, never>;
deleteLocal: (payload: string) => Effect.Effect<boolean, never, never>;
sizeLocal: Effect.Effect<number, never, never>;
sizeByNode: Effect.Effect<{ readonly [x: string]: number }, never, never>;
size: Effect.Effect<number, never, never>;
}
Persist the shard
Local keys are SQLite SSOT — one row per live (scope, key), not an event log. ShardMap.layer / serve open :memory: by default (always on); pass { filename } for a durable file. Boot loads rows once; mutations UPSERT / DELETE.
const const live: anylive = import ShardMapShardMap.serve(class SessionsSessions, {
filename: stringfilename: ".hyperlink-ts/sessions.sqlite",
}).pipe(import LayerLayer.const provide: <any>(that: any) => <A, E, R>(self: Layer.Layer<A, E, R>) => Layer.Layer<A, any, any> (+3 overloads)Feeds the output services of the dependency layer into the requirements of
this layer, returning a layer that only provides the services from this layer.
When to use
Use when you need to hide an implementation dependency layer from callers.
Details
In serviceLayer.pipe(Layer.provide(dependencyLayer)), the dependency layer is
built first and is used to satisfy the requirements of serviceLayer.
Example (Providing layer dependencies)
import { Context, Effect, Layer } from "effect"
class Database extends Context.Service<Database, {
readonly query: (sql: string) => Effect.Effect<string>
}>()("Database") {}
class UserService extends Context.Service<UserService, {
readonly getUser: (id: string) => Effect.Effect<{
id: string
name: string
}>
}>()("UserService") {}
class Logger extends Context.Service<Logger, {
readonly log: (msg: string) => Effect.Effect<void>
}>()("Logger") {}
// Create dependency layers
const databaseLayer = Layer.succeed(Database, {
query: Effect.fn("Database.query")((sql: string) => Effect.succeed(`DB: ${sql}`))
})
const loggerLayer = Layer.succeed(Logger, {
log: Effect.fn("Logger.log")((msg: string) => Effect.sync(() => console.log(`[LOG] ${msg}`)))
})
// UserService depends on Database and Logger
const userServiceLayer = Layer.effect(UserService, Effect.gen(function*() {
const database = yield* Database
const logger = yield* Logger
return {
getUser: Effect.fn("UserService.getUser")(function*(id: string) {
yield* logger.log(`Looking up user ${id}`)
const result = yield* database.query(
`SELECT * FROM users WHERE id = ${id}`
)
return { id, name: result }
})
}
}))
// Provide dependencies to UserService layer
const userServiceWithDependencies = userServiceLayer.pipe(
Layer.provide(Layer.mergeAll(databaseLayer, loggerLayer))
)
// Now UserService layer has no dependencies
const program = Effect.gen(function*() {
const userService = yield* UserService
return yield* userService.getUser("123")
}).pipe(
Effect.provide(userServiceWithDependencies)
)
@seeprovideMerge for retaining the dependency services@categoryproviding services@since2.0.0provide(import HyperlinkHyperlink.peersLayer(class SessionsSessions, class DropletEastDropletEast)))
// omit filename → in-memory SQLite (default)Partition ethic (v1)
ShardMap.consistentHash sorts node keys and picks with Hash.string modulo — stable for a fixed fleet. Membership change remaps keys; treat that as intentional. Unreachable owner → get is none, put returns false — miss beats silent wrong answer.
Runnable form: pnpm run example:shardmap-sessions. See also Fleets & Peers.