CepatEdge

Appearance

Logs architecture ​

How CepatEdge stores and queries logs in Durable Objects: monthly registries, unlimited shards, per-key entries, and cursor-based reads.

Related: Cache architecture (multi-shard reader) · Security logs (/me/security/logs).

Goals ​

  • Store logs per-key (timestamp-indexed) with pagination (500–1K per read)
  • Shard by month
  • Each month has a registry DO (metadata + active shard pointer only)
  • Create data shards dynamically (random IDs) so a month can grow without a hard shard cap

Monthly registry + shards ​

Registry (per month) ​

Name: <mainInstanceName>:<YYYY-MM>
Examples: usage-registry:2026-02, or the internal-logs registry naming used in code.

Holds only:

  • activeShardId
  • shards[] metadata (shardId, createdAt, lastWriteAt, minTimestampMs, maxTimestampMs, approxCount, optional approxBytes, status: active | sealed)

Data shard ​

Name: <mainInstanceName>:<YYYY-MM>:shard:<random-16-chars>

Stores log entries and optional indexes.

Keys inside a shard ​

KeyPurpose
usage:log:<timestampMs>:<id> (or internal:log:<id> for internal cache)Primary entry
internal:index:<shardId>Global newest-first { id, timestamp }[]
internal:userindex:<userId>Per-user newest-first index for fast user-scoped lists
Optional usage:user:<userId>:<ts>:<id> → primary keyDelete-by-user without full scans

Keep payloads small (no large request bodies in log values).

Write path ​

  1. Resolve month (YYYY-MM) from timestamp
  2. Call registry DO → get (or create) active shard
  3. If active shard exceeds threshold (approxCount, approxBytes, or time window) → create shard, seal old, switch pointer
  4. Write entry (+ update global / user indexes)
  5. Update registry shard metadata

Read path ​

Inputs: fromTimestampMs, toTimestampMs, limit, optional cursor.

  1. Resolve month registries overlapping the range
  2. Registry returns shards whose [min,max] overlaps the range
  3. Multi-shard reader (see cache.md) reads indexes, fetches entries, filters, merges newest-first
  4. Return { entries, nextCursor?, hasMore } — stop fan-out once limit is satisfied

Delete path ​

  • By time: registry → overlapping shards → batch list + delete
  • By user: prefer user-index prefixes; delete index keys and primary keys together

Helper reference: apps/worker/src/lib/services/cache/usage/delete.ts (batch deletion patterns).

Internal logger (current behavior) ​

Per-user index ​

When an entry has userId, the internal cache also updates internal:userindex:{userId} on that shard. getLogs({ userId }) uses the user index instead of scanning the global index.

Cursor pagination ​

  • API style: limit + cursor (no offset pages)
  • Cursor format: timestamp:id
  • Result: { entries, nextCursor?, hasMore }

Early exit ​

Multi-shard reader stops requesting more shards once enough entries are collected.

Types / code ​

PieceLocation
Entry / filtersapps/worker/src/types/cache/internal.ts
Internal cacheapps/worker/src/lib/services/cache/internal/
Multi-shard readerapps/worker/src/lib/services/cache/multi-shard-reader/
User logs serviceapps/worker/src/lib/services/me/logs/
Logger facadeapps/worker/src/lib/services/internal/

Methods that accept userId / ipAddress where relevant: logAuthEvent, logSecurityEvent, logSystemEvent, logUserEvent, logMaintenance, plus warning / error / suspicious / info.

Privacy ​

Treat IP / location as personal data; prefer coarse location; retain detailed logs for a short window (e.g. 30–90 days) then delete or anonymize. Operators own retention policy for their instance — see ownership.