Write architecture

BlockDB is designed for blockchain compute: upstream data is always produced at block granularity, so it naturally arrives batch by batch. BlockDB therefore treats batch writes as the default, not traditional single-row writes. The only difference is the batch size — small, near real-time batches go through real-time writes, while large backfill batches go through batch writes.

All writes are asynchronous

Every BlockDB write is asynchronous. The SDK does not expose a sync parameter; all write interfaces return WriteResult { affectedRows, jobID }, which does not mean the data has already landed. As a result, the plain write interfaces do not satisfy classic read-after-write consistency — reading immediately after a write returns does not guarantee you will see the data you just wrote. But BlockDB’s logical models close this gap:

Block tables: after a successful write, the write service publishes a BlockSubscribeEvent through Kafka. Once a downstream consumer receives the event through Subscribe and then reads that block’s data, BlockDB guarantees the data has already landed and is visible. In a subscription-driven compute pipeline, write → event → read is strictly read-after-write consistent.
Time tables: after a write, GetValue queries the _archive table and likewise returns historical bucket data that has already landed.

So although the underlying writes are asynchronous, the read/write/subscribe interfaces of Block tables and Time tables strictly satisfy the consistency requirements of blockchain compute scenarios.

SDK write interfaces

Real-time writes

Interface	Description
`table.upsert_rows(rows)`	Real-time write to a regular table
`block_table.write_block(block, rows)`	Real-time write to a Block table at block granularity
`time_table.upsert_time_rows(rows, time_at)`	Real-time write to a Time table by time bucket

Batch writes (S3 staging, up to 10 million rows)

Batch writes go through a single gRPC service, BatchWriteService. In the InitWriteJob request you choose the target table type by selecting the strategy field:

Strategy	Table type	Description
`normal_upsert`	Regular table	Large batch write to a regular table
`block_overwrite`	Block table	Batch write to a Block table across a height/bundle range
`time_upsert`	Time table	Batch write to a Time table (backfill)

message InitWriteJobRequest {
  string table_id = 1;
  oneof strategy {
    BlockTableOverwriteStrategy block_overwrite = 2;
    NormalTableUpsertStrategy   normal_upsert   = 3;
    TimeTableUpsertStrategy     time_upsert     = 4;
  }
}

For example, starting a Time-table batch write job:

grpcurl -plaintext -d '{
  "tableId": "token_time_demo",
  "timeUpsert": {}
}' localhost:19002 blockdb.v1.BatchWriteService/InitWriteJob

All write methods return a uniform WriteResult { affectedRows, jobID }.

Real-time write architecture

Real-time writes are handled by a simple stateless writer using an in-memory async queue plus multi-threaded reactive writes:

Enqueue: the write request enters the in-memory async queue (asyncqueue) and returns a jobID immediately. Users are never blocked by backend ingestion speed.
Consume: multiple worker threads pull tasks from the queue concurrently and write to the underlying physical database in batches.
Traffic shaping: the queue has a capacity limit; when the backend cannot keep up, requests are rejected (returning QUEUE_OVERLOAD). This converts unbounded real-time traffic into bounded write traffic and protects the underlying database.

Design goals: real-time traffic never blocks user submissions, and idle threads respond to write requests almost immediately, achieving near real-time latency.

Batch write architecture

When writes reach millions of rows (up to 10 million), the in-memory queue used by real-time writes is not viable. Batch writes use object storage staging plus asynchronous workers:

Job lifecycle

InitWriteJob: the client requests a write job. BlockDB creates a job record in the table’s _write table and returns an S3 presigned upload URL plus a jobID.
Upload the data file: the client uploads the data to the returned S3 URL.
CommitWriteJob: the client commits the job. BlockDB marks it pending and background workers start consuming it.
Worker execution: the worker pulls the job file from S3, then sorts, dedups, and ingests the rows in batches.

Architecture characteristics

Unbounded submission throughput: jobs are staged through S3 and do not consume the real-time memory path, so you can submit continuously at high throughput.
Fast ingestion: workers operate in batches, reaching hundreds of thousands to over a million rows per second (depending on the table type and the size of the underlying database).
Cancellable: queued write jobs can be cancelled with the CancelWriteJob API, releasing the S3 resources and the corresponding job record in the _write table.
Observable job state: jobs live in the _write table with states such as initialized, committed, processing, completed, and failed, which helps with operational troubleshooting.

Real-time vs batch writes

Dimension	Real-time writes	Batch writes
Data size	Small batches (for example one block’s compute output)	Large batches (for example backfilling millions of rows)
Staging	In-memory async queue	S3 object storage
Submission throughput	Near real-time, bounded by queue capacity	Unbounded
Ingestion speed	Near real-time	100K – 1M+ rows/s
Job management	No persisted job record	Jobs live in `_write`, queryable and cancellable

Getting started

Core concepts

Architecture

Guides

Reference

Write architecture

Write architecture

All writes are asynchronous

SDK write interfaces

Real-time writes

Batch writes (S3 staging, up to 10 million rows)

Real-time write architecture

Batch write architecture

Job lifecycle

Architecture characteristics

Real-time vs batch writes

​Write architecture

​All writes are asynchronous

​SDK write interfaces

​Real-time writes

​Batch writes (S3 staging, up to 10 million rows)

​Real-time write architecture

​Batch write architecture

​Job lifecycle

​Architecture characteristics

​Real-time vs batch writes

Write architecture

All writes are asynchronous

SDK write interfaces

Real-time writes

Batch writes (S3 staging, up to 10 million rows)

Real-time write architecture

Batch write architecture

Job lifecycle

Architecture characteristics

Real-time vs batch writes