> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blockdb.chaintable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Write architecture

> BlockDB's write architecture: batch writes by default, asynchronous at the core, with logical models guaranteeing consistency.

# Write architecture

BlockDB is designed for blockchain compute: upstream data is always produced at **block granularity**, so it naturally arrives batch by batch. BlockDB therefore treats **batch writes as the default**, not traditional single-row writes. The only difference is the batch size — small, near real-time batches go through **real-time writes**, while large backfill batches go through **batch writes**.

## All writes are asynchronous

**Every BlockDB write is asynchronous.** The SDK does not expose a `sync` parameter; all write interfaces return `WriteResult { affectedRows, jobID }`, which does not mean the data has already landed.

As a result, the plain write interfaces **do not satisfy classic read-after-write consistency** — reading immediately after a write returns does not guarantee you will see the data you just wrote.

But BlockDB's logical models close this gap:

* **Block tables**: after a successful write, the write service publishes a `BlockSubscribeEvent` through Kafka. Once a downstream consumer receives the event through `Subscribe` and then reads that block's data, BlockDB guarantees the data has already landed and is visible. **In a subscription-driven compute pipeline, write → event → read is strictly read-after-write consistent.**
* **Time tables**: after a write, `GetValue` queries the `_archive` table and likewise returns historical bucket data that has already landed.

So although the underlying writes are asynchronous, the read/write/subscribe interfaces of Block tables and Time tables strictly satisfy the consistency requirements of blockchain compute scenarios.

## SDK write interfaces

### Real-time writes

| Interface | Description |
| - | - |
| `table.upsert_rows(rows)` | Real-time write to a regular table |
| `block_table.write_block(block, rows)` | Real-time write to a Block table at block granularity |
| `time_table.upsert_time_rows(rows, time_at)` | Real-time write to a Time table by time bucket |

### Batch writes (S3 staging, up to 10 million rows)

Batch writes go through a single gRPC service, `BatchWriteService`. In the `InitWriteJob` request you choose the target table type by selecting the `strategy` field:

| Strategy | Table type | Description |
| - | - | - |
| `normal_upsert` | Regular table | Large batch write to a regular table |
| `block_overwrite` | Block table | Batch write to a Block table across a height/bundle range |
| `time_upsert` | Time table | Batch write to a Time table (backfill) |

```protobuf theme={null}
message InitWriteJobRequest {
  string table_id = 1;
  oneof strategy {
    BlockTableOverwriteStrategy block_overwrite = 2;
    NormalTableUpsertStrategy   normal_upsert   = 3;
    TimeTableUpsertStrategy     time_upsert     = 4;
  }
}
```

For example, starting a Time-table batch write job:

```bash theme={null}
grpcurl -plaintext -d '{
  "tableId": "token_time_demo",
  "timeUpsert": {}
}' localhost:19002 blockdb.v1.BatchWriteService/InitWriteJob
```

All write methods return a uniform `WriteResult { affectedRows, jobID }`.

## Real-time write architecture

Real-time writes are handled by a simple stateless writer using an **in-memory async queue plus multi-threaded reactive writes**:

1. **Enqueue**: the write request enters the in-memory async queue (`asyncqueue`) and returns a `jobID` immediately. Users are never blocked by backend ingestion speed.
2. **Consume**: multiple worker threads pull tasks from the queue concurrently and write to the underlying physical database in batches.
3. **Traffic shaping**: the queue has a capacity limit; when the backend cannot keep up, requests are rejected (returning `QUEUE_OVERLOAD`). This converts unbounded real-time traffic into bounded write traffic and protects the underlying database.

Design goals: real-time traffic never blocks user submissions, and idle threads respond to write requests almost immediately, achieving near real-time latency.

## Batch write architecture

When writes reach millions of rows (up to 10 million), the in-memory queue used by real-time writes is not viable. Batch writes use **object storage staging plus asynchronous workers**:

### Job lifecycle

1. **`InitWriteJob`**: the client requests a write job. BlockDB creates a job record in the table's `_write` table and returns an S3 presigned upload URL plus a `jobID`.
2. **Upload the data file**: the client uploads the data to the returned S3 URL.
3. **`CommitWriteJob`**: the client commits the job. BlockDB marks it pending and background workers start consuming it.
4. **Worker execution**: the worker pulls the job file from S3, then sorts, dedups, and ingests the rows in batches.

### Architecture characteristics

* **Unbounded submission throughput**: jobs are staged through S3 and do not consume the real-time memory path, so you can submit continuously at high throughput.
* **Fast ingestion**: workers operate in batches, reaching hundreds of thousands to over a million rows per second (depending on the table type and the size of the underlying database).
* **Cancellable**: queued write jobs can be cancelled with the `CancelWriteJob` API, releasing the S3 resources and the corresponding job record in the `_write` table.
* **Observable job state**: jobs live in the `_write` table with states such as initialized, committed, processing, completed, and failed, which helps with operational troubleshooting.

## Real-time vs batch writes

| Dimension | Real-time writes | Batch writes |
| - | - | - |
| Data size | Small batches (for example one block's compute output) | Large batches (for example backfilling millions of rows) |
| Staging | In-memory async queue | S3 object storage |
| Submission throughput | Near real-time, bounded by queue capacity | Unbounded |
| Ingestion speed | Near real-time | 100K – 1M+ rows/s |
| Job management | No persisted job record | Jobs live in `_write`, queryable and cancellable |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.