Streaming

Getting values out of a thread incrementally is a first-class concern: a script can push intermediate results to a consumer in real time, not just return a single terminal value. This page covers yield, the ->> streaming operator, pipe staging, and how emit, yield, and send differ.

Streaming output

A thread produces two kinds of output:

• emit — declares the final result. It is terminal (think return) and must satisfy the output contract.
• yield — pushes an intermediate value. It is repeatable and does not end the thread. Consumers receive yielded values in real time over WatchThread.

yield

yield "Label" followed by named fields pushes one streaming event. The shape of those events is declared in the stream block (see File Structure).

stream
  transcript : text
  meta       : object

-- ... later, in the body ...

yield "Transcript ready"
  data = transcript

Use yield for progress events and incremental results; use emit once at the end for the summary. A typical script does both:

pipe file from glob_result.matches
  do "Simulate embedding" with echo
    message = file
  -> embedding

  yield "Embedding Batch"
    embedding   = embedding.message
    batch_index = file
-> collected

emit "Complete"
  status        = "done"
  total_batches = count(collected)

The ->> streaming operator

The binding arrow -> collects a step’s output. The streaming arrow ->> forwards it as a stream into the next stage instead of buffering. It is supported on do (for streaming tools) and on pipe.

-- A tool that emits pages as a stream:
do "Extract PDF" with extract_pdf
  url    = resolved_url
  stream = false
->> pdf_pages

-- Consume the stream in the next stage:
pipe page from pdf_pages
  do "Collect Page" with echo
    message = page.text
-> extracted_pages

pipe staging

Chaining pipe stages with ->> and -> builds a streaming pipeline: each stage processes items as they arrive, and only the final stage collects. This keeps memory flat for large inputs. (Adapted from examples/stream_pipeline.scriptum.)

do "Find Rust files" with file_glob
  pattern = "crates/scriptum-compiler/src/*.rs"
-> glob_result

-- Stage 1: array source → stream forward
pipe file from glob_result.matches
  do "Tag file" with echo
    message = file
->> tagged

-- Stage 2: stream source → collect
pipe item from tagged
  do "Echo item" with echo
    message = item.message
-> collected

emit "Done"
  file_count      = glob_result.count
  processed_count = count(collected)

emit, yield, and send at a glance:

(send "Label" via "channel" dispatches to Slack or email using credentials from the env block — see File Structure.)

Next: recovering when steps fail in Error Handling.