Continuous Releases from main
Add streaming pipeline: ProtocolStream, TransactionalStream, CdcStream (#17)
* Add protocol stream with stateless reorg detection
Adds streamProtocol method to ArrowFlight service that wraps streamQuery
with protocol-level message handling:
- Data: New records with block ranges
- Reorg: Chain reorganization detected with invalidation ranges
- Watermark: Confirmation that block ranges are complete
Includes validation logic ported from Rust implementation:
- validatePrevHash: Genesis vs non-genesis block rules
- validateNetworks: Network consistency across batches
- validateConsecutiveness: Hash chain and gap detection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add TransactionalStream with crash recovery and reorg handling
Implements a full transactional layer on top of the protocol stream,
providing exactly-once semantics for data processing:
- Transaction IDs: Monotonically increasing IDs for each event
- StateStore: Pluggable persistence via Context.Tag (InMemoryStateStore included)
- Watermark Buffer: Tracks watermarks for reorg recovery point calculation
- CommitHandle: Explicit commit control with idempotent semantics
- Rewind Detection: Detects and invalidates uncommitted transactions on restart
- Retention Window: Prunes old watermarks outside configurable block window
Also adds protocol-stream reorg tests ported from the Rust implementation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Extract ProtocolStream into separate service
Refactors the protocol stream functionality from ArrowFlight into its own
ProtocolStream Context.Tag service:
- Add protocol-stream/service.ts with ProtocolStream service definition
- Remove streamProtocol, detectReorgs, and related code from arrow-flight.ts
- Update TransactionalStream to depend on ProtocolStream instead of ArrowFlight
- Update exports in protocol-stream/index.ts
This improves separation of concerns and allows the protocol stream logic
to be used independently or replaced with alternative implementations.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Move barrel exports to root-level module files
Replace subdirectory index.ts barrel files with root-level .ts files
that serve as the public API for each module:
- protocol-stream/index.ts → protocol-stream.ts
- transactional-stream/index.ts → transactional-stream.ts
The package.json exports field (./* → ./src/*.ts) already supports this
pattern, so imports like "@edgeandnode/amp/protocol-stream" resolve to
the new root-level files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Refactor transactional-stream tests to @effect/vitest
Replace vitest imports with @effect/vitest and convert all tests to
it.effect with per-test Effect.provide for isolation. Remove runWithStore,
runWithState, and runWithActor helpers. Use Effect.exit instead of
Effect.runPromiseExit for failure tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix import ordering in refactored test files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add rest of stream code
* Remove unnecessary thunks from service interfaces
Effect is already lazy, so wrapping service methods in `() =>` thunks
is redundant. Convert all zero-arg thunk methods to plain Effect values
across StateStore, StateActor, CommitHandle, and AdminApi interfaces.
Also remove unused imports/variables in reorg.test.ts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Refactor models into core/domain module and split arrow-flight into subdirectory
Move all domain models from src/models.ts to src/core/domain.ts with a
src/core.ts barrel, and split the monolithic src/arrow-flight.ts into
errors.ts, types.ts, transport.ts, and service.ts submodules — matching
the existing protocol-stream/transactional-stream directory pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add rulesync for managing coding agent settigs
* lint
* remove manifest builder
* Add CdcStream module with BatchStore and at-least-once delivery
Port Rust CDC stream to TypeScript/Effect. Wraps TransactionalStream with
a BatchStore to persist batch content, enabling Delete events with original
data during reorgs and rewinds.
- CdcEvent union: Insert (data forwarding) and Delete (lazy batch iterator)
- BatchStore service: pluggable persistence (append, seek, load, prune)
- InMemoryBatchStore: Ref-backed reference implementation
- CdcStream service: Data→Insert, Undo→Delete, Watermark auto-commit+prune
- 17 tests covering batch store conformance and stream integration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* lint
* fix type errors
* refactor token interceptor layer and export domain models
* fix type errors again
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>17 hours ago67234e1
Add a CLI package (#16)
* add cli and auth subcommands
* kebab case for module names
* fix oxlint config
* resolve json module
* fix auth token propagation in commands using arrow flight
* remove unused imports
* upgrade tsgo
* fix file name casinglast week6a6d697