SQLTide SQLTide by SQLStack

Streaming SQL pipelines for ClickHouse

Change live pipelines without downtime.

Write each step of your pipeline as SQL. When you change a step, SQLTide can build the new version alongside the running one and fill it with the events already stored in ClickHouse. Check the results, then point your queries at the new version.

Try the demo Quickstart View on GitHub
pip install sqltide
ExampleOrders pipeline

Change a filter, compare the results, then switch your query to the new version.

The change: only report orders of $1 or moreamount_cents >= 0 → >= 100
Your query doesn’t change SELECT * FROM tbl__order_report Reads the current version
Current version Serving queries
New version Getting new events
New events reach both versions; stored history fills the new one

Compare the two versions. Your queries still read the current one, and both keep getting new events.

Switching moves one view at a time, not the whole pipeline. How switching works

How a change goes live

By default sqt build drops the selected tables and refills them in place, while queries keep reading them. In virtual mode it builds next to the current tables instead, so you can check a change before your queries move to it. The steps below follow virtual mode.

  1. 01 · Write

    Write the pipeline in SQL

    Each model is a SQL file with typed columns. sqt compile checks them on your machine, without connecting to ClickHouse.

    Compile your models →
  2. 02 · Plan

    See what will change

    sqt plan lists the tables a change rebuilds and the history it replays. It doesn’t change anything in ClickHouse.

    Read a plan →
  3. 03 · Build and check

    Check the new version

    In virtual mode, sqt build creates the new tables next to the current ones and fills them. Your queries keep reading the current version while you compare.

    Build and check →
  4. 04 · Promote

    Switch your queries over

    Promoting points the views your queries read at the version you checked. A build never does this on its own.

    Promote a version →

What promotion switches, and what it doesn’t →

Replay, tests and a local UI

Replay stored history

Rebuild a table from the source history SQLTide keeps in ClickHouse. Replay can deliver a row twice, so your SQL and table engine need to handle duplicates.

How replay works →

Test models with example data

Write SQL tests with fixed inputs and run them before you deploy.

sqt test
SQL testing →

See what’s running

A local UI shows your models, sources, runs and deployments.

sqt dev
Open the UI demo →

Try it

Look around the UI in your browser, or run the playground on your machine with Docker.

Try the demoQuickstart
pip install sqltide sqt playground

sqt playground creates a small orders project and starts ClickHouse for it. How the playground works →