Skip to the page
Get started – free

FivetranComing soon

Load the live web data SpecStream finds into your Fivetran destination: every record, and one row per value with the page it was read from and when. It runs on Fivetran's Connector SDK. Whatever a SpecStream schedule refreshes shows up in the sync after it.

SpecStream for FivetranData & warehouses
SpecStream for Fivetran is coming soon

The connector is in review in Fivetran's community connectors, so fivetran init --template specstream can't find it yet. It runs in your own Fivetran account today: copy connector.py and configuration.json from the specstream folder (opens in a new tab) into an empty folder and start at step 3.

What you need

  • A SpecStream account. Signing up is free and takes no card.
  • A Fivetran account with a destination, and the Connector SDK set up on your machine.
  • A SpecStream API key. Step 1 makes it with read permission.

  1. 1. Make an API key

    Open Agents & API in the SpecStream workspace and go to API keys. Choose Create API key, set read permission and a spending limit of $0, because the connector only reads.

    Copy the key straight away. It appears once.

  2. 2. Start a Connector SDK project

    Install the Connector SDK as Fivetran's setup guide (opens in a new tab) describes. You don't need a requirements.txt: the connector uses only requests, which the Connector SDK runtime already includes.

    This command sets up a project folder with connector.py and configuration.json:

    Terminal
    fivetran init --template specstream

  3. 3. Put your key in the configuration

    api_key is the only setting, and the connector sends it as a Bearer token. A sync stops at once when the key is missing or still shows the placeholder. Keep configuration.json out of version control.

    Open configuration.json and replace the placeholder with your key:

    configuration.json
    {
      "api_key": "<YOUR_SPECSTREAM_API_KEY>"
    }

  4. 4. Run it on your machine

    Run fivetran debug in the project folder. The connector runs on your machine and reads your lists, so you see the sync work before anything reaches your warehouse.

  5. 5. Deploy it to your destination

    Deploy the project to your Fivetran destination with fivetran deploy, passing your configuration.json. The setup guide covers the options. Then set how often the connection syncs in Fivetran.

    The first sync reads every list the key can open. Later syncs read only lists created since the previous one.

Tables

The connector writes these tables to your destination:

TablePrimary keyColumns
listidname, project_id, imported, created_at, row_count
recordlist_id, record_identity_id, values, sources
record_valuelist_id, record_id, column_idlabel, value, raw_value, source_url, observed_at

Values and sources

In record, entity_id is the same wherever a company, place, person, or posting shows up. values is a JSON object of column label to value, and sources is a JSON array of the pages the record's values came from.

Each value SpecStream found also becomes a record_value row with its label, its displayed value, its raw value as JSON, the page it was read from, and when it was read. If SpecStream couldn't read a value, the cell is null.

Records of a list you imported carry the columns you provided and no source page. A combined list gets the value the list shows.

Incremental syncs

When a SpecStream schedule finds new or changed records, they arrive as a new list. The connector keeps lists_created_through in its state, the creation time of the newest list it synced, and reads only newer lists.

Lists load oldest first with a checkpoint after each one, so an interrupted sync resumes where it stopped. A list is written to list after its records, which means a list in the destination always has all of its records. A full re-sync in Fivetran clears the state and reads every list again.

Errors

If SpecStream refuses the key, the sync fails at once with a message to create a new key under Agents & API.

Rate limits, server errors, connection errors, and timeouts get up to five attempts with exponential backoff, and the connector waits as long as Retry-After asks.

Cost

The connector is read-only and free to run. It never starts a request or spends from your SpecStream balance. Your balance pays for the requests that find the data, per result that matches. Balance, quotes, and billing explains how prices work.

Disconnect

Pause or delete the connection in Fivetran. To cut access at the source, revoke the key under Agents & API; the next sync fails. Rows already in the destination stay there. Everything else SpecStream connects to is in Connect your tools. If something here doesn't work, email [email protected].

More

SpecStream for Fivetran covers what it does in Fivetran on one page. Every other tool has its guide in Connect your tools. You see the most a request can cost before it starts, and only the matches come out of your balance. Billing has the details. For help, email [email protected].