Pipeline Operations

Available Sources

Source	Description	Schedule
everflow	Affiliate conversions/revenue	Hourly 7am-11pm ET
redtrack	Ad spend tracking	Hourly 7am-11pm ET
s3_exports	CSV files from S3 (Sticky.io)	Daily 6am ET
posthog	PostHog analytics	Stubbed
mautic	Mautic contacts/campaigns	Stubbed
google_sheets	Google Sheets data	Stubbed

API Reference: See docs/API_REFERENCE.md for live docs, auth, and request/response examples.

Run Pipeline Directly (Local Testing)

# Activate environment
source .venv/bin/activate

# Run a specific source
python scripts/run_pipeline.py everflow
python scripts/run_pipeline.py redtrack
python scripts/run_pipeline.py s3_exports

# Dry run (see what would happen)
python scripts/run_pipeline.py everflow --dry-run

Trigger via Temporal (Production)

# Trigger and return immediately
python scripts/trigger_workflow.py everflow

# Trigger and wait for completion
python scripts/trigger_workflow.py everflow -w

# Trigger with notification on success
python scripts/trigger_workflow.py everflow -w --notify

Check Pipeline Status

# View worker logs (local)
make logs-worker

# View Fly.io logs (production)
fly logs

# Temporal Cloud UI
# https://cloud.temporal.io/namespaces/signalroom-713.nzg5u/workflows

Active Schedules

View in Temporal Cloud UI or:

python scripts/setup_schedules.py --list

Current schedules:

• hourly-sync-everflow-redtrack - Hourly 7am-11pm ET
• daily-sync-s3 - Daily 6am ET

Troubleshooting

"Unknown source" error: Source not registered in src/signalroom/pipelines/runner.py

Credentials error: Check .env has correct API keys

Database connection failed:

• Verify Supabase pooler settings (port 6543)
• User must be postgres.{project_ref}

Activity timeout: Pipeline took >30 min. Filter resources or increase timeout.