Data Pipeline Projects | ETL & Analytics PM
Manage data engineering work with clear workflows for ETL pipelines and data quality. GitScrum coordinates data engineers, analysts, and stakeholders.
8 min read
Data pipelines require careful coordination between data engineers, analysts, and stakeholders. GitScrum helps teams manage data infrastructure projects with clear workflows.
Pipeline Development
Pipeline Task Structure
DATA PIPELINE TASK BREAKDOWN:
┌─────────────────────────────────────────────────────────────┐
│ │
│ PIPELINE EPIC: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ DATA-100: Customer Analytics Pipeline ││
│ │ ││
│ │ Goal: Daily customer metrics for analytics ││
│ │ ││
│ │ Source: Production DB (customers, orders) ││
│ │ Target: Analytics warehouse ││
│ │ Schedule: Daily at 2 AM ││
│ │ ││
│ │ Tasks: ││
│ │ ├── DATA-101: Extract customer data ││
│ │ ├── DATA-102: Extract order data ││
│ │ ├── DATA-103: Transform customer metrics ││
│ │ ├── DATA-104: Load to warehouse ││
│ │ ├── DATA-105: Data quality checks ││
│ │ ├── DATA-106: Alerting and monitoring ││
│ │ └── DATA-107: Documentation ││
│ └─────────────────────────────────────────────────────────┘│
│ │
│ INDIVIDUAL STAGE TASK: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ DATA-103: Transform customer metrics ││
│ │ ││
│ │ Input: Raw customer + order data ││
│ │ ││
│ │ Output metrics: ││
│ │ • total_orders per customer ││
│ │ • total_revenue per customer ││
│ │ • avg_order_value ││
│ │ • first_order_date ││
│ │ • last_order_date ││
│ │ • days_since_last_order ││
│ │ ││
│ │ Business rules: ││
│ │ • Only count completed orders ││
│ │ • Exclude test accounts ││
│ │ • Handle refunds correctly ││
│ │ ││
│ │ Testing: ││
│ │ ☐ Unit tests for each calculation ││
│ │ ☐ Edge case: no orders ││
│ │ ☐ Edge case: all refunded ││
│ └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘
Data Contracts
DATA CONTRACT DEFINITION:
┌─────────────────────────────────────────────────────────────┐
│ │
│ WHY DATA CONTRACTS: │
│ │
│ • Define expected schema │
│ • Catch breaking changes early │
│ • Enable parallel development │
│ • Document data flow │
│ │
│ ─────────────────────────────────────────────────────────── │
│ │
│ CONTRACT TASK: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ DATA-110: Define customer_metrics contract ││
│ │ ││
│ │ Schema: ││
│ │ ┌─────────────────────────────────────────────────────┐││
│ │ │ customer_metrics │││
│ │ ├─────────────────────────────────────────────────────┤││
│ │ │ customer_id: STRING (required) │││
│ │ │ total_orders: INTEGER (>= 0) │││
│ │ │ total_revenue: DECIMAL(10,2) │││
│ │ │ avg_order_value: DECIMAL(10,2) │││
│ │ │ first_order_date: DATE (nullable) │││
│ │ │ last_order_date: DATE (nullable) │││
│ │ │ days_since_last_order: INTEGER (nullable) │││
│ │ │ computed_at: TIMESTAMP (required) │││
│ │ └─────────────────────────────────────────────────────┘││
│ │ ││
│ │ Constraints: ││
│ │ • No nulls in customer_id ││
│ │ • total_orders >= 0 ││
│ │ • first_order_date <= last_order_date ││
│ │ ││
│ │ Consumers: ││
│ │ • BI dashboard ││
│ │ • ML churn model ││
│ │ • Marketing automation ││
│ └─────────────────────────────────────────────────────────┘│
│ │
│ BREAKING CHANGES: │
│ • Removing columns │
│ • Changing types │
│ • Changing semantics │
│ → Require consumer coordination │
└─────────────────────────────────────────────────────────────┘
Data Quality
Quality Checks
DATA QUALITY IMPLEMENTATION:
┌─────────────────────────────────────────────────────────────┐
│ │
│ DATA QUALITY TASK: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ DATA-105: Data quality checks ││
│ │ ││
│ │ COMPLETENESS: ││
│ │ ☐ No null customer_ids ││
│ │ ☐ All customers have computed_at ││
│ │ ☐ Row count within expected range ││
│ │ ││
│ │ FRESHNESS: ││
│ │ ☐ computed_at within last 24 hours ││
│ │ ☐ last_order_date not in future ││
│ │ ││
│ │ ACCURACY: ││
│ │ ☐ total_revenue = sum of order amounts ││
│ │ ☐ avg_order_value = revenue / orders ││
│ │ ☐ Spot check against source ││
│ │ ││
│ │ CONSISTENCY: ││
│ │ ☐ Customer exists in customer table ││
│ │ ☐ No duplicate customer_ids ││
│ │ ││
│ │ REASONABLENESS: ││
│ │ ☐ avg_order_value within expected range ││
│ │ ☐ No negative revenue ││
│ │ ☐ Metric changes within bounds vs yesterday ││
│ └─────────────────────────────────────────────────────────┘│
│ │
│ QUALITY GATE: │
│ │
│ Pipeline succeeds only if: │
│ • All completeness checks pass │
│ • All accuracy checks pass │
│ • Freshness is acceptable │
│ │
│ Warnings (don't block): │
│ • Reasonableness thresholds exceeded │
│ • → Alert, but load data │
└─────────────────────────────────────────────────────────────┘
Quality Issues
DATA QUALITY BUG HANDLING:
┌─────────────────────────────────────────────────────────────┐
│ │
│ DATA QUALITY INCIDENT: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ 🔴 DATA-BUG-42: Duplicate customer records ││
│ │ ││
│ │ Discovered: 2024-01-15 monitoring alert ││
│ │ Impact: BI dashboards showing inflated counts ││
│ │ ││
│ │ Root cause: ││
│ │ Source system has duplicate customer_ids due to ││
│ │ merge issue from acquisition. ││
│ │ ││
│ │ Immediate fix: ││
│ │ ☐ Add deduplication to transform ││
│ │ ☐ Rerun pipeline for affected dates ││
│ │ ☐ Notify consumers of corrected data ││
│ │ ││
│ │ Long-term fix: ││
│ │ ☐ Work with source team on deduplication ││
│ │ ☐ Add uniqueness check to quality gates ││
│ │ ││
│ │ Labels: data-quality, pipeline, high-priority ││
│ └─────────────────────────────────────────────────────────┘│
│ │
│ CLASSIFICATION: │
│ │
│ Critical: Data is wrong, decisions affected │
│ → Fix immediately, notify stakeholders │
│ │
│ High: Data incomplete, some queries fail │
│ → Fix within 1-2 days │
│ │
│ Medium: Data delayed, freshness impacted │
│ → Fix within sprint │
│ │
│ Low: Minor issues, workarounds exist │
│ → Add to backlog │
└─────────────────────────────────────────────────────────────┘
Stakeholder Management
Requirements Gathering
DATA PROJECT REQUIREMENTS:
┌─────────────────────────────────────────────────────────────┐
│ │
│ STAKEHOLDER INTAKE TASK: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ DATA-REQ-01: Customer metrics requirements ││
│ │ ││
│ │ Stakeholder: Marketing team ││
│ │ Use case: Customer segmentation ││
│ │ ││
│ │ Questions answered: ││
│ │ ││
│ │ WHAT DATA: ││
│ │ ☑ What metrics do you need? ││
│ │ → Order count, revenue, recency ││
│ │ ││
│ │ WHY: ││
│ │ ☑ What decisions will this inform? ││
│ │ → Customer segmentation for campaigns ││
│ │ ││
│ │ HOW OFTEN: ││
│ │ ☑ How fresh does it need to be? ││
│ │ → Daily is sufficient ││
│ │ ││
│ │ VOLUME: ││
│ │ ☑ How many records? ││
│ │ → ~500K customers ││
│ │ ││
│ │ ACCESS: ││
│ │ ☑ Who needs access? ││
│ │ → Marketing analysts, BI tool ││
│ │ ││
│ │ DEFINITIONS: ││
│ │ ☑ What counts as an "order"? ││
│ │ → Completed, non-refunded orders ││
│ └─────────────────────────────────────────────────────────┘│
│ │
│ COMMON ISSUES TO CLARIFY: │
│ • Definition of metrics │
│ • Time zone handling │
│ • Currency conversion │
│ • Historical data needs │
│ • Backfill requirements │
└─────────────────────────────────────────────────────────────┘
Maintenance
Ongoing Pipeline Work
PIPELINE MAINTENANCE:
┌─────────────────────────────────────────────────────────────┐
│ │
│ MAINTENANCE TASKS (Recurring): │
│ │
│ WEEKLY: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ DATA-MAINT: Weekly pipeline health check ││
│ │ ││
│ │ ☐ Review pipeline failures ││
│ │ ☐ Check data quality trends ││
│ │ ☐ Review resource usage ││
│ │ ☐ Address any alerts ││
│ └─────────────────────────────────────────────────────────┘│
│ │
│ MONTHLY: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ DATA-MAINT: Monthly pipeline maintenance ││
│ │ ││
│ │ ☐ Review and update documentation ││
│ │ ☐ Check for deprecated dependencies ││
│ │ ☐ Optimize slow queries ││
│ │ ☐ Clean up old data/logs ││
│ │ ☐ Review monitoring thresholds ││
│ └─────────────────────────────────────────────────────────┘│
│ │
│ QUARTERLY: │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ DATA-MAINT: Quarterly pipeline review ││
│ │ ││
│ │ ☐ Review overall architecture ││
│ │ ☐ Check schema evolution needs ││
│ │ ☐ Update dependencies ││
│ │ ☐ Review access controls ││
│ │ ☐ Stakeholder check-in (still meeting needs?) ││
│ └─────────────────────────────────────────────────────────┘│
│ │
│ ALLOCATE CAPACITY: │
│ ~20% of data engineering time for maintenance │
│ Don't just build new - maintain existing │
└─────────────────────────────────────────────────────────────┘