SENIOR DATA ENGINEER - Python

Title

Senior Data Engineer

Skills (must have)

• 5+ years of professional experience in data engineering or related fields.
• Strong programming skills in Python, SQL and PySpark.
• Advanced experience building & optimizing ETL/ELT pipelines using Azure and open-source tools:
  • Azure Data Factory (ADF)
  • Azure Databricks (Spark, Delta Lake)
  • Apache Airflow
  • Azure Functions or Azure Synapse Pipelines
• Expert-level SQL development, including complex queries, stored procedures, analytical functions and performance tuning.
• Strong experience with Azure Snowflake, including:
  • Warehouse tuning
  • Cost/performance optimization
  • Snowpipe, Streams, Tasks
  • Snowpark for advanced processing
• Experience building scalable data models (Kimball, Data Vault, Lakehouse).
• Strong experience with Azure Cloud ecosystem, including:
  • Azure Data Lake Storage (ADLS Gen2)
  • Azure Synapse Analytics (Serverless & Dedicated SQL Pools)
  • Azure Databricks
  • Azure Key Vault
  • Azure Event Hub / IoT Hub
  • Azure Monitor / Log Analytics
• Experience integrating APIs, streaming data and external data sources.
• Strong understanding of data governance & data quality frameworks:
  • Microsoft Purview (cataloging, lineage, classifications)
  • Collibra
• Experience with testing frameworks: unit tests, integration tests, Great Expectations, dbt tests.
• Hands-on with CI/CD pipelines using:
  • GitHub Actions
  • Azure DevOps
  • Jenkins
• Experience working with Infrastructure as Code tools:
  • Terraform or Bicep

Skills (good to have)

• Experience with Azure Kubernetes Service (AKS) or Dockerized workloads.
• Experience building high-performance APIs using FastAPI, Flask, or Django.
• Experience with real-time/streaming systems:
  • Azure Event Hub
  • Azure Stream Analytics
  • Kafka
• Familiarity with Microsoft Fabric including:
  • Lakehouse
  • Data Pipelines
  • Warehouses
• Experience with ML Ops or Feature Store integration (Databricks Feature Store, Azure ML).
• Experience with security frameworks (RBAC, ABAC, managed identities).

Responsibilities

• Lead the design, development and deployment of high-performance ETL/ELT pipelines on Azure and Snowflake.
• Partner with business stakeholders, architects and data consumers to understand requirements and build scalable data solutions.
• Design and optimize Azure Lakehouse solutions using:
  • ADLS Gen2
  • Delta Lake
  • Azure Databricks
  • Synapse Analytics
  • OneLake (if using Microsoft Fabric)
• Design and optimize data models to support BI, analytics and machine learning.
• Build and maintain data ingestion frameworks for:
  • Batch pipelines
  • Real-time pipelines
  • API-based ingestion
• Implement robust data quality, data validation and metadata management processes.
• Drive performance tuning across Spark jobs, Snowflake warehouses, SQL pool queries and cost optimization.
• Monitor and maintain production data systems using Azure-native monitoring tools.
• Define, enforce and improve engineering standards, including automation, CI/CD and IaC best practices.
• Collaborate closely with DevOps/Platform teams to automate data infrastructure deployments.
• Troubleshoot complex issues across distributed systems, cloud networks and data platforms.
• Mentor junior and mid-level engineers, conduct code reviews and guide best practices.
• Maintain clear architectural and technical documentation.

Senior-Level Behavioral Expectations

• Provide technical leadership and drive decision-making during architecture and design discussions.
• Effectively communicate with both technical and non-technical audiences.
• Strong ownership mindset and proactive approach to solving technical challenges.
• Ability to break down complex requirements into actionable engineering tasks.
• Promote continuous learning, experimentation and innovation within the data engineering team.