Role Description
Lead Data Engineer (US Healthcare / Medicare Advantage) Position Overview We are seeking a Lead Data Engineer to architect, build, and optimize scalable, cloud-native data systems. In this role, you will lead the development of enterprise data platforms using Databricks and AWS. You must possess deep domain expertise in US Healthcare, with a strict requirement for hands-on experience handling Medicare Advantage data. Key Responsibilities Lead the design, implementation, and maintenance of robust data pipelines on AWS using Databricks. Architect scalable lakehouse structures leveraging Apache Iceberg to ingest, process, and store complex healthcare data types. Establish comprehensive data governance and data lineage frameworks using Databricks Unity Catalog. Implement secure access controls, data masking, and row-level security through Role-Based Access Control (RBAC) to ensure data security rules are enforced. Architect and scale ingestion frameworks capable of processing HL7 and FHIR data feeds natively into the lakehouse ecosystem. Oversee the end-to-end processing of Medicare Advantage data assets, ensuring accurate risk adjustment and reporting workflows. Ensure strict adherence to healthcare regulations, including HIPAA compliance and robust data governance frameworks. Mentor junior engineers, conduct code reviews, and drive technical best practices across the data team. Required Skills & Qualifications 7+ years of data engineering experience, with at least 2 years in a technical lead role. Production experience building and optimizing large-scale pipelines within Databricks. Production experience deploying data infrastructure on AWS (specifically S3, EMR, Lambda, IAM, and Glue). Comprehensive understanding of US Healthcare data, with explicit, hands-on experience processing Medicare Advantage data (e.g., CMS enrollment, RAPS/EDPS submissions, MMR/MOR files, or HCC risk adjustment models). Proven experience working with healthcare interoperability standards, specifically FHIR and HL7 models. Deep understanding and practical experience working with open table formats, specifically Apache Iceberg. Expert-level knowledge of Databricks Unity Catalog for centralized data governance. Proven experience implementing enterprise-level Role-Based Access Control (RBAC) and row-level security for protected health information (PHI). Mastery of Python, PySpark, and advanced SQL for complex distributed computing tasks. Good-to-Have Skills Databricks Certified Data Engineer Professional or AWS Certified Data Analytics certification. Familiarity with orchestration tools like Apache Airflow or AWS Step Functions.
Skills
Databricks, AWS, FHIR, Python, Apache Airflow
IN
