Data Engineer/Databricks Developer
New York - 3 days a week onsite work
Onsite Interview Required - local candidates only
Contract Opportunity
- Must have Investment Banking industry experience
Responsibilities
- Design and implement scalable data pipelines using Azure Databricks and Azure Data Factory.
- Develop and maintain robust ETL/ELT workflows using Apache Spark and PySpark DataFrames.
- Build and optimize data pipelines for efficient data ingestion, transformation, and processing of large datasets.
- Utilize data governance tools to manage data access, security, compliance, and data lifecycle.
- Implement data quality frameworks and maintain data lineage across enterprise data platforms.
- Design and support modern data architecture using Lakehouse and distributed data processing principles.
- Develop high-performance Spark and SQL-based data transformation.
- Apply Master Data Management (MDM) concepts to ensure data consistency, integrity, and standardization.
- Work within the Azure Databricks ecosystem to deliver scalable, reliable, and enterprise-grade data engineering solutions.
Required Skills
- Azure Databricks
- Apache Spark / PySparkAzure
- Data Factory
- SQL
- Data Pipelines and ETL/ELT
- Data Governance and Data Lineage
- Lakehouse Architecture
US
