Responsibilities
We are looking for a seasoned BAU Lead - Data Engineering to lead our data
operations and support function in a cloud-first environment. The ideal
candidate will bring strong technical leadership, hands-on expertise in AWS
and Snowflake, and proven experience in managing Business-As-Usual (BAU)
data platforms, production support, and continuous improvement initiatives.
▪ Develop tools to improve data flows between internal/external systems
and the data lake/warehouse.
▪ Work with stakeholders to understand needs for data structure, availability,
scalability, and accessibility.
▪ Build robust and reproducible data ingest pipelines to collect, clean,
harmonize, merge, and consolidate data sources.
▪ Understanding existing data applications and infrastructure architecture
▪ Build and support new data feeds for various Data Management layers and
Data Lakes
▪ Evaluate business needs and requirements.
▪ Support migration of existing data transformation jobs in Oracle, and MS[1]SQL to Snowflake.
▪ Lead the migration of the existing data transformation jobs in Oracle, Hive,
Impala etc. into Spark, Python on Glue etc.
▪ Able to document the processes and steps.
▪ Develop and maintain datasets.
▪ Improve data quality and efficiency.
▪ Lead Business requirements and deliver accordingly.
▪ Collaborate with Data Scientists, Architect and Team on several Data
Analytics projects.
▪ Collaborate with DevOps Engineer to improve system deployment and
monitoring process.
▪ Experience in critical production support and how the BAU functions is
preferred.
Requirements Key Skills:
▪ Lead a team of data engineers in managing day-to-day BAU operations,
production support, incident management, and platform stability.
▪ Strong AWS knowledge in terms of designing new architecture and
providing optimized solutions for existing ones. (S3, Glue, DMS, MWAA,
AMS, IAM).
▪ In-depth knowledge with respect to Snowflake and its architecture.
▪ Prefer prior Experience in BAU environment.
▪ Good knowledge on Airflow and MWAA.
▪ Hands-on experience in SQL/Python/Pyspark.
▪ Expertise in optimizing techniques in cloud environments.
▪ Should have the vision on data strategy and able to deliver the same.
▪ Should be able to lead the design and implementation of data management
processes, including data sourcing, integration, and transformation.
▪ Able to manage and lead a team of data professionals, providing guidance,
mentoring and foster a collaborative and innovative team culture focused on
continuous improvement.
▪ To evaluate and recommend data-related technologies, tools, and
platforms.
▪ Collaborate with IT teams to ensure seamless integration of data solutions.
▪ Should have experience in Implementing and enforcing data security
protocols and ensure compliance with relevant regulations.
Required Qualifications:
▪ At least 8+ years of Data Engineer Experience
▪ Bachelor qualification in a computer science or STEM (science, technology,
engineering, or mathematics) related field.
▪ At least 5+ years of recent hands-on professional experience (actively
coding) working as a Lead handling support & production issue.
▪ 2+ experience with large scale datasets, data lake and data warehouse
technologies such as AWS Redshift, Google BigQuery, Snowflake. Snowflake
is highly preferred.
▪ At least 3+ years of experience in ETL (AWS Glue), Amazon S3, Amazon
RDS, Amazon Kinesis, Amazon Lambda, Apache Airflows, Amazon Step
Functions.
▪ Professional experience working in an agile, dynamic and customer-facing
environment is required.
▪ Understanding of distributed systems and cloud technologies (AWS) is
highly preferred.
▪ Understanding of data streaming and scalable data processing is preferred
to have.
▪ Strong knowledge in scripting languages like SQL ,Python, UNIX shell and
Spark is required.
▪ Understanding of RDBMS, Data ingestions, Data flows, Data Integrations
etc.
▪ Technical expertise with data models, data mining and segmentation
techniques.
▪ Experience with full SDLC lifecycle and Lean or Agile development
methodologies.
▪ Knowledge of CI/CD and GIT Deployments.
▪ Ability to work in team in diverse/ multiple stakeholder environment.
▪ Ability to communicate complex technology solutions to diverse teams
namely, technical, business and management teams
Soft Skills
▪ Ability to work in a collaborative environment and coach other team
members on coding practices, design principles, and implementation
patterns that lead to high quality maintainable solutions.
▪ Excellent communications and stake-holder management are required.
▪ Ability to handle senior leadership and report to them if required.
▪ Ability to work in a dynamic, agile environment within a geographically
distributed team.
▪ Ability to focus on promptly addressing customer needs.
▪ Ability to work within a diverse and inclusive team.
▪ Technically curious, self-motivated, versatile and solution-oriented
