Responsibilities:
● Design, construct, install, test, and maintain highly scalable data management systems.
● Ensure systems meet business requirements and industry practices.
● Build high-performance algorithms, prototypes, predictive models, and proof of concepts.
● Research opportunities for data acquisition and new uses for existing data.
● Develop data set processes for data modeling, mining, and production.
● Recommend ways to improve data reliability, efficiency, and quality.
● Collaborate with data architects, modelers, and IT team members on project goals.
Requirements:
● Proven experience as a Data Engineer or in a similar role.
● Strong expertise in Apache Airflow to orchestrate complex data workflows.
● Deep understanding of big data technologies and architecture, including MapReduce and streaming processing.
● Proficiency in at least one data queue framework such as Apache Kafka.
● Demonstrated experience in building and optimising data pipelines, architectures, and data sets.
● Solid understanding of data warehouse concepts and experience with data warehouse solutions.
● Mastery of SQL, including complex query writing, data manipulation, and optimisation.
● Experience with cloud services (AWS, Azure, GCP) and understanding of how they can be leveraged in data engineering.
● Strong analytical skills with the ability to collect, organise, analyse, and disseminate significant amounts of information with attention to detail and accuracy.
● Excellent problem-solving abilities and willingness to think outside the box.
● Bachelor’s degree in Computer Science, Engineering, or related field; Master’s degree preferred.
Desirable Skills:
● Knowledge of machine learning algorithms and data science principles is a plus.
● Experience with non-relational & relational databases (SQL & NoSQL).
● Familiarity with Python, Java, or Scala for data engineering tasks.
Working Location - Robinson