About The Role
In this role, you will drive a critical migration of AWS data pipelines from HDF5 files to Iceberg tables, working with C++, Python, and large datasets to modernize a high-performance data architecture. Our Client is a London-based leader in harmonized historical order book and market data, powering advanced trading analytics and AI-driven insights. You’ll collaborate in a focused team to deliver scalable, efficient data foundations that directly impact global capital markets.
Responsibilities
- Design and implement new AWS data pipeline components in C++ and Python to switch output from HDF5 files to Iceberg tables, improving efficiency and modernizing the data architecture
- Own end-to-end migration tasks within the AWS-based data pipelines, ensuring scalable handling of large datasets and minimizing risk to existing data flows and performance
- Collaborate closely with a small, focused team to coordinate changes across C++ services, Python tooling, and AWS data workflows during the migration from HDF5 to Iceberg
- Analyze existing C++ logic that generates HDF5 files and refactor or extend it to produce Iceberg tables, maintaining data integrity and schema consistency across large datasets
- Integrate Python-based utilities and scripts into the AWS data pipelines to support data transformation, validation, and loading of large datasets into Iceberg tables
- Optimize data processing paths in C++ and Python for large-scale datasets within AWS, aiming to enhance throughput, reliability, and scalability of the new Iceberg-based pipelines
- Troubleshoot, debug, and resolve issues arising during the transition from HDF5 file generation to Iceberg table creation in AWS, ensuring stable and predictable pipeline behavior
Requirements
- Strong proficiency with modern C++
- Practical experience with C++ development for data processing pipelines, including performance optimization and debugging
- Hands-on Python skills and experience with building and maintaining data processing scripts and utilities
- Hands-on experience with designing, implementing, and maintaining data pipelines in AWS environments
- Upper-Intermediate English level or higher
- Analytical and solution-oriented mindset with strong problem-solving and troubleshooting skills in complex data environments
- Practical experience with data formats and serialization, including HDF5/Iceberg, and migrating or transforming data between formats (nice-to-have)
- Experience with big data frameworks or ecosystems (as an advantage)
SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.
PL
