About ISx4
ISx4 is a dynamic and innovative team, we harness the power of artificial intelligence and cloud computing to drive real-world impact. We work at the forefront of Analytics and AI technology, leveraging leading cloud platforms to build scalable, intelligent solutions. We provide Insight through Strategy, Solutions, Services & Support (Sx4). Our industry experience help derive competitive advantage for our clients, taking them on the journey from Hindsight, through Insight to Foresight.
Key Responsibilities
- Design and implement ELT/ETL pipelines using PySpark and Databricks
- Build scalable batch and streaming data pipelines using Spark and Kafka
- Develop optimised SQL and Python pipelines for data transformation and integration
- Integrate external data sources using REST APIs and data ingestion frameworks
- Optimise Spark jobs and cluster performance for reliability and cost efficiency
- Implement data quality checks, validation and monitoring
- Apply CI/CD practices and version control for pipeline deployment
- Work closely with solution architects, DevOps engineers and business analysts to deliver data products
- Produce technical documentation, architecture diagrams and operational runbooks
Key Requirements
- Eligible and willing to pass relevant background checks
- 3+ years experience building production data pipelines
- Strong Python and Microsoft SQL Server development skills
- Hands-on experience with Databricks, Spark and PySpark
- Experience with Delta Lake, Unity Catalog and Databricks Workflows
- Experience with data pipeline design, testing and deployment
- Familiarity with test-driven development, CI/CD practices and Git-based development
- Strong collaboration and communication skills
- Working knowledge of Hadoop
- REST APIs and Kafka experience
Nice to Have
- Cloud experience with AWS, Azure or GCP
- Certifications in Databricks, AWS or data engineering
- Infrastructure as Code (Terraform)
- Orchestration: Airflow or AWS Glue
- Streaming technologies including Structured Streaming
- Observability tooling for data pipelines
- Experience working with regulated industries such as banking or financial services
This role is Hybrid and requires 3 days a week working onsite in Belfast city centre.
UK
