Dice is the leading career destination for tech experts at every stage of their careers. Our client, Purple Drive Technologies LLC, is seeking the following. Apply via Dice today!
Data / ML Platform Engineer (Python, LLM Pipelines, Batch Processing)
Job Summary
We are looking for a
Data / ML Platform Engineer to design and manage
Python-based batch pipelines for evaluating machine learning and LLM models.
The role involves building scalable pipelines that process large datasets, interact with LLM APIs, and store structured outputs for analytics and benchmarking.
Key Responsibilities
ML / Data Pipeline Development - Build and maintain Python batch pipelines for:
- Model evaluation
- Dataset processing
- Design resumable and fault-tolerant workflows
LLM Integration - Integrate with:
- GPT / Claude / other LLM APIs
- Execute large-scale evaluation workflows
Data Engineering - Load and process data from:
- Cloud storage (S3, GCS, Azure Blob)
- Transform and structure data for analytics
Database & Storage - Store results in:
- MongoDB (structured outputs)
- Ensure efficient data retrieval and storage
Performance & Scalability - Optimize:
- Pipeline execution
- Cost and latency
- Handle high-volume batch jobs
Cloud & DevOps - Deploy pipelines on:
- AWS / Google Cloud Platform / Azure
- Use:
- CI/CD pipelines
- Version control (Git)
Monitoring & Reliability - Implement:
- Logging and monitoring
- Error handling & retries
- Ensure pipeline stability
Required Skills
Core Skills - Strong Python development
- Experience with:
- Batch pipelines / ETL
- Data processing
AI / ML - Hands-on with:
- LLM APIs (GPT, Claude)
- Prompt engineering
- Experience with:
- RAG pipelines (preferred)
Data & Storage
- MongoDB / NoSQL databases
- Data structuring and transformation
Cloud
- AWS / Google Cloud Platform / Azure
- Cloud storage services
DevOps
- Git / GitHub
- CI/CD pipelines
Experience Required
- 5-8 years total experience
- 2+ years in ML / data pipelines / LLM-based systems
US
