Freelance PySpark Expert – Data Warehouse Deployment
📍 Location: Luxembourg (4 days on-site, 1 day remote per week)
📄 Contract Type: Freelance
⏳ Duration: 80 days (Time & Material)
🗣 Languages: English (required); French/German/Luxembourgish (a plus)
About the Role
We are seeking a highly skilled PySpark expert to support the deployment of a high-performance real-time data warehouse. As an external consultant, you will play a critical role in optimizing real-time data ingestion and processing while ensuring a smooth production rollout.
Key Focus Areas:
- Apache PySpark for large-scale data transformation
- Apache Kafka for real-time streaming data
- Debezium for Change Data Capture (CDC) from PostgreSQL
This role requires hands-on expertise in PySpark, Kafka, and Debezium, as well as a collaborative mindset to help our internal team upskill while delivering a robust data infrastructure.
Key Responsibilities
✅ Design and implement efficient Spark-based data processing workflows
✅ Optimize real-time data ingestion pipelines for performance and reliability
✅ Apply best practices for scalability and maintainability of the data warehouse
✅ Fine-tune Kafka and Debezium for seamless CDC log processing
✅ Train and support the internal team for a smooth post-deployment transition
Required Skills & Qualifications
✔ Expert-level proficiency in PySpark (performance optimization, structured job management)
✔ Strong experience in real-time data processing and transformation
✔ Knowledge of data engineering best practices and data warehouse modeling
✔ Experience with Kafka and Debezium for CDC log management
✔ Proficiency in PostgreSQL and TimescaleDB (a plus)
✔ Understanding of real-time data architectures and CI/CD pipelines
✔ Fluency in English (spoken & written); French, German, or Luxembourgish is a plus
Work Schedule & Contract Details
- On-site: 4 days per week
- Remote: 1 day per week
- Contract Type: Freelance (Time & Material)
- Duration: 80 days
Ready to bring your expertise and make an impact? Apply now!
