Data Engineer – ETL & Streaming

Data Engineer – ETL & Streaming (Kafka, Airflow, Spark) Location: UAE Experience: 4–7 years Department: Data Engineering About the Role We are looking for a Data Engineer to develop robust and scalable data pipelines for both batch and real-time processing using Apache Kafka, Airflow, and Spark. This role focuses on implementing business logic, transformations, and data delivery into data lakes and warehouses. Key Responsibilities • Build ETL and ELT pipelines integrating multiple source systems (RDBMS, APIs, events). • Develop data ingestion and transformation jobs in Spark (PySpark or Scala). • Design and deploy Airflow DAGs to orchestrate end-to-end data workflows. • Implement real-time data ingestion and processing using Kafka (streams, connectors, topics). • Ensure data quality, consistency, and lineage across datasets. • Collaborate with data modelers and BI teams to deliver curated datasets. • Troubleshoot pipeline failures, optimize jobs for performance and cost efficiency. • Work closely with the platform team to integrate pipelines into the enterprise data platform. • Follow DevOps and CI/CD best practices for deployment and monitoring. Skills & Qualifications • Strong programming skills in Python (preferred) or Scala/Java. • Hands-on experience in Kafka (producers, consumers, stream processing, schema management). • Proficient in Airflow DAG development, scheduling, and task dependency handling. • Solid experience with Spark (batch, streaming, dataframes, and optimization). • Working knowledge of SQL and experience with data modeling (Star/Snowflake). • Experience with data lake and warehouse environments (Azure Synapse, Databricks, Snowflake, Redshift, etc.). • Understanding of data versioning, partitioning, and incremental load strategies. • Familiar with Git-based development, CI/CD, and observability practices. Job Type: Contract Contract length: 12 months Pay: AED15,000.00 - AED30,000.00 per month