Role Overview:
ETL & Data Engineer to design, build, and maintain robust data pipelines and backend services that power AI-driven operations. The role involves working with high-volume IT and cloud data, optimizing ETL processes, and integrating with AIOps platforms and ML pipelines.
Key Responsibilities:
• Build and maintain scalable ETL pipelines for batch and real-time data ingestion, transformation, and loading from diverse sources (IT infrastructure, cloud, monitoring systems, APIs).
• Implement data validation, cleansing, and normalization for consistent AI model input.
• Develop backend services and APIs to support data ingestion, metadata management, and configuration.
• Optimize ETL jobs for performance, fault tolerance, and low latency.
• Integrate with AIOps platforms and ML pipelines using REST APIs or event-driven architectures.
• Schedule and monitor ETL workflows using tools like Airflow, Prefect, or Dagster.
• Support CI/CD pipelines for deploying ETL services and full-stack applications.
Required Skills & Tools:
• Programming & Scripting: Python, Go (Golang), Java, Ruby, JavaScript/TypeScript (Next.js)
• ETL & Data Engineering: Apache NiFi, Spark, Airflow, Flink, Kafka, Talend
• Orchestration: Airflow, Prefect, Dagster
• Data Storage & Lakes: PostgreSQL, MongoDB, Elasticsearch, Snowflake, BigQuery, S3, GCS, Azure Blob
• Streaming Platforms: Kafka, Kinesis, Pub/Sub
Good to Have:
• Experience with AIOps & Observability tools like Splunk, Dynatrace, AppDynamics, New Relic, Elastic Stack
• Familiarity with ITSM systems (ServiceNow) and CMDB integrations
• Understanding of metrics, logs, and traces for AI-driven operations
