Overview
The Senior Data Engineer will lead the design, development, automation, and optimization of scalable cloud and on-premises data infrastructure supporting enterprise applications, advanced analytics, and AI/ML workloads. Operating within an agile environment, this role requires close collaboration with software engineering, infrastructure, and data science teams to deliver secure, reliable, and high-performance data systems aligned with enterprise standards.
Responsibilities
• Design, build, and optimize scalable ETL/ELT pipelines using Databricks, Delta Lake, Python, SQL, and related data engineering technologies.
• Demonstrate expert-level proficiency in PySpark and Databricks Lakehouse architectures, including schema evolution, ACID transactions, data quality enforcement, and cost and performance optimized handling of large, complex datasets.
• Improve data ecosystem efficiency by implementing Lakehouse based bronze/silver/gold architectures, reducing processing times through optimization techniques, enforcing data quality frameworks such as Great Expectations, and enhancing storage and retrieval patterns.
• Collaborate with cross functional teams—data scientists, engineers, product owners, and business stakeholders—to translate business requirements into robust technical designs supporting AI/ML enablement, feature engineering, real time analytics, and production grade data workflows.
• Strengthen CI/CD and data governance practices through automation, version control, testing frameworks, and scalable deployment patterns for data pipelines and platform components.
Qualifications
Strong background across Azure, Snowflake, Delta Lake, Databricks and CI/CD automation, bringing the cross cloud, governance minded, and automation driven engineering approach required for the organizations integrated enterprise-scale data architecture.
Top 3 Skills:
• Azure
• Data Knowledge on Snowflake, Delta Lake and Databricks
• CI/CD
