As a Senior Azure Data Engineer, you will be a key member of our data engineering team responsible for designing, implementing, and optimizing data pipelines and solutions on the Azure cloud platform. You will play a crucial role in building and maintaining data infrastructure, ensuring data availability, reliability, and performance. This position requires a strong background in Azure data services, ETL processes, and a commitment to best practices in data engineering.
Main Duties & Responsibilities
The role must execute the following core responsibilities:
• Design, develop, and maintain end-to-end data pipelines, leveraging Azure Data Factory, Azure Databricks, and other Azure data services.
• Implement efficient data extraction, transformation, and loading (ETL) processes.
• Experience in building end to end architecture for Data Lakes, Data Warehouses and Data Marts
• Collaborate with data architects and other stakeholders to design scalable and robust data solutions.
• Ensure data integration across various source systems and data stores.
• Perform data transformation, cleansing, and enrichment to support analytics and reporting needs.
• Optimize data processing for performance and scalability.
• Leverage Azure cloud services, such as Azure SQL Data Warehouse, Azure Synapse Analytics, and Azure Data Lake Storage, to build and manage data solutions.
• Implement best practices for Azure resource management and cost optimization.
• Implement data governance policies and ensure data security and compliance with industry standards.
• Manage access controls, encryption, and auditing.
• Monitor data pipelines and infrastructure performance, identifying and resolving bottlenecks.
• Continuously optimize data processes for efficiency.
• Maintain comprehensive documentation of data pipelines, architecture, and configurations. Share knowledge and mentor junior team members.
Qualification/ Experience
• Bachelor's or master's degree in computer science, Information Technology, or a related field.
• 5 to 10 years of experience in data engineering with a focus on Azure cloud technologies.
• Expert in Azure Data Factory, Azure Databricks, Azure SQL Data Warehouse, and Azure Data Lake Storage.
• Strong SQL and scripting skills for data manipulation and automation.
• Knowledge of data warehousing concepts, data modeling, and data integration patterns.
• Expertise in Python & PySpark development, DWH, Data Integration, Cloud, Architecture, Design, Data Modelling, Cloud data warehouse like Azure Synapse, Snowflake analytical warehouse
• Familiarity with data governance and compliance standards.
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration skills.
• Microsoft Certified: Azure Data Engineer Associate
• Microsoft Certified: Azure Solutions Architect Expert
