Summary:
We are seeking a skilled and collaborative Data Engineer with expertise in migrating data pipelines from Informatica to Databricks. The ideal candidate will have a strong background in data engineering, including experience in implementing data pipelines, ensuring data governance and security, maintaining data lineage, and working as a team player.
• Migrate data pipelines from Informatica to Databricks, ensuring seamless integration and efficient data processing.
• Design, develop, and optimize data processing workflows using SQL, PL/SQL, or python.
• Implement and enforce data governance and security measures to protect sensitive data and ensure compliance.
• Establish and maintain data lineage documentation to track the origin, transformations, and destinations of data.
• Collaborate with cross-functional teams to identify data requirements and implement optimal solutions.
• Monitor and troubleshoot data pipelines to ensure smooth and uninterrupted data flow.
• Conduct testing and validation of data pipelines to ensure accuracy, completeness, and data quality.
• Optimize data storage, retrieval, and archival processes in collaboration with database administrators and infrastructure teams.
• Stay updated with industry trends and advancements in data engineering, big data technologies, and cloud computing.
• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proven experience in migrating data pipelines from Informatica to Databricks.
• Strong proficiency in Oracle 12c, ADW, MSSQL, and MYSQL.
• Proficient in programming languages such as Python, Scala, or SQL for data processing and transformation.
• Solid understanding of data governance and security principles.
• Experience in maintaining data lineage documentation.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform
