JOB DESCRIPTION
Job Description
Job Title: Data Engineer - Databricks Performance & Optimization
Houston TX
Compensation: Base salary range: $130,000–$150,000 plus 5% bonus.
Benefits package available (details to be discussed).
Benefits: This position is eligible for benefits
Qualifications: Build and operate Lakehouse pipelines on Databricks (Bronze/Silver/Gold) using Delta Lake, Delta Live Tables (DLT), and/or Jobs.
• Optimize ingestion patterns (Autoloader, CDC, streaming).
• Model data, implement quality checks, and performance optimization.
• Profile and tune Spark/SQL workloads: partitioning, clustering, constraints, liquid clustering.
Job Description:• Engineer Delta tables for speed and cost: partitioning, Z-Ordering/clustering, constraints, file sizing; manage table health with Auto Optimize, OPTIMIZE, and VACUUM.
• Implement incremental processing (MERGE with Change Data Feed, APPLY CHANGES INTO) with idempotency and exactly-once delivery.
• Deliver reliable, well-documented datasets with clear SLAs.
• Design and implement dashboards and reports using Power BI and other visualization tools.
• Collaborate with business units to gather requirements and deliver technical solutions.
• Integrate data from multiple sources, including real-time field equipment and sensors.
• Educate and support stakeholders on data tools and best practices.
• Engage in continuous improvement and adoption of new data management technologies.
