Lead Data Engineer (Azure/Databricks)
Location: Sheffield (2 days a week) / Hybrid
Reporting to: VP of Engineering
The Team: 2 Data Scientists, 1 Data Engineer
The Opportunity
We are an intelligent asset management company specialising in industrial IoT data. We ingest high-frequency factory and sensor data, apply machine learning techniques, and present high value actionable insights to analysts and customers.
We are building a cutting-edge UNS aligned Industrial IoT data platform with a highly scalable architecture to handle our next phase of growth. We need a hands-on technical leader to own the data engineering domain, help re-architect and build our pipelines from the ground up, and mentor a small but talented team.
The Mission
You will not be maintaining a legacy system; you will be architecting the new one. You will take ownership of the data flows within the platform, designing a robust architecture that supports both real-time operational views and deep historical analysis.
The Tech Stack
• Core: Azure, MQTT, Databricks, Python, SQL, dbt
• Storage & Serving: Delta Lake, Postgres, TimescaleDB
• Modelling: MLflow
• Visualisation: Grafana
Primary Objectives (First 6–12 Months)
• Re-platforming: Own the redesign and rebuild of the data transformation layer (post-event bus) in Databricks. Move from ad-hoc scripts to software engineering standards (CI/CD, testing, modular code)
• Modelling Implementation: Support the implementation of the "Unified Name Space" (UNS) across the data estate, to include schema and path standardisation; machine hierarchies and semantic relationships; and methodologies for validation and processing of data against contracts.
• Enablement: Establish a stable data serving layer for Grafana and Analytics (via Databricks/MLflow), unblocking the Data Science team.
