Position: Data Engineer
Overview of the Role
An established organisation is currently undergoing a data transformation and is seeking a skilled Data Engineer to join its growing data team. This role plays a key part in building and deploying modern data solutions based on Azure Databricks, enabling faster and more informed business decisions.
You'll work hands-on with Azure Databricks, Azure Data Factory, Delta Lake, and Power BI to design scalable data pipelines, implement efficient data models, and ensure high-quality data delivery. This is a great opportunity to shape the future of data within the organisation while working with advanced cloud technologies.
Key Responsibilities and Deliverables
Design, develop, and optimise end-to-end data pipelines (batch & streaming) using Azure Databricks, Spark, and Delta Lake.
Implement Medallion Architecture to structure raw, enriched, and curated data layers efficiently.
Build scalable ETL/ELT processes with Azure Data Factory and PySpark.
Support data governance initiatives using tools like Azure Purview and Unity Catalog for metadata management, lineage, and access control.
Ensure consistency, accuracy, and reliability across data pipelines.
Collaborate with analysts to validate and refine datasets for reporting.
Apply DevOps and CI/CD best practices (Git, Azure DevOps) for automated testing and deployment.
Optimise Spark jobs, Delta Lake tables, and SQL queries for performance and cost-effectiveness.
Troubleshoot and proactively resolve data pipeline issues.
Partner with data architects, analysts, and business teams to deliver end-to-end data solutions.
Stay current with emerging data technologies (e.g., Kafka/Event Hubs for streaming, Knowledge Graphs).
Promote best practices in data engineering across the team.
Skills & Experience
Hands-on experience with Azure Databricks, Delta Lake, Data Factory, and Synapse.
Strong understanding of Lakehouse architecture and medallion design patterns.
Proficient in Python, PySpark, and SQL, with advanced query optimisation skills.
Proven experience building scalable ETL pipelines and managing data transformations.
Familiarity with data quality frameworks and monitoring tools.
Experience working with Git, CI/CD pipelines, and in Agile environments.
Ability to write clean, maintainable, and well-documented code.
Exposure to Power BI or similar data visualisation tools.
Knowledge of IoT data pipelines is a plus.
