Lead Data Engineer - Databricks | AWS | PySpark
Dublin (Hybrid - 3 Days Onsite)
12‑month contract
Financial Crime Analytics | Advanced Data Engineering | Global Brand
Are you a seasoned Lead Data Engineer ready to build high‑impact data platforms that help financial institutions fight fraud and money laundering?
This is an exceptional opportunity to join one of the world's most recognised technology brands, designing and evolving next‑generation Databricks + AWS lakehouse platforms used in real‑world financial crime detection.
Our client is scaling a high‑performing engineering team and is seeking a technical leader who brings deep expertise in distributed data processing, cloud architectures, orchestration frameworks, and modern data governance.
🌟 Why This Role Is Exceptional
- Work with one of the most respected global payment technology companies.
- Drive the design of a high‑value, enterprise-grade lakehouse powering fraud detection and AML insights.
- Influence engineering standards and mentor other data engineers across a cutting-edge data organisation.
- Gain exposure to advanced analytics, behavioural modelling, and financial crime detection use cases.
- Long-term contract with strong potential for extension and future opportunities.
🧠 What You'll Be Doing
As the Lead Data Engineer, you will:
🏗️ Architect & Build
- Own end‑to‑end design and optimisation of Spark / PySpark pipelines on Databricks (batch + streaming).
- Implement Medallion Architecture (Bronze / Silver / Gold), schema governance, metadata and lineage (Unity Catalog / Glue / OpenLineage).
- Build secure, scalable AWS data infrastructure (S3, IAM, KMS, Glue, Lake Formation, Lambda, EKS/EC2).
⚙️ Engineer for Scale & Reliability
- Implement orchestration using Airflow, Databricks Workflows, Step Functions.
- Embed data quality frameworks (expectations, anomaly detection, reconciliation, contract tests).
- Optimise Delta Lake performance (partitioning, ACID transactions, caching, file layout).
🔐 Ensure Compliance & Security
- Apply robust access controls, tokenisation, masking, encryption and key rotation.
- Build audit‑ready lineage and metadata frameworks for regulatory transparency.
🚀 Enable Continuous Delivery
- Drive CI/CD for data assets: Terraform/CloudFormation IaC, notebook testing, semantic versioning, automated deploys.
🤝 Collaborate & Lead
- Work closely with Data Science, Security, Compliance and Product teams.
- Mentor engineers on best practices in distributed systems, cost optimisation and performance tuning.
- Support incident response, root‑cause analysis and preventative engineering.
🎯 What We're Looking For
To thrive in this role, you will need:
Must‑Have Expertise
✔ 10+ years in data engineering
✔ Deep hands‑on experience with Databricks, Spark/PySpark, Python, SQL
✔ Strong AWS data stack: S3, Glue, Lake Formation, IAM, networking, encryption
✔ Orchestration experience: Airflow, Step Functions, Databricks Workflows
✔ CI/CD: Git, PR workflows, automated Databricks/AWS deployments, IaC
✔ Experience building large‑scale data pipelines (batch + streaming)
Nice-to-Have Skills
- Apache NiFi ingestion pipelines
- Delta Lake performance tuning
- Financial crime / AML domain experience
- OpenTelemetry or observability tooling
- Python packaging for shared libraries
📩 Interested? Apply Now
If you're a Lead Data Engineer ready to design mission‑critical data solutions in a world‑class engineering environment, we want to hear from you.
📮 Send your CV directly to Fiona.devine@harveynash.ie
📞 Or message us here on LinkedIn for a confidential discussion
