Senior Data Engineer - Databricks (3rd Party Contractor)

 to

Position:         Senior Data Engineer (Databricks & Cloud Data Platforms)

Reporting to:  Practice Head: Data and Automation

ROLE OVERVIEW 

As a Senior Data Engineer, you will play a critical consulting role in designing, building, and modernising enterprise data platforms, with a strong focus on Databricks lakehouse implementations across cloud environments.

You will work closely with solution architects, analysts, data scientists, and client stakeholders to deliver secure, scalable, and high-performing analytics platforms that drive real business impact.

This is a hands-on technical role suited to someone passionate about solving complex data challenges and delivering best-in-class cloud-native solutions.

responsibilities

RESPONSIBILITIES

• Design and implement scalable Databricks-based data platforms
• Build robust ETL / ELT pipelines for batch and streaming workloads
• Lead data migration and modernisation initiatives from legacy platforms to cloud-based lakehouse architectures
• Develop high-performance data processing solutions using PySpark, Python, and SQL
• Implement Delta Lake, Unity Catalog, and modern governance frameworks including Purview
• Design enterprise-grade lakehouse and medallion architectures
• Optimise Spark workloads for performance, scalability, and cost efficiency
• Build orchestration solutions using Databricks Workflows, ADF, Airflow, or similar tools in Azure/AWS
• Collaborate with technical and business stakeholders to define solution requirements
• Mentor junior engineers and contribute to technical best practices within the Data & AI practice
• Produce technical documentation, architecture artefacts, and reusable delivery assets
  

IDEALLY YOU ARE:

• A strong technical leader who can own complex delivery outcomes
• Comfortable engaging directly with clients and stakeholders
• A proactive problem-solver with excellent analytical skills
• Able to clearly communicate technical concepts to both technical and non-technical audiences
• Passionate about mentoring and uplifting engineering teams
• Delivery-focused, quality-driven, and adaptable in consulting environments
  

MINIMUM EXPERIENCE

• 8+ years in Data Engineering projects
• 3+ years of hands-on Databricks delivery experience
• Deep expertise in Python and PySpark
• Strong understanding of the Medallion Architecture and Lakehouse Architecture
• Hands-on experience building and maintaining live Databricks AI/BI Governance Dashboards using SQL
• Strong experience with Azure data services including Synapse, ADF, OneLake, Purview
• Practical exposure to AWS data platforms including Glue, S3, Redshift
• Proven track record delivering data migration and modernisation projects
• Experience working in consulting or client-facing environments
• Bachelor’s degree in computer science, Engineering, Information Systems, or equivalent practical experience

TECHNICAL REQUIREMENTS

• Databricks Expertise (Essential)
• Cloud Data Platform Experience (Azure Fabric, AWS) (Essential)