Lead Data Engineer

We are seeking a highly skilled and experienced Lead Data Engineer to lead a critical data warehouse migration project from an on-premises environment to a cloud-native Databricks Data Lake on Azure. This role will be instrumental in designing, developing and implementing robust data solutions that will power critical decision-making processes within the public sector. This role requires a proactive individual who thrives in an agile environment and can drive the product process from research to execution. Key Responsibilities

• Technical Leadership:
• Provide technical leadership and guidance to a team of data engineers and analysts.
• Define and enforce data engineering best practices, standards, and methodologies.
• Oversee the design, development, and implementation of data pipelines, ETL processes, and data models.
Data Warehouse Migration:
• Lead the migration of the existing data warehouse to a cloud-native Databricks Data Lake on Azure.
• Assess the current data warehouse infrastructure and identify migration strategies.
• Design and implement data migration plans, including data extraction, transformation, and loading (ETL) processes.
• Optimize data pipelines for performance and scalability.
Data Lake Design and Implementation:
• Design and implement a robust data lake architecture on Azure Databricks.
• Define data ingestion, storage, and processing strategies.
• Ensure data quality, security, and privacy standards are met.
Data Engineering and Development:
• Develop and maintain data pipelines using Databricks, Azure Data Factory, and other relevant tools.
• Write efficient and scalable data processing code in Python, or SQL (PySpark and SparkSQL).
• Collaborate with data analysts and business stakeholders to understand data requirements and translate them into technical solutions.
Data Governance and Security:
• Implement data governance policies and procedures to ensure data quality and integrity.
• Enforce data security best practices, including access controls, encryption, and data masking.
• Monitor data pipelines and systems for performance and security issues.
Power BI Integration:
• Integrate Databricks with Power BI to enable advanced analytics and reporting.
• Develop custom visualizations and dashboards to provide actionable insights.
Stakeholder Management:
• Communicate effectively with technical and non-technical stakeholders.
• Present complex technical concepts in a clear and concise manner.
• Manage project timelines and deliverables.
Skills needed for this role


• Expert-level: Data development process, Data integration design, Data modelling
• Practitioner-level: Data analysis and synthesis, Data innovation, Metadata management, Problem resolution (data), Programming and build (data engineering), Technical understanding
• Communicating between the technical and non-technical: Practitioner level
Soft Skills: Leadership and Collaboration
• Team Collaboration: Ability to work effectively with multidisciplinary teams, including developers, designers, delivery managers, and stakeholders.
• Stakeholder Management: Strong skills in communicating technical concepts to non-technical audiences and influencing decisions.
• Mentorship: Capability to guide and support less experienced team members, fostering knowledge sharing and skill development.
Agile and Delivery Focus
• Agile Working: Experience working in agile teams, contributing to ceremonies, and integrating technical goals into iterative delivery cycles.
• Delivery Management: Ability to prioritise tasks, manage technical risks, and ensure timely delivery of high-quality solutions.
Analytical and Problem-Solving Skills
• Requirements Analysis: Proficiency in translating business and user needs into technical requirements.
• Problem Solving: Skilled in diagnosing complex technical issues and implementing effective solutions.
• Data-Driven Decisions: Ability to use data, analytics, and performance insights to guide architectural decisions.
Knowledge of Public Sector Standards
• Government Digital Service (GDS): Familiarity with GDS service standards and the Technology Code of Practice.
These skills reflect the need for both technical depth and the ability to navigate the unique demands of the UK public sector environment.,
• Certifications in Azure, Databricks, or related technologies. Example https://WWW.databricks.com/learn/certification/data-engineer-professional
• Experience with public sector data initiatives and compliance requirements.
• Knowledge of machine learning and artificial intelligence concepts.
  
  Scrumconnect is a leading force in technology consultancy, proudly contributing to over 20% of the UK's most significant citizen-facing public services. Our award-winning team has made a substantial impact, delivering more than 64 services in the past two years alone. This work has not only reached over 50 million citizens but also achieved considerable savings for the taxpayer, amounting to over £25 million. At Scrumconnect, we foster a community of talented consultants who thrive on collaboration, sharing knowledge, and continuous learning to address and solve complex challenges. Our mission is to combine advanced software engineering, human-focused design, and data-driven insights to deliver unparalleled service to our clients., Your working time at Scrumconnect will be split between multiple locations, including from our HQ and hub locations, client site or home. Travel requirements vary in frequency and take into account requirements of your work, our clients and the team.
We welcome candidates from all identities, attributes and backgrounds to thrive with us. The diversity of our people should be reflected in the impact we deliver. Join us at Scrumconnect, where your highly demonstrable skills and expertise will drive the future of user-centred design in public services.