Lead Data Engineer

This job has now expired please search on the home page to find live IT Jobs.

Lead Data Engineer£80,000 - £98,000 + Excellent Pension!Central London Data Engineer / Data Developer / Data / Data Science / Big Data / Azure / Hadoop / Hive / SQL / PythonWorking for a Government funded regulatory body who are using cutting edge data focused technology, the successful candidate will have the opportunity to work in a well funded and talented Data Services team.The Lead Data Engineer will lead the technical development work directly reporting to the Head of Data. The Lead Engineer has the autonomy to shape and direct data technology development for Data Services, and therefore influence the wider data architecture across the business.Responsibilities:End-to-end data architecture and handling approach in the Data Services platform. This will involve defining and communicating technical needs for data collection, ingestion, transformation and calculation. (Reporting to the Head of Data)Design pipelines, create proof of concepts to production, run functional tests, end to end regression testing, quality/automation testsCarry out data modelling and work closely with data scientistsLead a team of engineers, supported by QA and analyst staff across London and GlasgowDeliver continuous improvements to the functionality to the Data Service platform and evolve the platform to meet business needs accordinglyAssure that your work and the work of colleagues meets governing standardsEnsure principles of privacy and security are assessed and designed into our data solutionsEncourage teamwork, communication and follow agile methodologies.Promote our team knowledge retention through documentation and providing training where required. Skills / Abilities:Hands on experience (including QA) in platforms such as Azure and the Hadoop ecosystem (HIVE, HDFS and Apache spark)Proven ability in designing pipelines, proof of concept to production, functional tests, end to end regression testing, quality automation testing and CI/CDExtensive experience in data Modelling, including dimensional models time series and data mining Master Data ManagementExperience working with and integrating into Business Intelligence toolsExpert competency with Python and SQLGood knowledge of LinuxExperience of effectively engaging and communicating technical concepts to a non-technical audienceBusiness minded manager and engineering team leader Proven ability to understand business priorities and design, deliver and communicate technical solutions to meet themExperience developing engineer's technical capabilityExperience defining and implementing effective data engineering working practices in agile teamsAbility to design and provide effective support for knowledge transfer / retentionProven experience of innovation and improving best practice