Lead Data Engineer
£80,000 - £98,000 + Excellent Pension!
Data Engineer / Data Developer / Data / Data Science / Big Data / Azure / Hadoop / Hive / SQL / Python
Working for a Government funded regulatory body who are using cutting edge data focused technology, the successful candidate will have the opportunity to work in a well funded and talented Data Services team.
The Lead Data Engineer will lead the technical development work directly reporting to the Head of Data. The Lead Engineer has the autonomy to shape and direct data technology development for Data Services, and therefore influence the wider data architecture across the business.
- End-to-end data architecture and handling approach in the Data Services platform. This will involve defining and communicating technical needs for data collection, ingestion, transformation and calculation. (Reporting to the Head of Data)
- Design pipelines, create proof of concepts to production, run functional tests, end to end regression testing, quality/automation tests
- Carry out data modelling and work closely with data scientists
- Lead a team of engineers, supported by QA and analyst staff across London and Glasgow
- Deliver continuous improvements to the functionality to the Data Service platform and evolve the platform to meet business needs accordingly
- Assure that your work and the work of colleagues meets governing standards
- Ensure principles of privacy and security are assessed and designed into our data solutions
- Encourage teamwork, communication and follow agile methodologies.
Promote our team knowledge retention through documentation and providing training where required.
Skills / Abilities:
- Hands on experience (including QA) in platforms such as Azure and the Hadoop ecosystem (HIVE, HDFS and Apache spark)
- Proven ability in designing pipelines, proof of concept to production, functional tests, end to end regression testing, quality automation testing and CI/CD
- Extensive experience in data Modelling, including dimensional models time series and data mining
- Master Data Management
- Experience working with and integrating into Business Intelligence tools
- Expert competency with Python and SQL
- Good knowledge of Linux
- Experience of effectively engaging and communicating technical concepts to a non-technical audience
- Business minded manager and engineering team leader
- Proven ability to understand business priorities and design, deliver and communicate technical solutions to meet them
- Experience developing engineer's technical capability
- Experience defining and implementing effective data engineering working practices in agile teams
- Ability to design and provide effective support for knowledge transfer / retention
- Proven experience of innovation and improving best practice