Lead Data Engineer - Edinburgh
Salary Up to £60k
Lead Data Engineers across Data Platforms (Data Lake, ODS and Data Warehouse) to join Data, Analytics & Platforms function. The new function has been created by consolidating teams across the business as well as building teams to support new cloud platforms and activities transferring from their existing IT provider.
- Design, build and test data solutions across Data Platforms (i.e. Data Lake, ODS and Data Warehouse) whilst following best practice to leverage platform processing power, creating reusable code and aligning to development standards
- Analysis and investigation of environment and source data to support incident and change management activities
- Drive continuous improvement to the data platforms through enhancement of new and existing data solutions, ongoing maintenance and troubleshooting of incidents and defects
- Develop strong working relationships with key stakeholders across business teams whilst working across the full software delivery life cycle (i.e. analysis, design, build, test and implementation)
Skills / Experience:
- This role requires strong business and data analysis skills to support data solution and design activity.
- Experience working with business stakeholders to understand business requirements and to translate these into a logical solution design.
- Experience querying source data using SQL to identify source data items and to document transformation logic required to apply business rules
- Strong background in database design and development, following best practice to leverage database processing power, create reusable code and align to data model build standards. Proven experience in data integration development that adheres to principles set out for scalability, logging, scheduling, alerts, re-runability and dependency management.
- Experience of delivering solutions that align with Data Governance principles for Data Lineage, Data Quality and Data Security.
- Familiarity of working within a formal System Development Life Cycle, particularly agile or iterative / incremental software development (i.e. Waterfall, Scrum, Kanban)
- Proven experience of delivering enterprise data solutions in a Financial Services environment.
- Technical skills for relevant Data Platform:
- Platform purpose - A real-time data store used to reduce processing on the mainframe and support customer facing applications through the provision of data APIs
- Data Storage - Kafka, Syclla DB
- Data Integration - IBM CDC, Springboot
- Data Consumers - API, GraphQL
- Data Lake
- Platform purpose - A real-time/batch data store used to consume large volumes of data from the source application across TSB. Data is formatted and prepared for consumption by various downstream consumers (including the Data Warehouse)
- Data Storage - S3, RDS (MySQL)
- Data Integration - EMR (Spark), Glue, Python
- Data Consumers - Power BI, AWB (Analytics Work Bench)
- Data Warehouse
- Platform purpose - An analytical environment to prepare structured data models and data sets for business user consumption to support reporting and analytics sourcing data from the Data Lake.
- Data Storage - Teradata, Dimensional Modelling
- Data Integration - DataStage, BTEQ, SQL
- Data Consumers - Power BI, MicroStrategy
Contact Click here to contact this recruiter or call [Telephone number removed] to apply.
Contact Name: Oliver King