Online since 1999 | 12,644 IT Jobs Live NOW

Lead Data Engineer

Premium Job From Nigel Frank International
Recruiter: Nigel Frank International
Listed on: 14th January
Location: Edinburgh
Salary/Rate: £50000 - £60000
Type: Permanent
Start Date: ASAP

This job has now expired please search on the home page to find live IT Jobs.

Lead Data Engineer - Edinburgh Salary Up to £60kLead Data Engineers across Data Platforms (Data Lake, ODS and Data Warehouse) to join Data, Analytics & Platforms function. The new function has been created by consolidating teams across the business as well as building teams to support new cloud platforms and activities transferring from their existing IT provider. Accountabilities:Design, build and test data solutions across Data Platforms (i.e. Data Lake, ODS and Data Warehouse) whilst following best practice to leverage platform processing power, creating reusable code and aligning to development standardsAnalysis and investigation of environment and source data to support incident and change management activitiesDrive continuous improvement to the data platforms through enhancement of new and existing data solutions, ongoing maintenance and troubleshooting of incidents and defectsDevelop strong working relationships with key stakeholders across business teams whilst working across the full software delivery life cycle (i.e. analysis, design, build, test and implementation) Skills / Experience:This role requires strong business and data analysis skills to support data solution and design activity.Experience working with business stakeholders to understand business requirements and to translate these into a logical solution design.Experience querying source data using SQL to identify source data items and to document transformation logic required to apply business rulesStrong background in database design and development, following best practice to leverage database processing power, create reusable code and align to data model build standards. Proven experience in data integration development that adheres to principles set out for scalability, logging, scheduling, alerts, re-runability and dependency management.Experience of delivering solutions that align with Data Governance principles for Data Lineage, Data Quality and Data Security.Familiarity of working within a formal System Development Life Cycle, particularly agile or iterative / incremental software development (i.e. Waterfall, Scrum, Kanban)Proven experience of delivering enterprise data solutions in a Financial Services environment.Technical skills for relevant Data Platform:ODSPlatform purpose - A real-time data store used to reduce processing on the mainframe and support customer facing applications through the provision of data APIsData Storage - Kafka, Syclla DBData Integration - IBM CDC, SpringbootData Consumers - API, GraphQLData LakePlatform purpose - A real-time/batch data store used to consume large volumes of data from the source application across TSB. Data is formatted and prepared for consumption by various downstream consumers (including the Data Warehouse)Data Storage - S3, RDS (MySQL)Data Integration - EMR (Spark), Glue, PythonData Consumers - Power BI, AWB (Analytics Work Bench)Data WarehousePlatform purpose - An analytical environment to prepare structured data models and data sets for business user consumption to support reporting and analytics sourcing data from the Data Lake.Data Storage - Teradata, Dimensional ModellingData Integration - DataStage, BTEQ, SQLData Consumers - Power BI, MicroStrategyContact or call 0191 814 7785 to apply.