Data Operations Engineer
Recruiter
Ofgem
Listed on
23rd September 2022
Location
Glasgow
Salary/Rate
£43650 - £59999
Salary Notes
National £43,650 - £55,999. London £46,250 - £59,999
Type
Permanent
This job has now expired please search on the home page to find live IT Jobs.
Use your data expertise to help us achieve exciting energy goals.At Ofgem we have bold Net Zero ambitions. We want to reduce the nation's reliance on fossil fuels and move quickly towards more environmentally-friendly alternatives that will provide a reliable and affordable source of energy. The recent rise in energy prices has made this goal even more pressing, and as such it is important that we become a fully digital regulator so that we can transform the energy sector. We are therefore embarking on a high-profile data and digitalisation programme, and it is within this scope of work that you could find a career-defining challenge.
Our newly-created team is working with the most cutting-edge cloud and data technologies, so this is a great opportunity to further your knowledge with the support of experienced colleagues. As such, you will help deliver our new Strategic Common Data Platform, which will connect internal and external data sources. Taking responsibility for designing and developing Azure pipelines (which will include data transformation and cleansing), you will create accessible data sets that can be easily distributed and analysed. You will also engineer manual data flows that are both scalable and repeatable, and you will document source-to-target mappings. Across all that you do, you will align your projects and datasets with Purview.
This is an exciting opportunity to network extensively with both internal colleagues and external partners. You will therefore be a natural fit in our team if you have good communication skills and you can bring a proactive mindset to identifying and resolving issues in databases, and data processes, products and services. With a background of working with Azure Data Lakes, data warehousing, and storage accounts, you will be capable of building all pipeline facets including data interfaces. Proficient in in T-SQL and Python, you will be able to transform data into appropriate storage formats, engineer data for analytics and devise error handling.
Ofgem relies on having a workforce that reflects the society we serve, so we welcome candidates from all backgrounds, and especially those from underrepresented groups. While this is a full-time role, flexible working patterns and job shares are welcome, and we offer a fully hybrid environment. We'll also support you with excellent training and development opportunities, plus a competitive pension and generous annual leave. We are hosting an Open Event for our recruitment for the Digital Delivery Team at lunchtime on the 29th Sept which you can find out more information on the roles and life at Ofgem. Link to register: https://www.eventbrite.co.uk/e/delivery-and-schemes-digital-delivery-open-recruitment-event-tickets-419568168687 Job description The Data Operations Engineer (Delivery and Schemes) is an expert engineering role to help accelerate data driven decision making, digital transformation and innovation across Ofgem through the provision of high quality, timely and valuable data and analytics. This role will focus on the Delivery and Schemes part of the Ofgem organisation which runs government schemes to promote transition to a low carbon economy and incentivise positive change. The role will be key to achieving Ofgem's vision of Digitisation and supporting the innovation and growth agenda of: Making OFGEM a recognised digitalisation leader with innovative approaches in the exploitation of data using emerging technologies Establishing the capabilities to focus on specialist areas including data analytics/insights, data innovation, artificial intelligence, data sourcing and automated data pipelines Training our workforce to be comfortable and skilled in understanding, using and asking questions of data, visualisations and analytics output Achieving cost efficiencies in targeted areas e.g. automate data preparation and processing, automate production of commonly used reporting, share data live online This role oversees the data pipelines, schedules and schemas in our modern cloud data platform. This platform includes; data ingestion, data processing, data quality improvement, master data management, data modelling, data science and data visualisation. All content has peer team leads who are either responsible for development of new content in that area or will collaborate to deliver but this role will be responsible for end to end automatic data operations once a service is live. This role will work with the Head of Data Management and Operations to ensure development of and clarity of monitoring of service, trust and value the platform delivers. There will also be an expectation in this role to suggest appropriate measures to improve: Levels of usage of data/data products e.g. user session count, user session duration Levels of value provided by each data product e.g. dashboard Levels of value provided by each underlying data set e.g. source file User trust and satisfaction e.g. net promoter scores or review feedback This role works closely with the Head of Data Management and Operations to deliver priority improvements, oversee current work and improve data platform service levels as well as react to fix or recover live production pipeline failures or service issues. Delivering improvements, ensuring service resilience and understanding the business benefit of each action will allow this role holder to clearly demonstrate individual and service value. Responsibilities Design, development and maintenance of data pipelines and improvements to them to deliver automated data refreshes for internal (e.g. databases, files, documents) and external (e.g. APIs) data sources Input into Data Management and Operations architecture diagram Input into Data Management and Operations work plan, milestones and benefit case Design and develop service level, usage and performance reporting and alerting High service levels of our Cloud Data Platform and D&S analytical systems (Data Management and Operations) input into: Strategic Data Platform Requirements Strategic Data Platform High Level Design Strategic Data Platform Detailed Design Innovation roadmap and pipeline Support and approval of IT teams that the Data Platform and Operations are fit for purpose e.g. Cyber, Security and Architecture team approvals
