Azure Technical Architect - HDInsight

This job has now expired please search on the home page to find live IT Jobs.

Insights Technical Architect

Duration: 6 Months

Location: London (Westminster)

Project Description

Delivering a new data platform to support reporting and data analysis within the organisation. The platform requires the ingestion of data from numerous sources, both old and new, and the provision of tooling to allow authorised users to perform their work on the data.

Role Description

The data platform will be based on a data lake architecture, hosted on the Azure cloud. We need to engage an HDInsight Technical Architect to help lead the development and deployment of schema-on-read data modelling tools within the platform. The candidate should have hands-on experience of developing with Apache Hive and Apache Spark on Azure HDInsight and a good awareness of best practice and anti-patterns in this field. We also need someone who can act as evangelist for the technology and share knowledge across the data teams.

The main activities for the role are:

* Design and develop data models in Apache Hive on HDInsight to make semi-structured data available to existing ETL processes.

* Work with the Cloud Infrastructure team to build HDInsight deployment processes based on secure and performant configurations.

* Work with data scientist community to build simple data models in Apache Spark on HDInsight to provide access to semi-structured data for R developers.

* Provide input to the design of the Data Platform architecture in order to get most benefit from the use of HDInsight.

* Work collaboratively with the in-house data management teams to educate and evangelise on the use of schema-on-read data modelling patterns.

Skills

Must have:

* Azure HDInsight

* Apache Hive on HDInsight

* Apache Spark on HDInsight

* Experience of working with semi-structured data, in particular JSON.

Useful:

* Familiarity/experience of R statistical programming.

* Experience of working with Azure CI pipelines, such as Jenkins, Terraform

General Qualities

The role requires someone who is knowledgeable and enthusiastic about the technology and who is both capable and happy to share his knowledge with others. Good communication skills are a must.

Suitable candidates should submit their CV in the first instance