Data Engineer/Architect - Cloudera, AWS, PySpark (SC Cleared)

This job has now expired please search on the home page to find live IT Jobs.

Data Engineer/Architect - Cloudera, AWS, PySpark (SC Cleared) , PySpark, Python, Spark, Amazon Web Services, AWS, Cloudera, Hive, Impala SQL, ETL, Integration, Jenkins, GIT, Version Control, JIRA, Development Lifecycle, Data Management, SC Cleared, SC Clearance, Security Cleared, Security Clearance, Government, Newport.

Our client based in Newport are looking for a Data Engineer/Architect (SC cleared) to join an existing project team and to work with an established product backlog to delivery system capability to a BAU team (responsible for publication of new/improved trade and investment data).

With the UK due to leave the European Union in 2020, the focus on trade and investment statistics has never been higher. There is a need to develop and transition new and existing processing systems. Their platform is based on both PySpark by Cloudera and Amazon Web Services technology.

Key Skills/Experience: Strong experience in development of relational database systems using the Cloudera stack (particularly PySpark, Hive and Impala SQL)Ability to produce technical documentation to support development of production systemsExperience in the development of Extract Transform and Load (ETL) pipelinesExperience developing continuous integration solutions (particularly in Jenkins)Experience in version control and software review using GITProficient in use of Jira for work item managementExperience of working across the full development lifecycleAwareness of good data security procedures and adherence to secure information management policiesAbility to work collaboratively with members in other technical roles, including Business Analysts and MethodologistsExperience of data management