Data Engineer

This job has now expired please search on the home page to find live IT Jobs.

Role - Data EngineerRate - £500/Day Outside IR35Location - 2 Days in Edinburgh/3 Days RemoteDuration - 4 Months + Extension ExpectedStart - ASAP I am currently working with an Energy & Utilities organisation based in Edinburgh who are looking to secure the services of a Data Engineer - they want to move quickly with this one, so apply immediately if you're interested. The RoleWe are seeking a talented individual to help us increase our pace of delivery, managing a continuous integration and deployment tools, helping us to define infrastructure as code, and helping us with estimating system requirements, as well as monitoring smooth operations.You will work with external and internal stakeholders as well as team members when capturing and documenting IT requirements.You will show attention to detail and show strong problem solving skills when helping to spec out projects and infrastructure.You will help with the development of data pipelines, setting up data lake and data warehousing infrastructure.You will help creating new integrations with third party providers and external data sources.You will help with the development of dashboards and other reporting tools.You will be comfortable working to Agile principles and development methodologies. SkillsRequiredExcellent knowledge of PythonYou will know common data query languages for relational and document databases.Experience setting up data pipelines, data extraction, transformation and load processes.Experience working with version control tools, understand the importance of code reviews and coding standards).Experience with Messaging queues and event driven development (eg. Rabbit MQ, SQS).Experience developing REST APIs.Understanding of common cloud metrics and alerting within AWS.DesirableExperience with Apache Spark and Databricks.Experience using the AWS eco system, especially VPC, EC2, RDS and Redshift and the AWS Cloud Development Kit, Lambda.Experience working with continuous integration, CI/CD processes.Experience with orchestration tools (eg. Cloud formation, Kubernetes, etc).Experience with workflow management tools (eg. Airflow, Luigi).