Data Engineer - Kafka - Outside IR35

This job has now expired please search on the home page to find live IT Jobs.

Data Engineer - Kafka - Outside IR35 Contract Length: 6 Months (Rolling Contract Long term)Rate: Negotiable Dependent on experienceStart Date: ASAPOUTSIDE IR35 I'm looking for seasonded Data Engineer with Python expertise and commercial Kafka, Redshift and ideally airflowThe Client is continuing to develop their core data insight platforms, and this includes a change approach to move from AWS Athena and Databricks to AWS Redshift using a custom framework for data ingestion, processing, and outputs.The solution uses : Apache Kafka, Apache Airflow (workflow scheduling, dependency control and monitoring) and shifting to AWS Redshift.OverviewPython (must be expert level), AWS Redshift, Apache Kafka and Apache AirflowDesign, Build and Testing of new ingestion processes using AWS Redshift Cloud Datawarehouse, Apacke Kafka, Apache Airflow.Work in a mixed development team include the Client and other 3rd party resources.Further areas of scope : Change Data Capture (CDC) from existing Oracle databases to AWS Redshift Change Data Capture (CDC) from new Policy Management system (EIS) to AWS Redshift Capture of streaming data from price engineDeprecate existing Data Lakes in ATHENA & DATABRICKS and move to new platform Ingestion of 3rd party data sources (various) MLOps - build solution for allowing Data Scientist to productionise modelsPlease send your CV to [email protected] or call me direct on 0203 879 8405