Data Engineer

This job has now expired please search on the home page to find live IT Jobs.

About the Role

This is the perfect opportunity for a Data Engineer who has developed scalable big data solutions in an agile software environment.

This role involves hands on development of services focused on big data processing and availability. You'll be developing, and produce best practices for data collection, processing and storage. Also, you will have product ownership, handle and resolve issues escalated from the production environment.

Working with other collaborative teams to build, test and roll-out systems utilising big data solutions whilst actively encouraging and building up the understanding, knowledge and skills of other engineers within the company.

About You

Essential

* Expert in Scala, and Java, whilst fluent in Python or similar.

* Experience in design and architecture experience with data and stream processing and extraction technologies such as Apache Kafka, Kinesis, Spark Streaming, Apache Storm, Samza, Flume.

* Experience with production systems implementing MapReduce and utilizing lambda architectures.

* Experience with SQL/NoSQL, ElasticSearch, Hadoop/Hive and distributed data storage systems like S3 or HDFS.

* Running production systems in cloud based infrastructures (EC2, Cloudera, Databricks)

Desirable

* Experience with Big Data ML toolkits, such as SparkML, BigDL, Mahout or H2O.

* Experience with open source data pipeline and workflow tools, such as Luigi, Azkaban, Oozie, or Airflow

To find out more about Progressive Recruitment please visit our website.

Award Winner for:

Best Medium Recruitment Company of the Year by Recruitment International 2018

Training & Development Initiative of the Year by Recruitment International 2018