Data engineer

This job has now expired please search on the home page to find live IT Jobs.

Job DescriptionI am working with a fast growing a hugely exciting AI & ML agency who are recruiting for a data engineer who will be working on our data pipeline architecture, with the aim of providing clean, usable data to Business Analysts and Data Scientists. You will be responsible for helping to build modular pipeline components and ensure technical documentation is created and maintained.

Role & Responsibilities Helping design, build, maintain and operate the data pipeline.Defining and building modular data pipeline components.Ensuring that solid development practices, such as proper use of source control, full testing processes and automated deployment mechanisms, are followed.Collaborate with data scientists and business analysts to discover where business value can be found within the data available.Maintaining existing systems and supporting migration to new data pipeline architecture.Training Client Analytics and Implementation teams in how to implement and support clients.Acting as a subject matter expert on all aspects of the data pipeline.Identifying potential performance issues, bottlenecks and pain points and recommend new and creative ways of resolving them.

Skills & QualificationsDistributed data processing, for example SparkNoSQL Databases, such as MongoDB or CouchbaseCloud computing platforms, such as Google Cloud Platform or AWSPipeline orchestration, for example AirflowTechnical understanding of infrastructure components, their dependencies, and interactions between servers, virtual systems, networks, databases, web applications, etcExperience3+ years of proven experience using Python to build data pipelines, including familiarity with python's core big data / data science libraries: e.g. pandas, pyspark, scikit-learn etcSolid understanding of database design and SQLExperience working in cross functional agile teams, particularly teams including Data Scientists, Software Engineers and Business AnalystsThe ability to communicate complicated technical solutions to non technical usersTake ownership of feature development and ongoing maintenanceTechnical understanding of infrastructure components, their dependencies, and interactions between servers, virtual systems, networks, databases, web applications, etc