Data Engineer-Perm

This job has now expired please search on the home page to find live IT Jobs.

Data Engineer- Full-time/Remote Company Overview:We provide solutions that leverage sophisticated machine learning, artificial intelligence, and natural language processing to provide highly actionable insights and recommendations to business users. Our solutions impact just about every aspect of the products you buy at your local store - from what is available to how its priced and even where it is placed on the shelf. Our SaaS solutions are deployed within private clouds - principally on Azure. What you will do: Build unique high-impact business solutions utilizing advanced technologies for use by world class clients.Create and maintain the underlying data pipeline architecture for the solution offerings from raw client data to

final solution output.Create, populate, and maintain data structures for machine learning and other analytics.Use quantitative and statistical methods to derive insights from data.Guide the data technology stack used to build our solution offerings.Combine machine learning, artificial intelligence (ontologies, inference engines and rules) and natural language processing under a holistic vision to scale and transform businesses. Responsibilities: Create and maintain optimal data pipeline architecture, incorporating data wrangling and ETL flows.Assemble large, complex data sets to meet analytical requirements - analytics tables, feature-engineering.Build the infrastructure required for optimal, automated extraction, transformation, and loading of datafrom a wide variety of data sources using SQL and other 'big data' technologies such as Databricks.Build automated analytics tools that utilize the data pipeline to derive actionable insights.Identify, design, and implement internal process improvements: automating manual processes,optimizing data delivery, re-designing infrastructure for greater scalability, etc.Design and develop data integrations and data quality framework.Develop appropriate testing strategies and reports for the solution as well as data from external sources.Evaluate new technology for use within the organization.Work with other client personnel to administer and operate client-specific instances of our solution offerings:Configure data pipelines to accommodate client-specific requirements to onboard new clients.Perform regular operations tasks to ingest new and changing data - implement automation where possible.Implement processes and tools to monitor data quality - investigate and remedy any data-related issues in daily solution operations. Requirements:Minimum of a bachelor's degree in Computer Science or related field3+ years hands on experience as a data engineer or similar position3+ years of commercial experience with Python or Scala Programming Language3+ years of SQL and experience working with relational databasesKnowledge of at least one of the following - Databricks, Spark, Hadoop or KafkaDemonstratable knowledge and experience developing data pipelines to automate data processing work flowsDemonstratable experience in data modellingDemonstratable knowledge of data warehousing, business intelligence, and application data integration solutionsDemonstratable experience in developing applications and services that run on a cloud infrastructure (Azure

preferred)Excellent problem-solving and communication skills If you are interested or would like to learn more, contact Emily today at [email protected]