Azure Data Engineer

This job has now expired please search on the home page to find live IT Jobs.

Data EngineerWho we are:We are the leading Autonomous Decision Science™ company providing data-enabled prescriptive and anticipatory analytics and insights for companies across a broad range of industries. We automate a new way of problem-solving and decision-making, going beyond advanced analytics to understand data, perform analytics, generate insights, answer queries, and make decisions at the speed of the market. This powerful capability uniquely enables "End-to-End Value Chain Optimization and Decision-Making" at scale and allows clients to uncover and understand the inter-relationships that lead to innovative new product development and innovation, heightened consumer understanding and targeted marketing, revenue growth tactics, and intelligent demand and supply-chain planning. We help transform market-leading companies into true data-driven digital enterprises.What you will do: The successful candidate will join a diverse team to:Build unique high-impact business solutions utilizing advanced technologies for use by world class clients.Create and maintain the underlying data pipeline architecture for the solution offerings from raw client data to final solution output.Create, populate, and maintain data structures for machine learning and other analytics.Use quantitative and statistical methods to derive insights from data.Guide the data technology stack used to build our solution offerings.Combine machine learning, artificial intelligence (ontologies, inference engines and rules) and natural language processing under a holistic vision to scale and transform businesses - across multiple functions and processes.Responsibilities Include:Work with other personnel to develop and enhance commercial quality solution offerings Create and maintain optimal data pipeline architecture, incorporating data wrangling and Extract-Transform-Load (ETL) flows.Assemble large, complex data sets to meet analytical requirements - analytics tables, feature-engineering etc.Build the infrastructure required for optimal, automated extraction, transformation, and loading of data from a wide variety of data sources using SQL and other 'big data' technologies such as Databricks.Build automated analytics tools that utilize the data pipeline to derive actionable insights.Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.Design and develop data integrations and data quality frameworkDevelop appropriate testing strategies and reports for the solution as well as data from external sources.Evaluate new technology for use within the company.Work with other client personnel to administer and operate client-specific instances of our solution offeringsConfigure the data pipelines to accommodate client-specific requirements to onboard new clients.Perform regular operations tasks to ingest new and changing data - implement automation where possible.Implement processes and tools to monitor data quality - investigate and remedy any data-related issues in daily solution operations.Requirements:Bachelor's degree in Computer Science or a STEM (Science, Technology, Engineering or Math) field requiredMinimum of 3 years hands on experience as a data engineer or similar position.Minimum of 3 years commercial experience with Python or Scala Programming LanguageMinimum of 3 years SQL and experience working with relational databases (Postgres preferred).Experience with at least one of the following - Databricks, Spark, Hadoop or KafkaDemonstratable knowledge and experience developing data pipelines to automate data processing workflowsDemonstratable experience in data modeling Demonstratable knowledge of data warehousing, business intelligence, and application data integration solutionsDemonstratable experience in developing applications and services that run on a cloud infrastructure Azure preferredExcellent problem-solving and communication skills

The following additional skills would be beneficial:

Knowledge of one or more of the following technologies: Data Science, Machine Learning, Natural Language Processing, Business Intelligence, and Data Visualization.Knowledge of statistics and experience using statistical or BI packages for analyzing large datasets (Excel, R, Python, Power BI, Tableau etc.).Experience with container management and deployment, e.g., Docker and Kubernetes