Senior Data Engineer
Recruiter
Listed on
Location
Salary/Rate
Type
Start Date
This job has now expired please search on the home page to find live IT Jobs.
What job we expect to do:
We are looking for a GCP Lead Engineer to join our Data Insights and Analytics team. The role will be responsible for building new data pipelines, maintaining existing ETL pipelines, and optimizing data flows using the GCP cloud stack.
The ideal candidate will be an experienced resource with strong experience in building data pipelines, data products and platforms from scratch as well as maintaining and fine tuning existing setups. The data engineer will need to support Business Analysts, Technical team members and other stakeholders.
He/she must be self-directed and comfortable supporting the data needs of multiple teams, systems and products.
Skills we expect to bring::
- AdvancedGCP knowledge, experience working on and migrating data products from on-prem to GCP
- Experienceof leading teams to deliver projects
- Experiencebuilding and optimizing 'big data' data pipelines, architectures and data sets using GCP services
- Experiencebuilding real-time data pipelines including unstructured datasets
- Build processes supporting data transformation, data structures as well ETL, cost, space optimization
- Strong experience in Data Ingestion and Storage including BigQuery, GCS, Dataflow, Datastream, Dataform, Airflow
- Experience with Big data Tools: Hadoop Spark, Kafka
- Experience with Stream processing systems: Spark-streaming, Kafka
- Strong problem solving, quantitative and analytical abilities
- Excellent communication, collaboration and delegation
What Tools and Technologies do we expect you to know? We understand one cannot be master of all.
- SQL using Teradata, DB2, Oracle, etc.
- Cloud native data warehouse like BigQuery
- Python programming including Pandas, Numpy
- Good experience of ETL, Data Modelling, Data Warehousing, etc.
- GCP - BigQuery, GCS, Dataflow, Datastream, Dataform, Airflow
