Data Engineer

Position: Data Engineer (Sr. Analyst)

JOB SUMMARY

The Data Engineer will have responsibility to use various methods to transform raw data into useful

data systems and should have strong analytical skills and the ability to combine data from different

sources. The person must have familiarity with several programming languages and knowledge of

learning machine methods.

ESSENTIAL JOB FUNCTIONS

The primary responsibilities of this position include:

Development of high-quality code (efficient and maintainable) in Python, PySpark, SQL to

ingest, process and transform large datasets

Design and implement scalable data pipelines using ETL and ELT processes in Azure Data

Factory, Databricks and Snowflake

Work closely with cross-functional teams to understand data requirements and deliver

solutions that meet business needs

Design, develop and optimize data models in Databricks and Snowflake environments

Ensure data quality and integrity by establishing and enforcing data modelling standards

and best practices

Support and optimize data tools and associated cloud environments for consumption by

downstream systems, data analysts and data scientists

Ensure code, configuration and other technology artifacts are delivered within agreed time

schedules and any potential delays are escalated in advance

Collaborate across developers as part of a SCRUM team ensuring collective team

productivity

Implement CI/CD pipelines and DevOps practices using Azure DevOps

Ensure that 100% of code is well documented and maintained in source code repository

TECHNICAL SKILLS AND EXPERIENCE

5-7 years professional experience as a Data engineer/Data Modeler

Strong experience working with Microsoft Azure Data platform services like: Azure Data Lake

Gen2, Azure Data Factory, Azure Databricks

Advanced in building ETL and data pipelines on Databricks using data engineering

languages like Python, PySpark.

Hands-on experience with Snowflake

Proficiency in SQL

Proven experience with all aspects of the ETL processes (Data Sourcing, Transformations,

Data Quality, etc.?)

Familiarity with Azure DevOps and CI/CD pipelines

Advanced knowledge of data warehousing concepts and best practices

Experience with relational and dimensional database modelling (Relational, Kimball, Data

Vault, Star Schemas)

NON-TECHNICAL SKILLS AND EXPERIENCE

Creative thinking, problem solving and decision making. Proactive and self-starter.

Has high degree of initiative, dependability and ability to work with little supervision.

Responds to and handles unfamiliar situations with little supervisory guidance.

Ability to solve problems that are complex, varied and only mildly related to those seen

before. Simply determining what the problem is requires major individual effort and/or

consultation with others within the department.

A high degree of analytical ability and inductive thinking may be required to solve highly

intricate, technically complex problems. Must be able to develop new and nonstandard

approaches.

Ability to complete work that requires developing imaginative and complex methods,

procedures, products or systems.

Strong understanding of the Business Intelligence lifecycle, from requirements gathering to

development and acceptance.

Has excellent written and verbal communication skills, interpersonal and collaborative skills,

and the ability to communicate technical concepts to both technical and nontechnical

audiences.

Requires regular external contacts to carry out organization programs and to explain

specialized matters. Position requires contacts with customers or the public involving persons

of some importance on matters requiring explanation and discussion and where the effort to

influence becomes important.

Strong communication and interpersonal skills to work within a team environment.

Excellent organizational and troubleshooting skills with attention to detail.

Strong presentation and documentation skills.

Must have high proficiency in spoken and written English.

EDUCATION

Bachelor's degree required in Mathematics, Statistics, Computer Sciences, or related field.

Qualification

Bachelor's Degree