Senior Data Engineer

Senior Data Engineer

A Senior Data Engineer is responsible for building and maintaining data pipelines, ensuring data quality and accessibility, and driving data-driven insights for various teams and projects. This involves collecting, cleaning, transforming, and managing data from diverse sources, and creating solutions for data storage, analysis, and visualization.

Here's a more detailed look at the typical responsibilities and requirements:

Experience: around 10 years

Key Responsibilities:

• Data Pipeline Development:

Designing, implementing, and maintaining data pipelines to collect, process, and transform data from various sources.

• Data Storage and Management:

Selecting and managing appropriate data storage solutions, ensuring data availability, and optimizing storage costs.

• Data Transformation and Cleaning:

Cleaning, transforming, and preparing data for analysis and visualization, ensuring data accuracy and integrity.

• Data Governance:

Implementing and enforcing data governance policies, ensuring compliance with relevant laws and regulations.

• Collaboration:

Collaborating with other engineers, product teams, and stakeholders to define data requirements and deliver solutions.

• Performance Monitoring:

Implementing performance monitoring protocols, building visualizations to monitor pipeline health, and implementing self-healing processes.

• Data Security:

Implementing and enforcing security measures to protect sensitive data.

Typical Skills and Requirements:

• Programming Languages: Strong proficiency in languages like Python, SQL
• Data Modelling: Experience in designing and implementing data models for relational and non-relational databases.
• Big Data Technologies: Familiarity with technologies like Hadoop, Spark
• Cloud Computing: Experience with cloud platforms like Azure ( Azure Data Factory , Azure Synapse Analytics, Azure Data Lake Storage ).
• ETL Tools: Experience with ETL tools like Azure Data Factory or other similar tools.
• Data Pipeline Design: Understanding of data pipeline architectures and best practices.
• Data Analysis and Visualization: Ability to perform data analysis and create visualizations to extract insights using Power BI .
• Communication and Collaboration: Excellent communication and collaboration skills to work with cross-functional teams.
• Problem-Solving: Ability to identify and solve complex data-related problems.

Requirements

• 4+ years of experience in data engineering within a production environment
• Advanced knowledge of Python and Linux shell scripting
• Experience with SQL/NoSQL databases (e.g., Redshift, Postgres, MongoDB)
• Proficiency in building stream processing systems using Kafka
• Familiarity with Docker, Kubernetes, and cloud services (AWS, GCP)
• Bonus: Experience with the ELK stack and machine learning knowledge