Data Engineer

This role is for one of the Weekday's clients

Min Experience: 5 years

Location: Bengaluru, India, Karnataka

JobType: full-time

We are seeking a highly skilled and experienced Data Engineer to join our growing Data Engineering team. This role demands deep technical expertise in big data platforms, distributed data processing, and modern data architectures. The ideal candidate will have hands-on experience working with Hadoop , Starburst , and The Databricks Lakehouse Platform , with a strong ability to design, build, and maintain efficient and scalable data pipelines across structured and unstructured data sources.

You will play a pivotal role in driving our data infrastructure strategy, ensuring the delivery of high-quality, reliable data that supports key business decisions, analytics, and machine learning initiatives. This is a high-impact opportunity to work with a cross-functional team of engineers, data scientists, and business stakeholders in a fast-paced and data-driven environment.

Requirements

Key Responsibilities:

• Design, develop, and maintain scalable and reliable data pipelines using modern big data tools and platforms.
• Work extensively on Hadoop ecosystems to process, manage, and distribute large data sets.
• Build and optimize high-performance Starburst (Trino) queries and integrate them seamlessly with various data sources.
• Implement and manage data workflows using The Databricks Lakehouse Platform , supporting both batch and streaming data ingestion and processing.
• Collaborate with data scientists, analysts, and business users to understand data requirements and translate them into robust technical solutions.
• Develop and enforce best practices for data governance, data quality, and metadata management.
• Monitor, troubleshoot, and optimize performance of data infrastructure and ETL jobs.
• Ensure security, reliability, and compliance across all stages of the data lifecycle.

Required Skills and Qualifications:

• Minimum of 5 years of professional experience as a Data Engineer or in a similar role focused on large-scale data processing.
• Strong experience with Hadoop and its ecosystem (HDFS, MapReduce, Hive, HBase, etc.).
• Hands-on expertise with Starburst (Trino/Presto) , including query optimization and federated querying.
• Proven experience with The Databricks Lakehouse Platform , including Delta Lake, Spark, and MLflow.
• Proficiency in SQL, Python, or Scala for data transformation and pipeline development.
• Solid understanding of data modeling, warehousing concepts, and distributed computing.
• Familiarity with version control systems (e.g., Git), CI/CD pipelines, and containerization tools (e.g., Docker, Kubernetes) is a plus.
• Strong problem-solving skills and the ability to work in a fast-paced, agile environment.
• Excellent communication skills with the ability to explain complex technical topics to non-technical stakeholders.

Preferred Qualifications:

• Experience in cloud data environments (AWS, Azure, or GCP).
• Exposure to data governance and privacy frameworks (GDPR, CCPA, etc.).
• Prior experience supporting data science and advanced analytics projects.