Specialist - Data Quality Engineer

Specialist-Data Quality Engineer

Date: Feb 11, 2025

Location: Gurgaon, HR, IN, 122002

Company: Corning

Requisition Number: 66941

Corning is vital to progress ? in the industries we help shape and in the world we share.

We invent life-changing technologies using materials science. Our scientific and manufacturing expertise, boundless curiosity, and commitment to purposeful invention place us at the center of the way the world interacts, works, learns, and lives.

Our sustained investment in research, development, and invention means we're always ready to solve the toughest challenges alongside our customers.

The Global Supply Management (GSM) Function is recognized as a critical function for business success.

It delivers the training, tools and opportunities needed to create innovative solutions for the function & the corporation.

GSM has leading edge, effective processes to anticipate and exceed customer requirements.

As a Data Engineer at you'll be taking the lead in designing and maintaining complex data ecosystems. Your experience will be instrumental in optimizing data processes, ensuring data quality, and driving data-driven decision-making within the Global Supply Chain (GSM) in Corning.

What will you be doing?

Design, develop, and optimize data workflows and notebooks using Databricks to ingest, transform, and load data from various sources into the data lake.

Create, Develop & Implement Data Quality rules within Databricks and connect?execute?Sync. Data visualization tool, i.e. Power BI.

Build and maintain scalable and efficient data processing workflows using Spark (PySpark or Spark SQL) by following coding standards and best practices.

Utilize Unity Catalog(databricks) to scan End to End Data pipelines i.e. from Data Source to Data Consumption and establish Data Mapping as well as Data lineage.

Proficiency in designing and building complex data pipelines and data processing systems.

Develop data models and schemas to support reporting and analytics needs.

Ensure data quality, integrity, and security by implementing appropriate checks and controls.

Developing and maintaining data pipelines for efficient data extraction, transformation, and loading (ETL) processes.

Ensuring data quality and integrity through data validation, cleansing, and error handling.

Develop Data Quality Rules using SQL/Python and integrate with Power BI for reporting and data visualization.

Utilize Alation for data catalog management, standardizing data terminology, and promoting a common understanding of data concepts.

Data Catalog Management: Utilize Alation to maintain a comprehensive data catalog, ensuring that data assets are accurately described, categorized, and searchable.

Data Lineage and Profiling: Create and manage data lineage diagrams using Alation to track the flow of data throughout our organization. Perform data profiling to identify data quality issues.

Data Glossary Management: Maintain and expand the data glossary in Alation, standardizing data terminology and promoting a common understanding of data concepts.

Develop and manage data architectures in line with business needs, considering scalability, reliability, and security.

Use front-end development tools to design intuitive user interfaces for data visualization.

Write, test, and debug code in Python, SQL, and JavaScript, following best coding practices.

Manage database operations and optimization using various database tools.

Leverage AWS services for data storage, processing, and analytics tasks.

Who will you be interacting with?

IT solution architects

Business DMOs

GSM and Shared Services global process owners

Power users & super users

What education and experience you will bring?

Bachelor's Degree required, ideally in Computer Science, Engineering, Supply Chain Management

Minimum of 8 years in leading data-related activities with direct continuous and process improvement experience

Minimum of 3 years of manufacturing/operations experience in Supply Chain areas ? Plan, Make, Source, Deliver

Minimum of 8 years of experience in Data Engineering, Data Quality, Metadata Management along with various Database tools and AWS services

Previous experience and knowledge of PeopleSoft data tables, SAP MDG/S4 & ECC data tables

Key Skillsets Required:

Proven experience as a Data Engineer or similar role.

Strong knowledge of Databricks, Alation, and Power BI.

Extensive experience with data quality checks, data architecture, and front-end development tools.

Proficiency in Python, SQL, Scala and JavaScript.

Experience with various database tools cloud platforms like AWS,Azure,GCP.

Experience with ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes.

Product development experience for developing in house data quality tool along with proficiency for various data quality tools like attacama, syniti.

Experience with various frontend tools Angular, React along with various data visualization tools like Power BI, Thoughtspot,D3.JS

Experience for working on various API tools like Postman, Apache JMeter etc.

Extensive knowledge and experience with big data tools such as Kafka, Spark, and Hadoop.

Familiarity with databases such as Cassandra and Postgres.

Knowledge of pipeline tools like Airflow, Luigi, and Azkaban.

Proficiency with AWS cloud services like Redshift, RDS, EMR, and EC2.

Experience with stream-processing systems like Spark-Streaming and Storm.

Strong knowledge of Databricks and Alation is a must.

Excellent problem-solving abilities and attention to detail.

Strong communication and collaboration skills.

What skills you need to demonstrate?

Demonstrated track record in project management abilities

Bias for action ? ability to simplify complex situations guiding teams to deliver results

Knowledge of technologies, techniques and best practices in data governance, master data administration and systems implementations data-related activities

Strong verbal and written communication skills to share project progress, scope, and risks

Agile, flexible, and a change champion

Ability to define and drive global standards across a portfolio of cross-functional teams, leading to consistency of operation and execution, maximizing efficiency and business performance

Problem solving, analytical, and exceptional follow through skills

Comfortable influencing without authority

Collaborative in an environment with multiple stakeholders across different organizations

Travel Requirements (domestic and/or international)