Job Description
Leads projects for the design, development, and maintenance of a data and analytics platform. Ensures efficient processing, storage, and availability of data for analysts and other consumers. Collaborates with key business stakeholders, IT experts, and subject-matter experts to plan, design, and deliver optimal analytics and data science solutions. Works on one or multiple product teams simultaneously.
Note:- Even though the role is categorized as Remote, it will follow a hybrid work model.
Key Responsibilities:
Design and automate the deployment of distributed systems for ingesting and transforming data from various sources (relational, event-based, unstructured).
Develop frameworks for continuous monitoring and troubleshooting of data quality and integrity issues.
Implement data governance processes for metadata management, data access, and retention policies for internal and external users.
Provide guidance on building reliable, efficient, scalable, and quality data pipelines with monitoring and alert mechanisms using ETL/ELT tools or scripting languages.
Design and implement physical data models to define database structures and optimize performance through indexing and table relationships.
Optimize, test, and troubleshoot data pipelines.
Develop and manage large-scale data storage and processing solutions using distributed and cloud-based platforms such as Data Lakes, Hadoop, Hbase, Cassandra, MongoDB, Accumulo, DynamoDB, and others.
Utilize modern tools and architectures to automate common, repeatable, and tedious data preparation and integration tasks.
Drive automation in data integration and management by renovating the data management infrastructure.
Ensure the success of critical analytics initiatives by employing agile development methodologies such as DevOps, Scrum, and Kanban.
Coach and mentor less experienced team members.
RESPONSIBILITIES
Technical Skills:
Expert-level proficiency in Spark, including optimization, debugging, and troubleshooting Spark jobs.
Solid knowledge of Azure Databricks for scalable, distributed data processing.
Strong coding skills in Python and Scala for data processing.
Experience with SQL, especially for large datasets.
Knowledge of data formats such as Iceberg, Parquet, ORC, and Delta Lake.
Experience developing CI/CD processes.
Deep understanding of Azure Data Services (e.g., Azure Blob Storage, Azure Data Lake, Azure SQL Data Warehouse, Synapse Analytics, etc.).
Familiarity with data lakes, data warehouses, and modern data architectures.
Competencies:
System Requirements Engineering
Translates stakeholder needs into verifiable requirements, establishing acceptance criteria and assessing the impact of requirement changes.
Collaborates
Builds partnerships and works collaboratively with others to meet shared objectives.
Communicates effectively
Develops and delivers clear, audience-specific communications.
Customer focus
Builds strong customer relationships and delivers customer-centric solutions.
Decision quality
Makes timely and informed decisions to keep the organization moving forward.
Data Extraction
Performs ETL activities from various sources using appropriate tools and technologies.
Programming
Develops, tests, and maintains computer code and scripts to meet business and compliance requirements.
Quality Assurance Metrics
Uses IT Operating Model (ITOM) and SDLC standards to assess solution quality.
Solution Documentation
Documents solutions for improved productivity and knowledge transfer.
Solution Validation Testing
Ensures configuration changes and solutions meet customer requirements.
Data Quality
Identifies, understands, and corrects data flaws to enhance information governance.
Problem Solving
Uses systematic analysis to identify root causes and implement robust solutions.
Values differences
Recognizes and appreciates diverse perspectives and cultures.
QUALIFICATIONS
Education, Licenses, Certifications:
Bachelor's or master?s degree in Computer Science, Information Technology, Engineering, or a related field.
Experience:
8+ years of experience in data engineering or a related field, with experience in a leadership role.
Intermediate experience in relevant disciplines is required.
Knowledge of the latest data engineering technologies and trends is preferred, including:
Analyzing complex business systems, industry requirements, and data regulations.
Processing and managing large datasets.
Designing and developing Big Data platforms using open-source and third-party tools.
SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka or equivalent.
SQL query language.
Cloud-based clustered compute implementation.
Developing applications requiring large file movement in a cloud-based environment.
Building analytical solutions.
Intermediate experience in the following is preferred:
IoT technology.
Agile software development.
Job
Systems/Information Technology
Organization
Cummins Inc.
Role Category: Remote
Job Type: Exempt - Experienced
ReqID: 2411156
Relocation Package
No
Note:- Even though the role is categorized as Remote, it will follow a hybrid work model.
Key Responsibilities:
Design and automate the deployment of distributed systems for ingesting and transforming data from various sources (relational, event-based, unstructured).
Develop frameworks for continuous monitoring and troubleshooting of data quality and integrity issues.
Implement data governance processes for metadata management, data access, and retention policies for internal and external users.
Provide guidance on building reliable, efficient, scalable, and quality data pipelines with monitoring and alert mechanisms using ETL/ELT tools or scripting languages.
Design and implement physical data models to define database structures and optimize performance through indexing and table relationships.
Optimize, test, and troubleshoot data pipelines.
Develop and manage large-scale data storage and processing solutions using distributed and cloud-based platforms such as Data Lakes, Hadoop, Hbase, Cassandra, MongoDB, Accumulo, DynamoDB, and others.
Utilize modern tools and architectures to automate common, repeatable, and tedious data preparation and integration tasks.
Drive automation in data integration and management by renovating the data management infrastructure.
Ensure the success of critical analytics initiatives by employing agile development methodologies such as DevOps, Scrum, and Kanban.
Coach and mentor less experienced team members.
RESPONSIBILITIES
Technical Skills:
Expert-level proficiency in Spark, including optimization, debugging, and troubleshooting Spark jobs.
Solid knowledge of Azure Databricks for scalable, distributed data processing.
Strong coding skills in Python and Scala for data processing.
Experience with SQL, especially for large datasets.
Knowledge of data formats such as Iceberg, Parquet, ORC, and Delta Lake.
Experience developing CI/CD processes.
Deep understanding of Azure Data Services (e.g., Azure Blob Storage, Azure Data Lake, Azure SQL Data Warehouse, Synapse Analytics, etc.).
Familiarity with data lakes, data warehouses, and modern data architectures.
Competencies:
System Requirements Engineering
Translates stakeholder needs into verifiable requirements, establishing acceptance criteria and assessing the impact of requirement changes.
Collaborates
Builds partnerships and works collaboratively with others to meet shared objectives.
Communicates effectively
Develops and delivers clear, audience-specific communications.
Customer focus
Builds strong customer relationships and delivers customer-centric solutions.
Decision quality
Makes timely and informed decisions to keep the organization moving forward.
Data Extraction
Performs ETL activities from various sources using appropriate tools and technologies.
Programming
Develops, tests, and maintains computer code and scripts to meet business and compliance requirements.
Quality Assurance Metrics
Uses IT Operating Model (ITOM) and SDLC standards to assess solution quality.
Solution Documentation
Documents solutions for improved productivity and knowledge transfer.
Solution Validation Testing
Ensures configuration changes and solutions meet customer requirements.
Data Quality
Identifies, understands, and corrects data flaws to enhance information governance.
Problem Solving
Uses systematic analysis to identify root causes and implement robust solutions.
Values differences
Recognizes and appreciates diverse perspectives and cultures.
QUALIFICATIONS
Education, Licenses, Certifications:
Bachelor's or master?s degree in Computer Science, Information Technology, Engineering, or a related field.
Experience:
8+ years of experience in data engineering or a related field, with experience in a leadership role.
Intermediate experience in relevant disciplines is required.
Knowledge of the latest data engineering technologies and trends is preferred, including:
Analyzing complex business systems, industry requirements, and data regulations.
Processing and managing large datasets.
Designing and developing Big Data platforms using open-source and third-party tools.
SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka or equivalent.
SQL query language.
Cloud-based clustered compute implementation.
Developing applications requiring large file movement in a cloud-based environment.
Building analytical solutions.
Intermediate experience in the following is preferred:
IoT technology.
Agile software development.
Job
Systems/Information Technology
Organization
Cummins Inc.
Role Category: Remote
Job Type: Exempt - Experienced
ReqID: 2411156
Relocation Package
No
