Top

Big Data Engineer

Austin, TX, USA

265 Days ago

Job Description


Are you passionate about handling massive datasets and building the infrastructure that enables complex data analysis and machine learning at scale? Do you excel in creating robust, scalable data pipelines that fuel data-driven decision-making? If you're ready to tackle the challenges of big data, our client has the perfect role for you. We're seeking a Big Data Engineer (aka The Data Pipeline Innovator) to architect and maintain high-performance data systems that empower analytics and support advanced data processing needs.

As a Big Data Engineer at our client , you'll collaborate with data scientists, analysts, and software engineers to design, implement, and optimize big data platforms. Your expertise in data engineering, distributed systems, and cloud infrastructure will be critical to ensuring that our data ecosystem is efficient, reliable, and scalable.

Key Responsibilities:

  1. Design and Build Scalable Data Pipelines:
    • Architect and implement data pipelines for ETL processes using tools like Apache Spark, Kafka, and Hadoop. You'll create data workflows that handle high-volume, high-velocity data and ensure seamless integration across systems.
  2. Optimize Big Data Storage and Processing:
    • Develop and manage data storage solutions (e.g., HDFS, S3, Cassandra) that are optimized for performance and cost-efficiency. You'll configure distributed processing systems to support efficient data retrieval and transformation.
  3. Collaborate on Data Strategy and Integration:
    • Work closely with data scientists, analysts, and other engineers to align big data architecture with analytics goals. You'll ensure data availability and integrity across systems to support business objectives.
  4. Implement Data Quality and Governance Standards:
    • Develop processes and tools to monitor data quality and enforce data governance policies. You'll ensure data is accurate, reliable, and secure through regular checks and validation processes.
  5. Enhance Data Processing with Automation:
    • Use tools like Apache Airflow or AWS Glue to automate data workflows and reduce manual processing. You'll implement scripts and automation that streamline data handling and improve efficiency.
  6. Monitor and Troubleshoot Data Systems:
    • Use monitoring tools to track system performance and address issues proactively. You'll troubleshoot and resolve any bottlenecks or failures to maintain optimal data processing capabilities.
  7. Stay Updated on Big Data Trends and Technologies:
    • Keep up with advancements in big data technologies and tools. You'll integrate new techniques and platforms that align with business needs and promote innovation.

Requirements

Required Skills:

  • Big Data Platform Proficiency: Extensive experience with big data technologies such as Apache Spark, Hadoop, Kafka, and Hive. You're skilled at handling high-volume data and distributed processing.
  • Data Pipeline and ETL Knowledge: Proven ability to design, build, and maintain ETL processes for massive datasets. You can handle both real-time and batch data processing requirements.
  • Programming and Scripting: Proficiency in programming languages like Python, Java, or Scala for data processing and automation. Experience with SQL for data querying and manipulation is essential.
  • Cloud Data Services Expertise: Familiarity with cloud platforms such as AWS, GCP, or Azure, including their big data and storage services (e.g., S3, BigQuery, Azure Data Lake).
  • Data Quality and Governance: Strong understanding of data quality standards and governance practices, with experience in implementing data validation and monitoring frameworks.

Educational Requirements:

  • Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related field. Equivalent experience in data engineering or big data management may be considered.
  • Certifications in big data or cloud technologies (e.g., Cloudera Certified Data Engineer, AWS Certified Big DataSpecialty, Google Professional Data Engineer) are a plus.

Experience Requirements:

  • 5+ years of experience in data engineering, with at least 3+ years focusing on big data technologies and high-scale data environments .
  • Experience in distributed systems and large-scale data storage management.
  • Familiarity with containerization (Docker, Kubernetes) for deploying data processing environments is advantageous.

Benefits

  • Health and Wellness: Comprehensive medical, dental, and vision insurance plans with low co-pays and premiums.
  • Paid Time Off: Competitive vacation, sick leave, and 20 paid holidays per year.
  • Work-Life Balance: Flexible work schedules and telecommuting options.
  • Professional Development: Opportunities for training, certification reimbursement, and career advancement programs.
  • Wellness Programs: Access to wellness programs, including gym memberships, health screenings, and mental health resources.
  • Life and Disability Insurance: Life insurance and short-term/long-term disability coverage.
  • Employee Assistance Program (EAP): Confidential counseling and support services for personal and professional challenges.
  • Tuition Reimbursement: Financial assistance for continuing education and professional development.
  • Community Engagement: Opportunities to participate in community service and volunteer activities.
  • Recognition Programs: Employee recognition programs to celebrate achievements and milestones.

Key Skills Required

JavaSQLArchitecturePythonAWSAutomationApacheApache AirflowAWS GlueAzureBatch Data ProcessingClouderaContainerizationData AnalysisData ArchitectureData EngineeringData GovernanceData LakeData ProcessingAnalysisAnalyticsApache SparkBig DataBig Data ArchitectureBig Data ManagementBig Data StorageBigQueryBusiness ObjectivesCloud InfrastructureCommunity EngagementComprehensiveComputer ScienceCounselingData AvailabilityData EcosystemData HandlingData ManagementData PipelinesData QualityData ServicesData StorageData Storage SolutionsData StrategyData ValidationDesignDevelopmentDisability InsuranceEcosystemEmployee RecognitionFinancial AssistanceGovernanceInfrastructureInnovationInsuranceIntegrationKubernetesLearningLife InsuranceMachine LearningManagementManual ProcessingMental HealthPassionatePerformance DataPipelineProfessional DevelopmentProven AbilityQuality StandardsReimbursementScienceSeamless IntegrationSPARKStrategySupport ServicesTelecommutingTrainingValidationVision InsuranceWellness ProgramsWork-Life Balance

Job Overview


Job Function: IT/Computers - Software & Software Services

Job Type: Full Time

Workplace Type: Remote

Experience Level: Mid-Senior level

Salary: Competitive & Based on Experience

Experience: 5 - 6 yrs

Contact Information


Company about us:

Unreal Gigs is a leading global Employment Expert team that specializes in providing innovative solutions for securing top talent. With a deep understanding of the challenges faced in today's competitive job market, we are dedicated to helping individuals and organizations achieve their full potential.

At Unreal Gigs, we pride ourselves on...

Company Name: Unreal Gigs

Recruiting People: HR Department

Website: https://www.unrealstaffing.com/

Location

Important Fraud Alert:
Beware of imposters. elsejob.com does not guarantee job offers or interviews in exchange for payment. Any requests for money under the guise of registration fees, refundable deposits, or similar claims are fraudulent. Please stay vigilant and report suspicious activity.

Similar Jobs

Oracle Database Administrator

Sun-itsolutions inc • Austin, TX, USA

Experience: 6 - 7 yrs

Salary: Competitive & Based on Experience

View Job
Quality Assurance Engineer

Dispel • Austin, TX, USA

Salary: $57,000 - $72,000 / Annual Salary

View Job
Senior Platform Engineer

DomainTools • Austin, TX, USA

Experience: 7 - 8 yrs

Salary: Competitive & Based on Experience

View Job
Senior Software Development Engineer

Pushnami • Austin, TX, USA

Experience: 7 - 8 yrs

Salary: Competitive & Based on Experience

View Job
Salesforce Developer

Wipro Technologies • Austin, TX, USA

Experience: 5 - 6 yrs

Salary: $95,000 - $100,000 / Annual Salary

View Job
Security Engineer

DataVisor • Austin, TX, USA

Experience: 3 - 4 yrs

Salary: Competitive & Based on Experience

View Job
Salesforce Developer

iSoftTek Solutions Inc • Austin, TX, USA

Salary: Competitive & Based on Experience

View Job
Senior Technical Support Analyst (Tier 3)

GCS Technologies • Austin, TX, USA

Experience: 5 - 6 yrs

Salary: Competitive & Based on Experience

View Job
Platform Engineer

Vortexa • Austin, TX, USA

Salary: Competitive & Based on Experience

View Job
Full Stack Platform Engineer

Mancomm, Inc. • Austin, TX, USA

Salary: Competitive & Based on Experience

View Job