Top

Data Engineer

San Francisco, CA, USA

91 Days ago

Job Description


Data Engineer:

Our client is Delivering the most comprehensive identity insights, our client's platform equips businesses with fully automated KYB (Know your Business) solutions for Risk, and Fraud management, setting new standards in business verification.

The solution is designed for FS institutions to emphasize their interest in a coordinated effort to mitigate B2B fraud, reduce the risk associated with working with small businesses, and create a centralized, privacy-compliant entity for data-sharing between financial institutions.

Looking for a data engineer to assist with data collection across a variety of fragmented data sources available from both government, public, and private databases.

Responsibilities:

  • Designing the database schema
  • Migrating production tables
  • Working with large scale data in the terabyte range
  • Maintaining operational efficiency of database
  • Normalizing disparate schemas to a single unified scheme
  • Abstracting reusable components
  • Strength in minimizing the amount of new code for new pipelines and instead creating internal packages that allow a high level of reusability

Requirements

Experience with:

  • ETL (extract, transform, load)
  • Database design
  • Primary key, foreign key
  • Indexing
  • Partitioning
  • Access patterns
  • Migrations
  • Data pipelining
  • Core data concepts: ACID transactions, Idempotency, Orchestration

Technologies:

  • Airflow
  • Google Cloud Platform (GCP)
  • GCP Dataflow (aka Apache Beam)
  • PostgreSQL
  • Python
  • Pydantic (a python library)
  • Distributed systems

Qualification

Bachelor's Degree

Key Skills Required

PythonPostgreSQLApacheApache BeamB2BCore DataComprehensiveData CollectionDatabaseDatabase DesignDatabase SchemaDesignGoogle Cloud PlatformManagementOperational EfficiencyOrchestration

Job Overview


Job Function: IT/Computers - Software & Software Services

Job Type: Full Time

Workplace Type: Not Specified

Experience Level: Mid-Senior level

Salary: Competitive & Based on Experience

Experience: 0 - 0 yrs

Contact Information


Company Name: Novel Technology Services

Recruiting People: HR Department

Website: http://www.noveltechnologyservices.com

Location

Important Fraud Alert:
Beware of imposters. elsejob.com does not guarantee job offers or interviews in exchange for payment. Any requests for money under the guise of registration fees, refundable deposits, or similar claims are fraudulent. Please stay vigilant and report suspicious activity.

Similar Jobs

Senior Data Engineer

LoopMe • San Francisco, CA, USA

Experience: 4 - 5 yrs

Salary: Competitive & Based on Experience

View Job
Lead Data Engineer

LGND AI, Inc. • San Francisco, CA, USA

Experience: 10 - 11 yrs

Salary: Competitive & Based on Experience

View Job
Senior Data Engineer, 3PX Private Pricing - Analytics & Insights

Amazon • San Francisco, CA, USA

Experience: 5 - 6 yrs

Salary: Competitive & Based on Experience

View Job
Senior Data Engineer, Analytics & Insights

Amazon • San Francisco, CA, USA

Experience: 5 - 6 yrs

Salary: Competitive & Based on Experience

View Job
Big Data Engineer

Unreal Gigs • San Francisco, CA, USA

Experience: 5 - 6 yrs

Salary: Competitive & Based on Experience

View Job
Data Engineer

Unreal Gigs • San Francisco, CA, USA

Experience: 3 - 4 yrs

Salary: Competitive & Based on Experience

View Job
Data Engineer

Unreal Gigs • San Francisco, CA, USA

Experience: 3 - 4 yrs

Salary: Competitive & Based on Experience

View Job
Cloud Data Engineer

Unreal Gigs • San Francisco, CA, USA

Experience: 5 - 6 yrs

Salary: Competitive & Based on Experience

View Job