Role: Java Developer with Web Crawler Experience
Location: Austin TX(Hybrid)
Responsibilities:
1.Web Crawler Development: Design and implement efficient and scalable web crawlers in Java to collect data from various online sources.
2.Data Extraction: Develop and maintain systems for structured data extraction, handling various data formats (HTML, JSON, XML, etc.).
3.Data Storage and Processing: Design data storage and processing pipelines, ensuring extracted data is clean, structured, and easily accessible.
4.Performance Optimization: Optimize web crawling processes for speed, efficiency, and accuracy, while ensuring minimal impact on source websites.
5.Error Handling and Logging: Implement error-handling mechanisms and logging systems to detect and resolve issues during crawling operations.
6.Data Integrity and Compliance: Ensure data collection practices are ethical, legal, and compliant with relevant regulations (e.g., robots.txt, copyright laws).
Requirements:
Proficiency in Java and experience with Java-based web scraping libraries (e.g., Jsoup, Apache HttpClient).
Knowledge of web crawling frameworks and tools, such as Scrapy, Selenium, or Puppeteer.
Strong understanding of HTML, CSS, JavaScript, and web data structures.
Familiarity with data parsing and handling techniques for JSON, XML, and other common formats.
Experience with database technologies (SQL, NoSQL) to store and manage scraped data.
Knowledge of HTTP protocols, headers, proxies, and load handling.
Job Function: IT/Computers - Software & Software Services
Job Type: Full Time
Workplace Type: Not Specified
Experience Level: Not Specified
Salary: Competitive & Based on Experience
Experience: 0 - 0 yrs
Important Fraud Alert:
Beware of imposters. elsejob.com does not guarantee job offers or interviews in exchange for payment. Any requests for money under the guise of registration fees, refundable deposits, or similar claims are fraudulent. Please stay vigilant and report suspicious activity.
Sun-itsolutions inc • Austin, TX, USA
Experience: 6 - 7 yrs
Salary: Competitive & Based on Experience
View JobDispel • Austin, TX, USA
Salary: $57,000 - $72,000 / Annual Salary
View JobDomainTools • Austin, TX, USA
Experience: 7 - 8 yrs
Salary: Competitive & Based on Experience
View JobPushnami • Austin, TX, USA
Experience: 7 - 8 yrs
Salary: Competitive & Based on Experience
View JobWipro Technologies • Austin, TX, USA
Experience: 5 - 6 yrs
Salary: $95,000 - $100,000 / Annual Salary
View JobDataVisor • Austin, TX, USA
Experience: 3 - 4 yrs
Salary: Competitive & Based on Experience
View JobiSoftTek Solutions Inc • Austin, TX, USA
Salary: Competitive & Based on Experience
View JobGCS Technologies • Austin, TX, USA
Experience: 5 - 6 yrs
Salary: Competitive & Based on Experience
View JobMancomm, Inc. • Austin, TX, USA
Salary: Competitive & Based on Experience
View Job