We are seeking a contract web scraping and data mining developer to reliably collect and structure data from a wide range of public online sources. The ideal candidate delivers work quickly, prioritizes accuracy, and can independently take a source from initial assessment to clean, usable data. Responsibilities: * Build and maintain scrapers and crawlers to extract structured data from websites, APIs, and public records, including sites with pagination, dynamic JavaScript content, and anti-bot measures. * Work with a variety of source types and formats (HTML, JSON, PDFs, scanned * documents, non-English pages) and deliver clean, well-structured output (CSV/JSON, or written directly into our pipelines and databases). * Develop reusable scripts and utilities to automate recurring collection and monitoring tasks. * Diagnose and repair scrapers when source websites change, and adapt to new sources as needed. * Document all work clearly so that other members of the team can run and maintain it.
Required skills and experience * Strong proficiency in Python, with hands-on experience using Scrapy, Selenium (or Playwright), BeautifulSoup, and requests. * Solid experience working with APIs (REST, authentication, rate limits) and SQL. * Demonstrated ability to address common anti-scraping challenges, including rotating proxies, headers and session management, CAPTCHAs, and rate limiting. * Experience handling large volumes of unstructured or inconsistent data, with a consistent record of accurate, verified results. * Ability to work independently through the full development cycle: scoping a source, building the solution, and delivering usable data. * Clear communication and dependable turnaround.
Preferred qualifications * Additional programming languages (Go, Perl, Java, JavaScript) and reverse-engineering experience for complex sources. * Experience with non-English-language sources and OCR. * Familiarity with cloud- and queue-based scraping architectures (e.g., AWS, containerized workers, commercial proxy services). * Experience with public records, corporate registries, court filings, or similar structured and semi-structured sources. * Interest in supporting investigative, transparency, or public-interest work.
Attributes of a strong candidate * A commitment to accuracy and quality. * Strong problem-solving skills and the ability to learn new technologies quickly. * Proactive communication and reliable adherence to deadlines. * Discretion and reliability when handling sensitive projects.
