A great opportunity to join an established team working with Business Intelligence systems
We are a leading player in the data-driven market intelligence industry, and are seeking an experienced Web Scraping Specialist and Crawler Developer to join our remote and dynamic team. The ideal candidate will possess the technical know-how and creative problem-solving skills necessary to effectively gather data from e-commerce websites, bypass blocking technologies, and contribute to the growth of our data-driven solutions.
You will have the ability to pick up, understand and become an expert in our existing platforms quickly, as well as bringing to the table innovative ideas on how we may improve our scraping tools, which are definitely not as modern as we would like them to be! Naturally if you have skills outside of scraping data we would like to hear about this.
This is a fast-paced environment. You will naturally be a logical thinker, comfortable working with large data sets, multiple servers and code that has grown quickly and is far from perfect.
So, what will you be doing?
- Design, develop, and maintain custom web scraping and crawling solutions to gather data from various e-commerce websites
- Utilise advanced techniques to bypass blocking technologies, including CAPTCHAs, IP blocking, and user agent restrictions
- Collaborate with the data team to define requirements and deliver high-quality, structured data
- Monitor, troubleshoot, and optimise the performance of web crawlers and scrapers
- Ensure compliance with applicable data privacy regulations and ethical web scraping practices
- Document and maintain technical specifications, guidelines, and best practices for web scraping projects
- Stay up-to-date with the latest web scraping and data extraction technologies, tools, and techniques
Who would be a great fit?
- 2+ years of experience in web scraping, crawling, and data extraction, with a strong focus on e-commerce websites
- Proficient in Python, and experience with web scraping libraries such as Scrapy, BeautifulSoup, and Selenium
- Demonstrated knowledge of web technologies (HTML, CSS, JavaScript, AJAX) and understanding of web protocols (HTTP, HTTPS)
- Experience with proxy management and rotation and tools Strong problem-solving skills, with the ability to circumvent website blocking technologies
- You will have no problems working remotely and online, as part of a team that is not sat in the same office
- Someone who can help to refine and improve the existing systems and processes that we have in place
- Have fantastic planning and organisational skills and are motivated to work independently
- Excellent English written and verbal communication skills. As we talk often via Zoom or Microsoft Teams or other audio/video. Being comfortable with this is a definite requirement
- Ability to communicate complex ideas in a straightforward way
- Thrive under pressure, in a fast-paced environment, with the ability to adjust priorities at short notice
- A self-starter who can work independently, but also someone who can also work well within a team - lone wolves need not apply!
- You will have a fast, stable internet connection, as this is essential due to the nature of the work
This position is a remote based role, meaning that you can work wherever suits you! Our head office is based in the North East of the UK, however we have a global team so you’ll get to work with a diverse group of really talented people. Working hours are flexible, providing that you give our team the support they need, develop great software and are available to communicate with our team at agreed times.
Ideally we are looking for this to be a full-time position, but one of our strengths, as a company, is our flexibility. Let us know what hours you can do and your timezone. To start we will be seeking to work on an hourly basis until we get to know you and then we will guarantee a monthly salary.
How to apply
We’re looking for candidates that stand out from the crowd, so here are a few pointers to help you:
- If you have a Github profile, it would be great to see it!
- Examples of systems where you have major contributions
- Companies or products that you have worked on
- Your passions, hobbies and interests outside of programming
- Where are you from? Geography doesn’t matter to us, but it’s always nice to find out where people live and work! Our current team is all around the world, so tell us something interesting about your environment
- Anything else about you and what you have done previously that might make us choose you over someone else
- We will respond to everyone who writes to us with the above information
If you are interested in the role and becoming part of our team, please visit the following form to upload your CV/Resume and covering letter (top tip - this is our favourite part!):
We are looking forward to hearing from you
