DigitalPebble can help your organisation by advising on best practice and identifying suitable resources, designing scalable solutions as well as implementing them. We can help you deploy and monitor your project on your premises or on the cloud.
We are the authors and maintainers of StormCrawler, one of the leading open-source solutions for web crawling. Used by numerous companies all over the world, it is both scalable and highly configurable.
We can help you customise StormCrawler and run it on your premises or in the cloud, or, alternatively, DigitalPebble can run it on your behalf.
We have a large experience of using leading search tools such as Elasticsearch, OpenSearch or Apache SOLR.
Whether you want to index and search texts or any other type of documents, we can help you to design a search solution to fit with the rest of your architecture.
Some of our clients have billions of documents indexed, and with our solid background in Natural Language Processing and Machine Learning, there is a lot we can do to enrich your documents.
Processing data on a large scale either in streaming or batch can be done with platforms such as Apache Flink or Apache Storm.
In fact, we have built some of our open source solutions on these platforms and have a large experience of using them for our clients.
Combined with our know-how and expertise in cloud computing, we are confident we can help you deliver your project, no matter how much data you have.
Having studied Russian language and culture in Paris and taught French in a school in Kyiv, Ukraine, Julien went on to graduate in Text Engineering and Natural Language Processing. He moved to the UK to work as a researcher at the University of Sheffield in 2005 and founded DigitalPebble in 2008.
Julien has been involved in several open source projects, mainly at the Apache Software Foundation, and was the PMC chair for Apache Nutch. He is a member of the Apache Software Foundation.
Julien runs workshops on web crawling, speaks at conferences and reviews technical books. He has over 20 years experience in the Java programming language.
Advice on best practices for web crawling and customisation of StormCrawler.
Customization of StormCrawler for crawling images with Elasticsearch.
Customisation of StormCrawler. Text classification with ApacheSpark.
Low latency web scraper for job adverts based on StormCrawler.
Design and implementation of a WARC exporter for Nutch, StormCrawler pipeline for crawling the news dataset
Auditing, redesign and optimisation of a SOLR setup for a real estate search system using geo-location.
