crawler4j - Open Source Web Crawler for Java

crawler4j - Open Source Web Crawler for Java - Google Project Hosting

Crawler4j is an open source Java crawler which provides a simple interface for crawling the Web. You can setup a multi-threaded web crawler in 5 minutes!

NOTE: version 3.0 is deprecated and should not be used. Please use the latest 3.3 version.

Sample Usage

You need to create a crawler class that extends WebCrawler . This class decides which URLs should be crawled and handles the downloaded page. The following is a sample implementation:

      
        public
      
      
      
      
        class
      
      
      
      
        MyCrawler
      
      
      
      
        extends
      
      
      
      
        WebCrawler
      
      
      
      
        {
      
      
        

        

                
      
      
        private
      
      
      
      
        final
      
      
      
      
        static
      
      
      
      
        Pattern
      
      
         FILTERS 
      
      
        =
      
      
      
      
        Pattern
      
      
        .
      
      
        compile
      
      
        (
      
      
        ".*(\\.(css|js|bmp|gif|jpe?g"
      
      
        

                                                                  
      
      
        +
      
      
      
      
        "|png|tiff?|mid|mp2|mp3|mp4"
      
      
        

                                                                  
      
      
        +
      
      
      
      
        "|wav|avi|mov|mpeg|ram|m4v|pdf"
      
      
        

                                                                  
      
      
        +
      
      
      
      
        "|rm|smil|wmv|swf|wma|zip|rar|gz))$"
      
      
        );
      
      
        

        

                
      
      
        /**
        

                 * You should implement this function to specify whether
        

                 * the given url should be crawled or not (based on your
        

                 * crawling logic).
      
    

crawler4j - Open Source Web Crawler for Java - Google Project Hosting