• 开源吧 > JAVA开源项目 > 组件类库 > Web爬虫

---

Smart and Simple Web Crawler是一个Web爬虫框架。集成Lucene支持。该爬虫可以从单个链接或一个链接数组开始，提供两种遍历模式：最大迭代和最大深度。可以设置过滤器限制爬回来的链接，默认提供三个过滤器ServerFilter、BeginningPathFilter和 RegularExpressionFilter，这三个过滤器可用AND、OR和NOT联合。在解析过程或页面加载前后都可以加监听器。

What is the Smart and Simple Web Crawler?

    * Smart and easy framework that crawls a web site
    * Integrated Lucene support
    * It's simple to integrate the framework in own applications
    * The crawler can start from one or from a list of links
    * Two crawling models available:
          o Max Iterations: Crawls a web site through a limited number of links: Fast model with a small memory footprint and cpu usage.
          o Max Depth: A simple graph model parser without recording in and outcoming links. Fast as the max interations model.
    * Accept filter interface to limit the links to be crawled
    * Core accept filters available: ServerFilter, BeginningPathFilter and RegularExpressionFilter
    * Combining the accept filters with AND, OR and NOT possible
    * Plugable http connection libraries HttpClient (default) and HTMLParser (optional)
    * Own listeners can be added in the parsing process or before and after loading a page
    * The framework is not a GUI based tool to mirror a website and browse the site offline!