Overview

SperoSuite is a set of related components for creating local static HTML mirrors of remote sites, including a search engine Java applet.

Overview

Sperowiderconsists of four different products :

Sperowider, which makes a local mirror of a remote website, and performs indexing for the Sperosearch component.
Sperosearch, a Java applet designed to perform search functionality using the indexes generated by Sperowider and Simple Spider.
Simple Spider is a simplified toolkit, built on the same technology as Sperowider, for generating search indexes. It does not download, but shares much of the same configuration flexibility.
Masterdoc, which generates multi-project Javadocs. It isn't really related to the rest of the code base, but was generated as part of it.

Useful Links

http://sourceforge.net/projects/sperowider/
http://www.sperowider.org

Documents

Sperowider has, in org/erowid/sperowider directory, some useful documentation for the Sperowider application, specifically information about how to run Sperowider and ... more critically ... a sample configuration.

It is possible to have the Sperowider application output this information at runtime, using the --help and --sample flags.

Software Components

SperoSuite is built using other software packages compatible with the BSD license : Log4j, Lucene, Hypersonic SQL, JDOM HttpClient

This product includes software developed by the Apache Software Foundation (http://www.apache.org/).
This product includes Hypersonic SQL.
This product includes software developed by the JDOM Project (http://www.jdom.org/).

License & Copyright Summary

Generally, Sperowider's source code is free for use as long as attribution and copyright notice are retained. Sperowider's source code is compliant with the Creative Commons Attribution License, but there are some additional restrictions around the use of the name Sperowider. Absolutely no warranty of any kind is implied. Use of the name Sperowider or other unique names for this project in advertising is prohibited without prior approval. Erowid is a registered trademark of Erowid.org and Erowid retains control of the name Sperowider. For more information about the licensing issues, see the Sperowider License.

Erowid is a non-commercial project, supported by donations, dedicated to improving the quality, quantity, and availability of information.

Packages
org.erowid.masterdoc	Masterdoc is designed to create an enterprise Javadoc site, with multiple local Javadoc repositories each cross-linked, along with references to useful external repositories.
org.erowid.sperosearch	The applet (and supporting classes) that provides search functionality against the search index generated by Sperowider, during its index phase.
org.erowid.sperosearch.splitindex	Takes one or more files in a directory that have been split into equally sized segments and gzip compressed, and presents them for reading via an input stream.
org.erowid.sperowider	Provides the core Sperowider functionality of downloading, spidering, rectifying, and indexing (for the SperoSearch applet) a website.
org.erowid.sperowider.configuration	The configuration, default configuration, and CLI override model for Sperowider.
org.erowid.sperowider.hsqldb	Provides a Hypersonic SQL backed `ISperowiderModel`.
org.erowid.sperowider.htmlshredding	Provides HTML shredding (taking an HTML file and breaking it into individual elements) and URL mongling (throwing events when URLs are found, and providing an opportunity to replace them) functionality.
org.erowid.sperowider.modelreporting	Configurable hook model for intercepting and reporting on Sperowider events.
org.erowid.sperowider.simple	A simple, lightweight spider that uses Sperowider core technology to walk a website and create a search index.
org.erowid.sperowider.urlfilter	URL filter intefaces and implementations to allow for control over Simple Spider spidering decisions, and Sperowider spidering, rectifying, downloading, and indexing decisions.