SPEROWIDER CONFIGURATION FILES

Introduction to Sperowider Config Files #

Sperowider configuration files are XML-text files that contain at least one target "run", but may contain any number. Target runs are specified on the command line using the syntax: java -jar sperowider.jar >targetname<. If no target is specified on the command line, Sperowider looks for the attribute "default-target" in the outter sperowider-config element. If it is unable to find the default target, Sperowider will quit with an error message.

Syntax Description #

Since the config syntax is XML, it is important to note that the files must be well formed or Sperowider will produce errors when reading the config file. Each <> element must have attributes set with double quotes and each attribute must be closed properly (either with a </elementname> or by self closing <elementname />). Sperowider 1.0 does not provide extremely robust debugging of bad XML files, so if you experience config problems, verify the integrity of the XML file by hand or with another tool.

Probably the simplest way to get to know Sperowider config files is to just take a look at the config examples which include some documentation and are relatively straight forward.

Every config file must start with an XML file declaration (<?xml version="1.0"?>) and the next element should be a "sperowider-configs" wrapper element for any targets in the file. Each sub-element of "sperowider-configs" is a called a 'target' and the element name is "sperowider-config" element. Inside each specific target are the elements "model", "locations", "command", "url-filter", and "logging", each with their own settings, described below.

Config File Look & Feel #

A config file looks a bit like the following:

<?xml version="1.0"?>
<sperowider-configs default-target-name="main">
 <sperowider-config target-name="main">
  <model class-name="org.erowid.sperowider.hsqldb.SperowiderModel">
	<repository name="sperowider-download" delete-old-data="true" />
  </model>
  <locations download-root="output/"
		starting-url="http://www.domain.org/path/to/start.htmll" />
  <command throttle="500" limit="0">
	<action download="true" rectify="true" index="true" />
  </command>
    <url-filter class-name="org.erowid.sperowider.SimpleURLFilter" >
       <includes>
           <include pattern="http://www.domain.org/path/to/*" />
       </includes>
       <excludes>
           <exclude pattern="*.zip" />
       </excludes>
    </url-filter>
	<logging log4j-config-filename="log4j.config" />
 </sperowider-config>
</sperowider-configs>

Config File Full Examples #

These examples can be cut and pasted into files and are also included with the distribution.

• Simple Config Example
• Multiple-Target Config Example
• Regex-Filter Config Example

Sperowider Config Structure: 1.0 #

• default-target-name = string
  Name matching the sperowider-config to be run if no target is specified on the command line.

• target-name = string
  Name of this configuration for command line use or for reference by sperowider-configs::default-target-name. The target-name is critical.

• class-name = "org.erowid.sperowider.hsqldb.SperowiderModel" / "org.erowid.sperowider.BasicSperowiderModel"
  Which Sperowider model to use. A Sperowider model basically provides storage and retrieval and state information for the Sperowider. There are currently two :
  
  • org.erowid.sperowider.BasicSperowiderModel
    Uses in-RAM data structures (primarily hash maps) to store state. State cannot, therefore, be preserved between runs.
  • org.erowid.sperowider.hsqldb.SperowiderModel
    Uses an HSQLDB database to store state. Therefore, state can be preserved between runs. This allows, for example, the spidering to be split between multiple sessions.
• support-spider-map = true/false
  Only available with hsqldb model. Tracks on which page a URL was found. This is very useful for debugging 404s and other problems, but adds to the memory and disk requirements and slows Sperowider down a little.

• name = string
  Name for the directory in which the hsqldb.SperowiderModel will save its tracking data.
• delete-old-data = true/false
  Whether to delete the data in the status repository directory.

• download-root = string
  The path to the directory where Sperowider will put the actual downloaded files and/or where to find the already downloaded files during the Rectify and Index Actions.
• starting-url = string
  The initial URL to being spidering at, may be excluded for Rectify and Index runs.

• throttle = integer
  In milliseconds, the minimum amount of time that Sperowider will do downloads. It is a semi-guarantee that no more than 1 download per xxx milliseconds will happen.
• limit = integer
  The maximum number of files that will be downloaded. If limit is set to 0, the Sperowider will download files until there are no more files to download.

• download = true/false
  Should be set to "true" if you want files to be downloaded. They will be placed in a tree rooted at the directory named in the download-root property in the locations declaration section.
• rectify = true/false
  Set to "true" if you want files to be rectified.
• index = true/false
  Set to "true" if you want the downloaded files to be indexed for searching with Sperosearch.

• class-name = org.erowid.sperowider.SimpleURLFilter / org.erowid.sperowider.RegexURLFilter
  Specify whether to use simple or regex filtering. Simple filtering can include a single *, regex filtering is standard regular expressions with the note that it attempts to match the entire URL string (an implied ^ and $).

• pattern = string
  The pattern for the simple model may contain only one * or must match a URL exactly. The pattern for regexs are standard regex, with assumed ^ and $.

• pattern = string
  The pattern for the simple model may contain only one * or must match a URL exactly. The pattern for regexs are standard regex, with assumed ^ and $.