Frequently Asked Questions

OutOfMemoryError: Sperowider stops sometimes with the error: "Exception in thread 'main' java.lang.OutOfMemoryError", what do I do?

For large sites, Sperowider 1.01's rectification and indexing can take up a lot of memory. The default maximum heap size for a java application is 128MB and Sperowider just runs out of room and is killed by java. We have been unable, so far, to track down what is gobbling up the most memory and hopefully 1.1 should be a little trimmer in its memory usage. For runs of less than 10-20,000 files, we have not run into this problem, but it does come up with large sites. The solution to this problem is to simply increase the maximum allowed memory for the run. The syntax to increase the heap size to 256MB is :

java -Xmx256M -jar sperowider.jar

Increasing the heap size to 256MB also dramatically increases the speed of rectification for large sites. 256MB is large enough to rectify a site with 35,000 files and (on a dual 1.2gigahertz pIII with scsi ultra160 drives) the increased stack size allowed the rectification to take only 15 minutes.

Why didn't it convert URLs in my javascript?

One of the major reasons we had to develop Sperowider is that the other robots don't handle archiving javascript linked-data very well. Sperowider will open included javascript files and localize URLs it can find. It can be hard to determine exactly what is a URL. I'm not entirely sure how this is handled internally, but I think it may be simply looking for things that look like file names inside double quotes.

So, menu_data.js might contain something like:

addMenu("setting", 1,2, 3, "Title of Menu Item", "Location Info", "/path/to/link/target.html");

Sperowider would detect what appears to be a filename (and thus URL) in the code and would mongle it into either absolute or local form. So the result might be something like:

addMenu("setting", 1,2, 3, "Title of Menu Item", "Location Info", "../../path/to/link/target.html");

or if it was excluded from the set:

addMenu("setting", 1,2, 3, "Title of Menu Item", "Location Info", "http://www.domain.org/path/to/link/target.html");

Also, Sperowider looks in .js files for the <spero-mongle> tag which can contain N <pattern> tags. The syntax for these looks like:

// <spero-mongle><pattern>"((/[^"]+)|(http://[^"]+))"</pattern><group>1</group><replace-with> SperoDocRoot + "includes/headers/" + "${URL}"</replace-with></spero-mongle>

Sperowider ignores the comment tags which allow these to be transparent to javascript.