When used to archive erowid or any very large site, we run in three waves: downloading, rectifying, and indexing using the following command line syntax:
- Step 1: java -Xmx300m -jar /path/to/sperowider.jar --download.enabled=true --rectify.enabled=false --index.enabled=false --config=mirror-config.xml
- Step 2: java -Xmx300m -jar /path/to/sperowider.jar --download.enabled=false --rectify.enabled=true --index.enabled=false --config=mirror-config.xml
- Step 3: java -Xmx300m -jar /path/to/sperowider.jar --download.enabled=false --rectify.enabled=false --index.enabled=true --config=mirror-config.xml
- All At Once: java -Xmx300m -jar /path/to/sperowider.jar --download.enabled=true --rectify.enabled=true --index.enabled=true --config=mirror-config.xml