This file contains two main pieces of information - firstly, information on how you can efficiently download and/or mirror sections of Wiretapped, and secondly information on how you can make your site more accessible for sites such as Wiretapped to mirror.

Efficient Downloading

If you're wishing to download and/or mirror sections of Wiretapped, there are a number of pieces of software that will make this task easier and at the same time, reduce the load on our server.

wget:

wget is a freely-available utility for network downloads. It's quite powerful, but it's easy to accidentally download a lot more than you intended if it's not used properly. The problem is that unlike most other Unix utilities, the authors supply no man page. The wget.info file they provide is poorly organised, and does not clearly outline the implication of some command-line options.

If you wish to download a single, one-off snapshot of something, wget is ideal. For an ongoing mirror of something, I strongly recommend you consider using lftp or fmirror instead, as wget does not track file deletions or changes.

Downloading a directory via FTP

If you want to download ftp://ftp.wiretapped.net/pub/OpenBSD/2.9/i386/ and drop its contents into the current directory (ie .). You'd use the command:

wget -m -nH -np --cut-dirs=4 ftp://ftp.wiretapped.net/pub/OpenBSD/2.9/i386/

The -m specifies mirroring (this does the same as -r (recursion) and -N (timestamping)), the -nH specifies not to drop files into a directory called ftp.wiretapped.net, the -np ("no parent") prevents wget from traversing back to 2.9 and into all it's other subdirectories and the --cut-dirs=4 tells wget to remove the /pub/OpenBSD/2.9/i386/ from the directory name of the local target. (We could always do -nd instead of -nH here, but if the i386 directory contains any subdirectories, we lose this directory structure with -nd; it is retained when we use -nH).

Downloading a directory via HTTP

The options here are almost the same as above, though we add an extra command line flag to prevent your local target getting cluttered up too much. Apache is great in that it offers multiple sorting views of each directory listing it creates (Name, Last Modified, Size and Description) but this makes it difficult for wget-style apps on all platforms that aren't aware that the resulting file contains exactly the same links as the master. Even Google suffers from this...

To grab the contents of http://www.mirrors.wiretapped.net/security/packet-capture/ and drop them into a directory called packet-capture, retaining the directory structure of all deeper links and without cluttering up the directory, we use the following command:

/usr/local/bin/wget -m -nH -np --cut-dirs=1 -R "=D","=A" http://www.mirrors.wiretapped.net/security/packet-capture/

In this case, we cut the hostname by using -nH and only the leftmost directory name with --cut-dirs=1. We use the -R "=D","=A" option here to "reject" the links that Apache creates for differently sorted directory listings (?N=D = Name sort, ?M=A = Last Modified sort, ?S=A = Size sort, ?D=A = Description sort).

lftp:

lftp is described by its developer as a sophisticated http/ftp client & file transfer program. It's all this and more. What it truly does best is track changes in index-listed files on http and ftp servers. That is, if you're mirroring from a web page (eg an index.html file somewhere), it might not do such a good job. If you're mirroring from an Apache-generated index/listing, it's untouchable. Ditto FTP indexes/listings.

lftp is available at: http://lftp.yar.ru/

If you're mirroring files from Wiretapped, you'll broadly want to follow this convention:

cd /target/directory/xinetd/
/usr/local/bin/lftp -e 'o http://www.mirrors.wiretapped.net/security/port-loggers/xinetd/ && mirror --delete --verbose && quit'

lftp is usually assumed to work on the basis of commands fed to it in a sequence using a shell-like command syntax. This is best illustrated by looking at the command above. The first part, o http://url/ tells lftp where it should open a connection to (o stands for open). Although the notion of opening a connect then later issuing a command is ftp-like, lftp follows the same convention even with http:// urls. The && characters specify that the mirror command only runs upon successful completion of the o http://url/ command. lftp also supports ||, but see the manual page for full details. The mirror command in the example specifies that remote deletions should be reflected locally and that the command should execute showing file download progress. A mirror command without a directory argument (as above) specifies that we are mirroring . (the current directory).

An example where we wish to prevent local deletion of a file (in this case, a README file):

cd /target/directory/xinetd/
/usr/local/bin/lftp -e 'o http://www.mirrors.wiretapped.net/security/port-loggers/xinetd/ && mirror --delete --verbose -x "xinetd-README.txt" && quit'

The -i (include) and -x (exclude) flags for the mirror command are based on egrep(1) regular expressions and are fully documented in the manual page.

fmirror:

fmirror is a C-based re-write/clone of the popular mirror.pl perl script used for mirroring via FTP. fmirror is vastly faster at most operations and uses less memory than perl/mirror.pl. It's configuration is similar in nature, and I've only ever noticed a couple of FTP servers that do not work with it (mostly those using old Unix ftp daemons or obscure Windows NT directory listings). fmirror is downloadable from here and should build cleanly on most Unix variants. fmirror has a great manual page, so I'll only provide one or two configuration examples here.

To mirror a directory without any exclusions

This is an example of a fairly simple fmirror configuration file which I have called ssh-official.conf:

# SSH Official

username:       anonymous
password:       mirroring@yoursitename.com
host:           ftp.ssh.com
remotedir:      /pub/ssh
localdir:       /path/to/local/target/ssh/
dirmode:        0755
file_or_mask:   0755
use_mdtm:       2

To actually run the mirror, we use this command line:

/usr/local/bin/fmirror -Rf ssh-official.conf

The -R command line flag tells fmirror to set the timestamps of the local files to the same as the remote files (this works for existing files whose timestamps need to be resynced and for files that have been freshly downloaded) and the -f flag simply specifies the configuration file.

To mirror a directory with local and remote exclusions

This is where things get a bit trickier and often require some tweaking and/or trial and error. In this case we want to mirror the contents of a remote directory, excluding several subdirectories and ignoring the local presence of a subdirectory (ie something else we've put there). To do this, here is an example configuration file for fmirror:

# 128-bit Netscape

username:       anonymous
password:       mirroring@yoursitename.com
host:           64.12.168.249
remotedir:      /pub/communicator/english
localdir:       /path/to/local/target/Netscape/
dirmode:        0755
file_or_mask:   0755
use_mdtm:       2

exclude:        p       4.75/
exclude:        p       4.76/
exclude:	p	4.77/
exclude:	p	4.78/
exclude:        xnp     (^|/)netscape6/

Now, the top part of the configuration file is substantially the same as the previous example. The main difference is in the exclude: commands given. Here, we wish to prevent the 4.75, 4.76 etc directories from being downloaded, which we achieve with a pathname pattern match (ie p) on the directory names. We also wish to ignore the local presence of the netscape6 directory, which we have another fmirror process taking care of separately. To do this, I use xnp and a regular expression to describe the directory name. Be aware that although fmirror is supposed to be aware of regular expressions in the same way mirror.pl does, it doesn't always work out that way. For me, the exclude: xnp (^|/)netscape6/ works perfectly. Then again, it's a fairly obscure example.

Some additional fmirror notes:

fmirror only sends a "\n" (LF) as it logs into an FTP site, but some FTP servers may require "\r\n" (CR+LF) to be sent. In particular, this affects Serv-U v2.5 (and possibly other servers). The bug will appear most likely as a failure to log in correctly, resulting in a stalled fmirror. Thanks to zhasper for bringing this to my attention. To fix this, you can apply the following patch:

For fmirror 0.84:

--- fmirror-0.8.4/fmirror.c.orig        Tue Mar  5 12:13:32 2002
+++ fmirror-0.8.4/fmirror.c     Tue Mar  5 12:13:50 2002
@@ -642,7 +642,7 @@
     va_list args;
     va_start(args, format);
     vsprintf(buffer, format, args);
-    l = fprintf(out_file, "%s\n", buffer); 
+    l = fprintf(out_file, "%s\r\n", buffer); 
     fflush(out_file);
     buffer[l - 1] = 0;
     LOG(6, cmd, ("---> %s", buffer));

For fmirror 0.85pre1:

--- fmirror-0.8.5-pre1/fmirror.c.orig   Tue Mar  5 12:06:19 2002
+++ fmirror-0.8.5-pre1/fmirror.c        Tue Mar  5 12:06:29 2002
@@ -645,7 +645,7 @@
     va_list args;
     va_start(args, format);
     vsprintf(buffer, format, args);
-    l = fprintf(out_file, "%s\n", buffer); 
+    l = fprintf(out_file, "%s\r\n", buffer); 
     fflush(out_file);
     buffer[l - 1] = 0;
     LOG(6, cmd, ("---> %s", buffer));

Mirroring Wiretapped

If you're interested in mirroring Wiretapped, there's a few things you should take into consideration before doing so. Please read on.

Firstly, the archive is now quite large, and a significant proportion of it is consumed by the operating-systems directory. For this reason, we recommend you explicitly ignore this directory in any mirroring of Wiretapped that you carry out. Should you wish to mirror a particular directory for your local (read: private) usage, please feel free to refer to this 'du' output to determine the required amount of space on your server. ('du' output is updated daily.)

Secondly, we always like to know what sites are mirroring Wiretapped, even if they aren't publicly available (most aren't). We therefore ask that before you start mirroring, please send us an email to web2005a[at]year2005a.wiretapped.net

Lastly, we recommend you use fmirror (see above) to mirror Wiretapped - not lftp and certainly not wget. Therefore, we've provided an fmirror config below for you to modify and use. Copy it to a file and run using the supplied command.

Wiretapped mirroring config for fmirror:

# Wiretapped mirror
# WWW: http://www.wiretapped.net/
# FTP: ftp://ftp.wiretapped.net/
# Contact: web2005a[at]year2005a.wiretapped.net

username:       anonymous
password:       mirroring@yoursitename
host:           ftp.wiretapped.net
remotedir:      /pub/security
localdir:       /path/to/local/directory/security/
dirmode:        0755
file_or_mask:   0755
use_mdtm:       2

# Exclusions (uncomment as required)
# The "exclude: ip (^|/)directory/" means
# that fmirror will case insensitively exclude
# anything in a directory with the name that follows.
# See fmirror manpage for more detailed exclusion.

# exclude:	ip	(^|/)authentication/
# exclude:	ip	(^|/)cryptography/
# exclude:	ip	(^|/)development/
# exclude:	ip	(^|/)firewalls/
# exclude:	ip	(^|/)host-intrusion-detection/
# exclude:	ip	(^|/)host-monitoring/
# exclude:	ip	(^|/)host-security/
# exclude:	ip	(^|/)info/
# exclude:	ip	(^|/)network-intrusion-detection/
# exclude:	ip	(^|/)network-mapping/
# exclude:	ip	(^|/)network-monitoring/
# exclude:	ip	(^|/)network-security/
# exclude:	ip	(^|/)operating-systems/
# exclude:	ip	(^|/)packet-capture/
# exclude:	ip	(^|/)packet-construction/
# exclude:	ip	(^|/)steganography/
# exclude:	ip	(^|/)vulnerability-scanning/
# exclude:	if	^.header.html
# exclude:	if	^.footer.html

(download here)

To run:

fmirror -Sf /path/to/wiretapped.conf

How to setup a site for mirroring

If you have a resource you'd like mirrored by sites such as Wiretapped, it's fairly simple to make sure your site provides the correct index listings and permissions for us.

FTP

By far, the easiest way to make a resource available for mirroring is to provide access to it via FTP and supply a "README" file at the base directory so visitors to the mirror sites can easily determine the contents of any subdirectories.

An ideal setup might be similar to the one provided by SSH Communications at this location: ftp://ftp.ssh.com/pub/ssh. This is ideal because it provides a README file, because it provides only the latest version in the top-level directory (older releases are moved into a directory called "old"), and because all the files have a relatively uniform naming scheme.

A poor setup might be similar to the one provided by AOL for their instant messaging software at this location: ftp://ftp.newaol.com/aimgen/73010/. This is poor because it provides no README file, because there are multiple versions for multiple operating systems in the one directory, and because the naming scheme changes from version to version of the software.

As to the question of security and/or keeping your master site hidden from the screaming hordes, all mirroring packages are capable of authenticated logins, so having a username and password on your master site is fine.

HTTP

The second-easiest way to make a resource available is to provide a directory on your website containing all the files you would like mirrored, and ensure that your web server provides an index listing when queried directly. As with FTP, it is also helpful to supply a "README" file.

An ideal setup might be similar to the one provided by Jean-Sebastien Morrisset at his rc.firewall website: http://rcf.mvlan.net/dist/. This is idea because it provides a README file, because it provides only the latest version in the top-level directory (older releases are moved into a directory called "old"), and because all the files have a consistent naming scheme.

A poor setup is such as the one provided by Snort. Snort has a download directory, but it's surrounded by all the other pretty, flashy gumph from the rest of the site. Which makes it really difficult to mirror. Compare this setup with the Snort directory we manually maintain on Wiretapped, located here. Is this a more ideal setup? We think so.

It is worth noting that all the abovementioned mirroring tools capable of downloading over HTTP (wget, lftp) do not handle pages created with PHP at all. They simply don't. This precludes the use of things like custom generated directory listings and such. We used to have a problem mirroring ettercap until the software authors allowed directory listings in their download directory. Such a solution would work ideally in the case of Snort.

In Apache, to allow directory indexes in a form that lftp and wget can understand, use a setup such as this:

<Directory /path/to/directory>
Options Indexes
</Directory>

To summarise, here is what a site should do to assist mirroring:

• Indexes, with an FTP directory or a directory on your website that allows indexes.
• A well organised directory structure.
  • A "README" file at the top-level directory if possible.
  • Old files in an "old" directory.
  • A consistent naming scheme for your files.
• Everything in the one directory structure, so multiple mirrors are not necessary.
• Strenuously avoid "dynamic" web pages (PHP, Perl, ASP) in areas requiring mirroring, because it just won't work.

If you have found this page useful or would like to offer other feedback or suggestions, please feel free to contact us at web2005a[at]year2005a.wiretapped.net

$Id: mirroring.html,v 1.31 2007/08/19 08:00:09 gbayley Exp $