Documentation




Handbooks

SiLK Installation Handbook [ online ] [ pdf ]

Explains how to install SiLK and run the Collection and Packing system to create SiLK Flow files.

Analysts' Handbook: Using SiLK for Network Traffic Analysis [ pdf ]

Presents a tutorial on the SiLK tools and on using them for analyzing network traffic.

The SiLK Reference Guide [ online ] [ pdf ]

Provides the manual page for each tool in a single document.

SiLK Analysis Suite

The SiLK Analysis Suite is a collection of command-line tools for querying packed NetFlow data. The most important tool is rwfilter, an application for querying the central NetFlow data repository for NetFlow records that satisfy a set of filtering options.

The tools in the SiLK Analysis Suite include:

rwaddrcount

rwaddrcount {--print-recs|--print-stat|--print-ips} [SWITCHES] [FILES]
Summarize SiLK Flow records by source or destination IP; with the --print-recs option will produce textual ouput with counts of bytes, packets, and flow records for each IP, and the time range when the IP was active. When no files are given on command line, flows are read from STDIN.

rwbag

rwbag [SWITCHES] [FILES]
Read SiLK Flow records and builds binary Bag(s) containing key-count pairs. Key is protocol or source or destination address or port. Counter is sum of flows, packets, or bytes. Reads SiLK Flows from named files or from the standard input.

rwbagbuild

rwbagbuild {--set-input=FILE | --bag-input=FILE} [SWITCHES]
Create a binary Bag file from either a binary IPset file or from a textual input file. Use "stdin" as the file name to read from the standard input. The Bag is written to the standard output or the location specified with the --output switch.

rwbagcat

rwbagcat [SWITCHES] [BAG_FILES]
Print binary Bag files as text.

rwbagtool

rwbagtool [SWITCHES] BAG_FILE [BAG_FILES...]
Perform operations on bag files. Requires at least one bag file read from stdin or given on the command line.

rwcount

rwcount [SWITCHES] [FILES]
Summarize SiLK Flow records across time, producing textual output with counts of bytes, packets, and flow records for each time bin. When no files given on command line, flows are read from STDIN.

rwcut

rwcut [SWITCHES] [FILES]
Print SiLK Flow records in a |-delimited, columnar, human-readable format. Use --fields to select columns to print. When no files are given on the command line, flows are read from the standard input.

rwfilter

rwfilter <app-opts> <partition-opts> {<selection-opts> | <inputFiles>}
Partitions SiLK Flow records into one or more 'pass' and/or 'fail' output streams. The source of the SiLK records can be stdin, a named pipe, files listed on the command line, or files selected from the data-store via the selection switches. There is no default input or output; these must be specified.

rwgroup

rwgroup [SWITCHES]
Groups flows together by specified id-fields and delta-field; marks the group ID in next hop IP; requires pre-sorting.

rwmatch

rwmatch --relate=FIELD_PAIR QUERY_FILE RESPONSE_FILE MATCHED_PATH
Groups records as queries and responses

rwnetmask

rwnetmask [--source-prefix=N] [--destination-prefix=N] [--next-hop-prefix=N]
Read SiLK Flow records from STDIN, mask-off the lower (32-N) bits of the specified IP address(es), and write the resulting records to stdout. Does not process files.

rwpcut

Outputs a tcpdump file as ASCII, in a form similar to rwcut.

rwpdedupe

rwpdedupe <SWITCHES>
Detects and eliminates duplicate records. Duplicate records are defined as having the same 5-tuple and payload, and whose timestamps are within a user-configurable amount of time of each other.

rwpmapbuild

rwpmapbuild --input=<INPUT_FILE> --input=<INPUT_FILE>
Reads textual input and creates a binary prefixmap file for use with the Address Type (addrtype) and Prefix Map (pmapfilter) Plug-Ins.

rwpmatch

rwpmatch <SWITCHES>
Filter a tcpdump file by outputting only packets whose 5-tuple and timestamp match corresponding flows in a rw-file. Outputs the filtered tcpdump file to stdout.

rwptoflow

rwptoflow [SWITCHES] TCPDUMP_FILE
Read TCPDUMP_FILE and generate a SiLK Flow record for every packet. Write the SiLK Flows to stdout, which must not be connected to a terminal.

rwset

rwset [SWITCHES] [FILES]
Read SiLK Flow records and generate binary IPset file(s). When no files are given on command line, flows are read from STDIN.

rwsetbuild

rwsetbuild <INPUT_FILE> <OUTPUT_FILE>
Reads IP addresses in dotted-quad or CIDR notation from input-file and writes a binary IPset file to output-file. Use "stdin" as the input-file to read the IPs from the standard input, and use "stdout" as the output-file to write the IPset to the standard output when the standard output is not a terminal.

rwsetcat

rwsetcat [SWITCHES] [IPSET_FILES]
By default, prints the IPs in the specified IPSET_FILES. Use switches to control format of the outout and to optionally or additionally print the number of IPs in the file, the network structure, or other statistics. If no IPSET_FILEs are given on the command line, the IPset will be read from the standard input.

rwsetintersect

rwsetintersect --add=<file> [options] {--print-ips | --set-file=<file>}
Generates a new IP-set by intersecting all --add-set binary IP-set files, then removing all --remove-set binary IP-set files from the new IP-set. Prints the resulting IP-set to stdout and/or writes it to the specified file. Supports 1-4 add sets and 0-4 remove sets.

rwsetunion

rwsetunion <OUTPUT_IPSET> <INPUT_IPSET1> [<INPUT_IPSET2> ...]
Merge the input binary IPSet files into the output IPSet; an IP in any input file will be in the output file.

rwsort

rwsort --fields=<FIELDS> [SWITCHES] [FILES]
Read SiLK Flow records, sort them by the specified FIELD(S), and write the records to the named output path or to the standard output. When no FILES are given on command line, flows are read from the standard input.

rwstats

rwstats <SWITCHES> [FILES]
Summarize SiLK Flow records by one of a limited number of key/value pairs and display the results as a Top-N or Bottom-N list. The N can be a fixed value, a certain percentage of the innput, or a threshold value. Alternatively, provide statistics for each of bytes, packets, and bytes-per-packet giving minima, maxima, quartile, and interval flow-counts across all flows or across user-specified protocols. When no files are given on command line, flows are read from STDIN.

rwtotal

rwtotal <KEY> [SWITCHES] [FILES]
Summarize SiLK Flow records by a specified key and print the byte, packet, and flow counts for flows matching the key. When no files are given on the command line, flows are read from STDIN.

rwuniq

rwuniq --fields=N [SWITCHES] [FILES]
Summarize SiLK Flow records into user-defined keyed bins specified with the --fields switch. For each keyed bin, print byte, packet, and/or flow counts and/or the time window when key was active. When no files are given on command line, flows are read from STDIN.

Several analysis tools load shared libraries to enhance their functionality. These shared libraries are:

libaddrtype

The Address Type plug-in provides a way to map an IP address to an integer denoting the IP as internal, external, or non-routable.

libccfilter

The Country Code plug-in provides a mapping from an IP address to two-letter, lowercase abbreviation of the country that "owns" the IP address.

libpmapfilter

The Prefixmap plug-in provides a way to map field values to string labels based on a user-defined map file.

SiLK Collection and Packing Tools

The SiLK Packing System is a server application that receives NetFlow V5 PDU's and converts them into a more space efficient format, recording the packed records into service-specific binary flat files. Files are organized in a time-based directory heirarchy with files cover an hour at the leaves.

The tools that make up the SiLK Packing System are:

flowcap

flowcap <SWITCHES>
flowcap is a daemon which listens to devices which produce flow data (flow sources), homogenizes the data, stores it, and forwards as a compressed stream to a flowcap client program.

rwflowpack

rwflowpack <SWITCHES>
Read NetFlow V5 PDU records from a socket or from a file and pack the flow records into hourly flat-files organized in a time-based directory structure.

sensor.conf

Configuration file for sensors and probes.

Utilities

Additional utility programs:

mapsid

mapsid [SENSORS]
Maps between sensor names and sensor IDs. Prints a list of all sensors when no command line arguments are given.

num2dot

num2dot [SWITCHES]
Read pipe (|) delimited text from the standard input, convert integer values in the specified column(s) (default first column) to dotted-decimal IP addresss, and print result to standard output.

rwappend

rwappend [SWITCHES] TARGET-FILE SOURCE-FILE1 [SOURCE-FILE2...]
Append the SiLK Flow records contained in the second through final filename arguments to the records contained in the first filename argument. All files must be SiLK flow files; the TARGET-FILE must not be compressed.

rwcat

rwcat [SWITCHES] [FILES]
Reads SiLK Flow records from the FILES named on the comamnd line, or from the standard input when no FILES are provided, and writes the SiLK records to the specified output file or to the standard output if it is not connected to a terminal.

rwfglob

rwfglob [SWITCHES]
A utility to simply print to stdout the list of files that rwfilter would normally process for a given set of file selection switches.

rwfileinfo

rwfileinfo [SWITCHES] <FILES>
Print information (type, version, etc.) about a SiLK Flow, IPset, or Bag file. Use the fields switch to control what information is printed.

rwip2cc

rwip2cc [SWITCHES]
Maps from IP address to country code using the specified MAP file or the default map. Either one or more addresses must be specified.

rwrandomizeip

rwrandomizeip [SWITCHES] <INPUT_FILE> <OUTPUT_FILE>
Substitute a non-routable IP address for the source and destination IP addresses of <input-file> and write the result to <output-file>. You may use "stdin" for <input-file> and "stdout" for <output-file>. Gzipped files are o.k.

rwswapbytes

rwswapbytes [SWITCHES] <ENDIAN_SWITCH> <INPUT_FILE> <OUTPUT_FILE>
Change the byte-order of <INPUT_FILE> as specified by <ENDIAN_SWITCH> and write result to <OUTPUT_FILE>. You may use "stdin" for <INPUT_FILE> and "stdout" for <OUTPUT_FILE>. Gzipped files are o.k.

rwtuc

rwtuc [SWITCHES] [FILES]
Generate SiLK flow records from textual input; the input should be in a form similar to what rwcut generates.