SiLK-0.10.5 Release, 2006-Dec-12

• Data file version number bump
  • Fix a forward compatibility issue in SiLK between releases prior to 0.10.0 and releases 0.10.0 through 0.10.4 when data compression is enabled (either via the --enable-output-compression switch to 'configure' or the --compression-method switch to various applications). Versions of SiLK prior to 0.10.0 did not check the value of the 'compression' byte in the header; when reading a SiLK file from 0.10.0 with compression enabled, these versions will silently attempt to read the data section without uncompressing it, leading to incorrect output.
    
    The issue is resolved in SiLK 0.10.5 by incrementing the version number of every SiLK file format that supports compression of the data section of the file (IPsets, Bags, and the output from rwfilter, rwcat, rwsort, rwflowpack, and rwptoflow).
    
    We recommend using the "silk-version-bump-0-10-5" script included with the distribution to increment the version number of files created with releases of SiLK prior to 0.10.5 that have compression enabled. The script will only modify SiLK files that have compression enabled; it will not modify non-SiLK files nor SiLK files that do no have compression enabled.
• rwcount change
  • IMPORTANT. The default binning mode (load-scheme) has changed. The former scheme put each flow's entire volume into the first second of the flow. The new scheme evenly divides the volume across each second of the flow's duration, which should help reduce "spikiness" in the data. Any scripts that rely on the former method should have "--load-scheme=1" added explicitly to rwcount's invocation.
• rwuniq enhancement and bug fix
  • New flag "--presorted-input" makes rwuniq assume that the data has been sorted with rwsort using the same set of "--fields". This reduces rwuniq's memory requirement and allows it to work like it's UNIX counterpart 'uniq'.
  • Fix a memory fault that could occur when using the --sip-distinct and/or --dip-distinct switches on large data sets.
• rwfilter changes
  • rwfilter will continue to process even if there is a problem with an input file.
  • rwfilter will now process multiple RWFILTER input files, though it prints a warning that file history is being lost.
  • rwfilter supports time filtering (via the --stime and --etime switches) to the millisecond
• New script rwp2yaf2silk:
  • rwp2yaf2silk converts a file of pcap data to SiLK Flow data; the script requires that the SiLK tool 'rwtuc' is installed and that the tools 'yaf' and 'yafscii' (http://tools.netsa.cert.org/yaf/) are installed.
• rwbagcat bug fix
  • Make certain the --bin-ips=linear switch properly handles Bag entries where the count is greater than 4294967295. These entries are now attributed to the maximum key unless the --maxcount value is used to filter out those entries.
  • When printing the output from --bin-ips=decimal, properly print the key when its value is greater than 4294967295
  • Set the output column width to 20 to maintain the columnar output when the value is very large.
  • Support values larger than 4294967295 in the --mincount and --maxcount switches
• rwbagtool bug fix
  • Fix a bug in the --invert switch which resulted in incorrect results in the output. This would occur when the value was larger than the current key.
  • Make certain the --invert switch properly handles Bag entries where the count is greater than 4294967295. These entries are now attributed to the maximum key unless the --maxcount value is used to filter out those entries.
  • Allow the --invert switch to support multiple Bag files by adding the Bags (making the switch consistent with the --coverset, --intersect, and --compliment-intersect switches). This fixes an assertion that would cause the program to abort.
  • Support values larger than 4294967295 in the --mincount and --maxcount switches
• rwflowpack input check
  • When processing NetFlow data from a file, rwflowpack now checks that the input data is in NetFlow v5 format. Previously, the version check was not made and the file would be processed as if it contained NetFlow v5 data.
• rwpmatch enhancement and bug fix
  • Provide --ports-compare and --msec-compare switches to have rwpmatch compare port data and compare times down to the millisecond.
  • Fix a bug that caused rwpmatch to assume every packet would have a corresponding flow.
  • Be more diligent about testing the length and type of packets we read.
• rwtuc change
  • Always print the SiLK header to the output, even when records were read from the input.
• flowcap fix
  • Fix a bug in flowcap that caused it to process data from only the final sensor listed in the sensor-configuration file.
  • Fix bugs in the flowcap control script.
• File relocation
  • The man page sensorconf.5 has been renamed sensor.conf.5.
  • The source POD for man pages has moved from src/APP/doc/APP.pod to src/APP/APP.pod

SiLK-0.10.3 Release, 2006-Nov-15

• Fix a major bug in rwbagbuild that caused rwbagbuild to ignore every other line of its input.
• Fix a bug in the prefixmap (pmap) support that caused rwsort to crash when attempting to sort using fields defined in a pmap.
• Fix syntax errors in the rwfpd script that runs rwflowpack. These errors were invoked when the compression was not set or when the name of the script included a sensor-name suffix.
• Add a --no-file-locking switch to rwflowpack. With this switch, rwflowpack will not attempt to get a write lock when writing flows to data files. This switch is required for rwflowpack use filesystems that do not support file locking. During normal operation multiple rwflowpacks should never attempt to write to the same file; the use of advisory locks is not strictly necessary, but it provides protection during unusual circumstances.
• Modify rwflowpack so that when it encounters a disk error (unable to open file, obtain a lock, write the flow, etc) when trying to write a flow, it stops processing flows for that probe. If all probes encounter disk errors, rwflowpack will exit.
• Fix a communication issue between flowcap and rwflowpack: on slow and noisy networks, the ACK which rwflowpack sends to flowcap indicating that it has received a file could be lost. Since flowcap never received the ACK, it would resend the same file to rwflowpack thinking the first attempt had failed. rwflowpack would store both files, resulting in duplicate flows in the packed data. rwflowpack now stores the name of the most recent file it received. If it receives a file with the same name, the second file is ignored.
• Fix a bug related to the sensor.conf file; the growth factor for an array was too small which caused rwflowpack to abort.
• Fix a bug in parsing time ranges when fractional seconds were present.
• Ensure that compressing flows with the LZO compressor always produces the same binary output by clearing the temporary buffer that is passed into LZO.

SiLK-0.10.0 Release, 2006-Oct-06

• There is a new Analysts' Handbook: Using SiLK for Network Traffic Analysis. This document provides a tutorial on learning the SiLK tools and describes doing analysis with the tools. The manual pages that used to be in that document have been moved into a separate document: The SiLK Reference Guide.
• The SiLK packing tools now support reading IPFIX records generated by the YAF Flow Sensor (http://tools.netsa.cert.org/yaf/). YAF must be installed prior to configuring SiLK.
• When used with YAF, SiLK supports additional fields for dealing with TCP data: The flags on the first packet on the flow are stored separately from the flags on the other packets in the flow. In addition, when a TCP session is broken into multiple flows, the flows are specially marked.
• SiLK now supports using an external compression library to further compress the "data" section of files, while leaving the "header" of the file uncompressed. This compression is available on SiLK Flow files, as well as IPsets and Bags. The supported compression methods are "none", "zlib", and "lzo1x", subject to library availability. Most tools allow one to specify the compression. The default compression is set when the 'configure' script is run (--enable-output-compression).
• The logging library has been rewritten, and now supports syslog(3). Logging messages can also be written to the standard error. "Legacy" logging is still supported (SiLK can still write its log files in a directory and rotate the files), but note that the format of log messages has changed. Also, rwflowpack will no longer automatically include the value passed to --sensor-name switch as part of the log file name and PID file name. (The rwfpd init script works around this; see the SiLK Installation Handbook.)
• For people upgrading from previous releases, note that the list of sensors has been moved from silk_site_generic.h to generic_sensors.h. Also note that the macros around the sensor list have changed; please edit carefully. See the SiLK Installation Handbook.
• A new library, libsksetbag, contains the functions to manipulate IPsets and Bags. libiptree has been removed; use libsksetbag instead.
• Additional manual pages have been added.
• Additional changes:
  • rwptoflow: does a better job of checking the validity of its input; has plug-in support; new switches allow it to produce "pass" and "fail" streams of pcap data and/or print statistics
  • rwsort: when it receives no input, it now produces a SiLK Flow file with no readers (only a header). Previously it would produce a completely empty file
  • rwfileinfo: output changed to include new compression method
  • flowcap: added a switch to manually set the ack timeout, which is useful on slow networks.

SiLK-0.9.10 Release, 2006-Aug-23

• Critical bug fix
  • Fix a byte-swapping bug in FT_RWWWW V3 records. When converting an rwRec from or to this format and where the conversion included a byte-swap, the record would be corrupted. As long as all SiLK data was handled in the machine's native byte order, the bug would not manifest itself (the initial read of the NetFlow data was/is handled correctly, so data on little endian (not network byte order) machines is correct so long as it has always remained on little endian machines).
    
    The bug corrupted data, resulting in any of these behaviors: the source and destination ports could be swapped, the service (web-side) port could be incorrect, the TCP flags could be incorrect, the packet and byte counts could be high (64 times higher than they should be), and the millisecond times could be wrong.
• Potential Incompatibilities
  • When using SiLK flow records in contexts that do not use the millisecond field, truncate the millisecond value instead of rounding.
  • rwbagcat, rwbagtool, rwcat: When file names are listed on the command line, do not attempt to read data from the standard input unless the user explicitly uses "stdin" as the name of an input file. This change is required to allow the tools to work with cron(1).
  • rwflowpack (sensor.conf): Allow a comma to occur between the IP addresses in an ipblock list. This means that a comma cannot occur within the wildcard IP address, but it is believed few people were using this functionality.
  • rwflowpack: minor log message changes; changed the log rotation hour to 00:00; modified the umask() of log files
• New feature: Address Type Plug-in (libaddrtype.so)
  • Support for partioning by or displaying the address type requires libaddrtype.so to exist in the $SILK_PATH/lib directory and the "address_types.pmap" file to exist in the $SILK_PATH/share/silk or $SILK_PATH/share directory.
  • To create this binary "address_types.pmap" file, first list CIDR blocks in a text file (my-ips.txt) and label each as "non-routable", "internal" or "external" (any address that is not listed in the file is considered "external"), and then run the commands:
    
    rwpmapbuild -i my-ips.txt -o address_types.pmap
    
    For the best results with the pmap code, the CIDR blocks should be as large as possible. One one to convert a list of IPs (ips.txt) into a list of large CIDR blocks (cidr.txt) is to run:
    
    rwsetbuild ips.txt stdout | rwsetcat --cidr > cidr.txt
  • For more information, see the rwpmapbuild man page and the man pages of rwfilter, rwcut, rwsort, and rwuniq.
• New feature: Prefix Map Plug-in (libpmapfilter.so)
  • Experimental creation and use of the user's own prefix maps (pmaps) for partitioning (rwfilter), sorting (rwsort), counting (rwuniq), and display (rwcut, rwuniq) is provided. The interface is still considered experimental and is subject to change.
  • The rwpmapbuild tool reads a text file and builds a pmap file that can be used by the tools. This file can relate IPs or Port/Protocol pairs to some attribute (this is how the country code and addrtype pmaps work).
  • For details, see the rwpmapbuild and libpmapfilter man pages.
• New feature: Record Partitioning via IP-Port Pairs (libipport.so)
  • The --ipport-any switch to rwfilter (provided by the libipport.so plug-in) will pass a record if its source IP and port or its destination IP and port are listed in the named text file.
  • To use this plug-in, one creates a text file where each line contains a single IP address (either in dotted-decimal notation or as an integer), whitespace, and a list of ports of interest for that IP. The port list can be a single number (80), a range of numbers ("6000-6100"), or comma-separated list of numbers and ranges ("6000-6100,80"). The file may also contain blank lines and comments; comments begin with the "#" character and continue to the end of the line.
  • Support in rwfilter for partitioning records by IP-port pairs requires libipport.so to exist in the $SILK_PATH/lib directory.
• Improved sorting
  • rwsort now supports getting fields from run-time plug-ins, like rwcut and rwuniq.
  • When merging multiple temp-files, rwsort now attempts to open them all and merge them in one step, considerably reducing the I/O overhead of the merge sort.
• Better support for ICMP data
  • rwfilter: new switches allow for filtering by the ICMP type and code (--icmp-type, --icmp-code)
  • rwcut, rwsort, rwuniq: A new "icmpTypeCode" value to the --fields switch is allowed. When this value is present, the ICMP type and code will be used as part of the key when sorting (rwsort) and counting (rwuniq), and it will be displayed (by rwcut and rwuniq) in separate columns labeled 'iType' and 'iCode' (which in columnar output will shorted to 'iTy' and 'iCo'). The --icmp-type-and-code switch on rwcut is still maintained for backwards compatibility, but its use is deprecated.
  • rwstats: Supports using the ICMP type and code as a key with the --icmp switch.
• Configuration and Build System Changes
  • In preparation of using the GNU AutoTools, we've made major changes to build and configure system that bring us more in-line with the AutoTools. Note that the 'release', 'debug', and 'profile' targets have gone away. Use the --enable-debugging and --disable-optimization switches to configure for a fully debuggable binary. See configure --help to see the full list of new options.
• Miscellaneous Improvements
  • rwcount: Add a new value to the --load-scheme switch that will weigh the values assigned to each bin by the number of seconds the flow spent in the bin.
  • rwfilter: new switch to filter on a negative next-hop IP (--not-next-hop-id)
  • rwfilter: Filtering by IPsets is now supported directly in the application itself. Previously, this was handled by a plug-in.
  • flowcap: There is a new version of the flowcap file format, 5. Version 5 is identical to version 3, save for the fact that the input and output interface fields have been expanded to 16 bits.
  • rwcut, rwsort, rwuniq: Provide numerical identifiers for fields (--fields switch) that hadn't had any previously.
• Bug fixes
  • rwgroup: Fix several bugs, the majority of which have to do with the interaction between summarization and other actions.
  • rwflowpack: Use fseeko() to fix an issue when writing large files on Solaris
  • rwfilter: Fix a crash that would occur when using a combination of the switches --dynamic-library --pass for certain dynamic libraries
  • rwmatch: Several bug fixes.
  • rwstats: Fix a bug that would cause rwstats to crash when attempting to compute the top-N when no records were read as input.
  • rwtuc: Fix a bug that occurred when the user provided the --fields switch and a title line was present
  • rwuniq: Fix a display bug by using the width of the value (versus the title) for setting width of columns that we get from plug-ins.
  • rwuniq: Zero out the record prior to output to avoid getting random data values in the millisecond fields. These random values were affecting the values in the time fields.
  • libflowsource: Fix a bug that prevented it from building when used with certain parser generators.

SiLK-0.9.5 Release, 2006-May-08

• New packing support: flowcap
  • The flowcap daemon allows the collection of flow data and the packing and storage of this flow data to occur on separate machines.
  • To use flowcap, the LZO real-time data compression library must be installed. If configure does not find the LZO library, flowcap will not be built.
  • Compilation and use of flowcap is optional.
• Improvements and significant changes to rwflowpack:
  • Splitting by IP address: Instead of using your router's SNMP interfaces to split traffic into inbound and outbound, rwflowpack can now split data by CIDR block.
  • rwflowpack now requires configuration via a separate sensor.conf file.
  • Many of rwflowpack's arguments have changed.
  • rwflowpack's control script, rwfpd, has been split into two parts.
• New local timezone support: Pass the --enable-localtime switch to the configure script to use the local timezone in time input and output. Without this switch, the tools will use UTC. (Data files continue to be stored in UTC.)
• Format of printed timestamps has changed, the new format is 2006/05/08T15:36:53.123. To enable the previous format by default, pass the --enable-legacy-timestamp switch to configure. The printed timestamp format can be set per invocation via the --legacy-timestamps switch.
• The tools that handle IPset files have been renamed. The old names are still supported for this release.
  • rwsetbuild replaces buildset
  • rwsetcat replaces readset
  • rwsetintersect replaces setintersect
  • rwsetunion replaces rwset-union
• New tool rwtuc: the text utility converter does the reverse of rwcut---it reads textual input and generates binary SiLK flow data from it.
• Manual pages are now included. Additional improvements to the documentation.
• Improvements to rwuniq:
  • Supports computing counts of unique source or destination IPs for small input sets; the memory requirements to support these counts can grow quickly.
  • Can be used with run-time plug-ins.
• Improvements to rwbagtool: Less memory is used during merging of multiple Bag files, and some recursive routines have been rewritten to reduce memory and increase speed.
• Changes to rwsetcat and rwbagcat: The output of the --network-structure switch has changed.
• For tools that produce textual output, columnar output and column separator can be controlled separately. These tools all support the --delimited switch; the former --delimiter switch which some tools supported is deprecated.
• Improvements to rwappend: Now supports "appending" to a nonexistent file. Restrictions on the types of files that rwappend supported have been removed.
• Configuration for multiple sites is easier, though the choice of which site to build for must still be made when you run the configure script.
• Significant rearrangement of the source code tree.

SiLK-0.8.2 Release, 2005-Nov-29

• Fix bug where the pthreads library was not being linked into rwflowpack
• Note: Options to configure script have changed. configure now does a better job (hopefully) of testing for libraries
• Most tools will now invoke a pager to page the output. Use the SILK_PAGER environment variable to override PAGER, or the --pager switch to override SILK_PAGER. Setting SILK_PAGER to the empty string will disable paging.
• Duplicate packet detection removed from rwptoflow; use rwpdedupe to remove duplicate packets.
• Bug fixes in rwptoflow.
• Bug fixes in rwbagcat.
• Bug fixes in statistics output of readset
• Some column headers have changed; test any supporting scripts you may have.
• rwset can now build multiple sets in a single pass. Use the --sip-file, --dip-file, and --nhip-file switches to create the IP set files.
• rwsort now supports the same fields as rwcut and rwuniq
• rwuniq can now bin the start-time and end-time with the --bin-time switch
• rwstats largely rewritten. New switches (though legacy switches are still supported); added support to rwstats for computing top-N lists based on packet counts or byte counts.
• readset will now read a binary IP set from stdin
• Fix compilation problems on RedHat64

SiLK-0.8.1 Release, 2005-Sep-28

• Bug Fix: Allow tools so write output to /dev/null.

SiLK-0.8 Release, 2005-Sep-26

• New packet-support tools
  • rwptoflow: Create a single-packet SiLK flow record for every record in a tcpdump file.
  • rwpmatch: Use a SiLK Flow file to filter the contents of a tcpdump file
  • rwpcut: Output a tcpdump dump file as ASCII
• New tool rwgroup: Groups multiple records together with a common tag
• New tool rwmatch: Matches records from two files together into a common stream
• New pipe-lining tool rwnetmask: Masks off lower bits of the source and/or destination addresses allowing one to aggregate output by CIDR block
• Support for 16bit SNMP interfaces: Packing and file output formats support the full 16bits of SNMP interface values as exported in NetFlow v5
• Support for 65535 sensors: Sensor ID is now processed and stored in a 16 bit integer
• Millisecond time support: Millisecond precision for start time, end time, and duration in the file output formats. Limited application support to access this field.
• New country-code support: Allow filtering and cutting by an IP's physical location
• Enhancements to rwfilter
  • New --print-volume-statistic switch gives bytes, packet, and flow counts for the passed and failed streams
  • New --any-address and --any-ipset switches allows matching source or destination IP addresses
  • New --nhip-set switch allows matching next-hop IP address
  • New --active-time switch allows printing flows that were active at a particular time
  • New --flags-all switch to allow (yet) another way to specify TCP flags
  • Allow filtering over class and type when reading a file generated by a previous run of rwfilter
• Enhancements to rwsort
  • Remove the previous 50 million record limit by using temporary disk files when RAM is exceeded
  • Enable sorting based on elapsed time
• Enhancements to rwuniq
  • In addition to flow counts, optionally keep totals of bytes and packets, as well as the time range over which the key was active.
  • On out-of-memory, print the bins as counted so far.
• Enhancements to rwcount
  • When --start-epoch is given, use that time as the edge of a bin. This lets you view traffic in 24 hour bins that runs from noon to noon, for example.
  • Be more memory stingy by not creating bins for records that occur before the --start-epoch
  • Accepting flows in any time order (previously assumed flows were close to time-sorted order)
  • Allow --start-epoch switch to take a time string like rwfilter accepts
  • Print file names when --print-files is given
  • Add final delimiter to each line of output
• Enhancements to rwaddrcount: Allow sorting of output records by IP address
• Enhancements to rwcat: New --xargs switch to allowing reading a list of file names; this allows rwcat to accept output from the UNIX find command
• Enhancements to readset: Added switches to print details about the structure of the IPs in the IP-set

SiLK-0.7 Release, 2005-Jan-03

• Critical Update. This version fixes a bug that prevents one from querying data for the new year. Any data you collected is correct; it's just that the tools prevented you querying this data.

SiLK-0.6 Release, 2004-Nov-30

• New binary file format (Bag) that maps IP address to a count of bytes, packets, or flows.
• Tools are included for manipulating these files: rwbag*
• Course filtering (fglob) support removed from all tools except rwfilter.
• New rwflowpack options; previous rwfpd scripts are incompatible with the rwflowpack from this release.
• Additional documentation in analysis handbook and the installation handbook.

SiLK-0.5 Release, 2004-Apr-27

• Added support to rwflowpack for accepting incoming flows from multiple interfaces.
• Fixed bugs in rwswapbytes and rwrandomizeip utilities

SiLK-0.4 Release, 2004-Mar-19

• Critical Update. Public releases of the SiLK Tool Suite prior to this release (SiLK-0.3 and earlier) contained a bug that affected the packing of web records. This bug caused the source and destination ports for web records to be swapped, e.g., web connections from your network to sourceforge.net would show the sourceforge.net web service on a high port and have your client machine on port 80.
• This SiLK-0.4 release fixes that bug, and we've provided a Perl script, rwpatchwww.pl, that will repair files you've packed with previous versions. The rwpatchwww.pl script will also migrate your all of your packed files to Version 2 of the SiLK file format. Release SiLK-0.4 of the SiLK Tools will read files packed either in Version 1 or Version 2 format.

SiLK-0.3 Release, 2004-Feb-06

• Added the rwfpd script that was accidentally omitted from the SiLK-0.2 release.
• Other minor fixes.

SiLK-0.2 Release, 2004-Jan-28

• Critical Update. This version fixes major bugs in the initial release of rwflowpack, including a problem that cause the system to produce corrupted packed data files.

SiLK-0.1 Release, 2003-Dec-22

• Initial public "preview" of the SiLK Analysis Suite and Packing System.