lr_anonymize1

Usage

Overview

lr_anonymize is typically used when sending logs to a responder. See the section on "Sending Anonymized Log Files To A Responder" in the chapter on "Using A Responder" in the Lire User Manual for usage examples.

lr_anonymize reads a file containing emailaddresses, ipnumbers, and hostnames (typically a logfile from an internet service) from stdin, and prints an "anonymized" version of this file to stdout. It dumps the information to "deanonymize" the file, using lr_deanonymize(1), to 5 Berkeley DB databases; the names of the files holding these databases is contructed by concatenating the strings 'inaddr', 'maildomain', 'email', 'ip' and 'domain' to dumpfilestem. dumpfilestem can be e.g. /tmp/dump, ../../var/dump or dump. The db files created silently overwrite stale ones, if found.

The script builds temporary files. By default, these are created in /tmp. One can override this by setting the TMPDIR environment variable.

CONNECTION WITH lr_deanonymize

When running

 $ lr_anonymize dump < log > log.anon
 $ lr_deanonymize dump < log.anon > log.new

then log and log.new have the same content (except for case, check it with diff -i).

Notes

We tried to optimize this script for memoryusage. This has the drawback the script will run for quite some time when anonymizing a big logfile.

We've run the script on a 25 M sendmail logfile.

Typical values in such a case are 2500 K for total amount of physical memory used, and 15m real, 8m user and 22s systime spent on a 64 MB system with a 300 MHz Pentium II processor. de_anonymizing this file took 11m real, 8m user and 17s system time.

We store maildomains in the dumpfile. These are used by lr_deanonymize(1), in case email addresses in our input file reoccur in the file read by lr_deanonymize(1) in split form, i.e. [email protected] occurs as both [email protected] and example.com in the to be deanonymized file.

All dumped objects are casted to lowercase.

Background

For your convenience, we quote a bit of rfc822:

 SPACE       =  <ASCII SP, space>            ; (     40,      32.)
 CTL         =  <any ASCII control           ; (  0- 37,  0.- 31.)
                 character and DEL>          ; (    177,     127.)
 specials    =  "(" / ")" / "<" / ">" / "@"  ; Must be in quoted-
             /  "," / ";" / ":" / "\" / <">  ;  string, to use
             /  "." / "[" / "]"              ;  within a word.
 atom        =  1*<any CHAR except specials, SPACE and CTLs>
 quoted-string = <"> *(qtext/quoted-pair) <">; Regular qtext or
                                             ;   quoted chars.
 word        =  atom / quoted-string
 domain-ref  =  atom                         ; symbolic reference
 domain-literal =  "[" *(dtext / quoted-pair) "]"
 sub-domain  =  domain-ref / domain-literal
 domain      =  sub-domain *("." sub-domain)
 local-part  =  word *("." word)             ; uninterpreted
                                             ; case-preserved
 addr-spec   =  local-part "@" domain        ; global address

and of rfc 2181

 The DNS itself places only one restriction on the particular labels
 that can be used to identify resource records.  That one restriction
 relates to the length of the label and the full name.  The length of
 any one label is limited to between 1 and 63 octets.  A full domain
 name is limited to 255 octets (including the separators).

rfc1123

 However, a valid host name can never
 have the dotted-decimal form #.#.#.#, since at least the
 highest-level component label will be alphabetic.

rfc819

 <domain> ::= <naming-domain> | <naming-domain> "." <domain>
 <naming-domain> ::=  <simple-name> | <address>
 <simple-name> ::= <a> <ldh-str> <let-dig>
 <ldh-str> ::= <let-dig-hyp> | <let-dig-hyp> <ldh-str>
 <let-dig> ::= <a> | <d>
 <let-dig-hyp> ::= <a> | <d> | "-"
 <a> ::= any one of the 52 alphabetic characters A through Z in upper
 case and a through z in lower case
 <d> ::= any one of the ten digits 0 through 9

Examples

A 'logfile' like e.g.

 blaat fkrf 1.2.3.4.in-addr.arpa] [email protected] bla 1 2 3 lj;agas;gag
 blaat 1.2.3.4 fkrf 3.2.3.4.in-addr.arpa] bla 1 www.hotsex.com 2 3 lj;agas;gag 
 [email protected] agagag
 blaat fkrf 4.2.3.4.in-addr.arpa] bla [email protected] www.hotsex.com 
 234.34.2.0 [email protected] 4.2.3.4.in-addr.arpa1 2 3 lj;agas;gag
 blaat fkrf tweede 3.2.3.4.in-addr.arpa] bla 1.2.3.4 1 blablabla.com 
 2 mdcc.cx 
 3 lj;agas;gag

wil get anonymized to

 blaat fkrf 1.0.0.10.in-addr.arpa] [email protected] bla 1 2 3 lj;agas;gag
 blaat 10.0.0.1 fkrf 2.0.0.10.in-addr.arpa] bla 1 1.example.com 2 3 lj;agas;gag 
 [email protected] agagag
 blaat fkrf 3.0.0.10.in-addr.arpa] bla [email protected] 1.example.com 
 10.0.0.2 [email protected] 3.0.0.10.in-addr.arpa1 2 3 lj;agas;gag
 blaat fkrf tweede 2.0.0.10.in-addr.arpa] bla 10.0.0.1 1 2.example.com 
 2 3.example.com 
 3 lj;agas;gag

The dumps will represent something like

 ip 234.34.2.0 10.0.0.2
 ip 1.2.3.4 10.0.0.1
 inaddr 3.2.3.4.in-addr.arpa 2.0.0.10.in-addr.arpa
 inaddr 1.2.3.4.in-addr.arpa 1.0.0.10.in-addr.arpa
 inaddr 4.2.3.4.in-addr.arpa 3.0.0.10.in-addr.arpa
 domain mdcc.cx 3.example.com
 domain blablabla.com 2.example.com
 domain www.hotsex.com 1.example.com
 email [email protected] [email protected]
 email [email protected] [email protected]

Version

$Id: lr_anonymize.in,v 1.5 2006/07/23 13:16:32 vanbaal Exp $