From memic at paniert.org Wed Sep 13 20:56:03 2006 From: memic at paniert.org (memic) Date: Wed, 13 Sep 2006 12:56:03 +0200 Subject: [netflow-tools] logrotation Message-ID: <4507E3C3.10207@paniert.org> hi, i was wondering whats the best solution to rotate the flowd logfiles, im doing it atm with a very simple bash script: # cat /usr/local/bin/flowd-logrotate.sh #!/bin/bash PID=`pgrep -f "flowd: monitor"` /bin/echo $PID DATE=`date +%Y%m%d` /bin/mv /var/log/flow/flowd.bin /var/log/flow/$DATE-flow.bin /bin/kill -HUP $PID with a cron like that: 0 0 * * * /usr/local/bin/flowd-logrotate.sh regards, memic From gijs at looze.net Thu Sep 14 16:02:17 2006 From: gijs at looze.net (Gijs Molenaar) Date: Thu, 14 Sep 2006 08:02:17 +0200 Subject: [netflow-tools] logrotation In-Reply-To: <4507E3C3.10207@paniert.org> References: <4507E3C3.10207@paniert.org> Message-ID: <4508F069.5050107@looze.net> memic wrote: > hi, > > i was wondering whats the best solution to rotate the flowd logfiles, im > doing it atm > with a very simple bash script: > > # cat /usr/local/bin/flowd-logrotate.sh > #!/bin/bash > PID=`pgrep -f "flowd: monitor"` > /bin/echo $PID > DATE=`date +%Y%m%d` > /bin/mv /var/log/flow/flowd.bin /var/log/flow/$DATE-flow.bin > /bin/kill -HUP $PID > > with a cron like that: > > 0 0 * * * /usr/local/bin/flowd-logrotate.sh > In the man page (flowd.8) you can find this: flowd may be terminated by sending it a SIGTERM signal. Upon receipt of a SIGUSR1 flowd will close and reopen its logfile. This is useful for rotating logs. When flowd receives a SIGHUP it will re-read its configuration and re-open its logfile. Some basic runtime statistics will be logged when flowd is signalled with SIGUSR2 or SIGINFO. Sending the HUP signal will work, but maybe some flows are lost (not sure). The best thing is to replace HUP with SIGUSR1. gr, gijs From memic at paniert.org Thu Sep 14 17:33:31 2006 From: memic at paniert.org (memic) Date: Thu, 14 Sep 2006 09:33:31 +0200 Subject: [netflow-tools] logrotation In-Reply-To: <4508F069.5050107@looze.net> References: <4507E3C3.10207@paniert.org> <4508F069.5050107@looze.net> Message-ID: <450905CB.7080300@paniert.org> ah, ok, somehow i have read over this, thx for poiting out. my rotation script now looks like this: #!/bin/bash PID=`pgrep -f "flowd: monitor"` /bin/echo $PID DATE=`date +%Y%m%d` /bin/mv /var/log/flow/flowd.bin /var/log/flow/$DATE-flow.bin /bin/kill -SIGUSR1 $PID are you sure no flows can get lost this way? Gijs Molenaar wrote: > memic wrote: >> hi, >> >> i was wondering whats the best solution to rotate the flowd logfiles, >> im doing it atm >> with a very simple bash script: >> >> # cat /usr/local/bin/flowd-logrotate.sh >> #!/bin/bash >> PID=`pgrep -f "flowd: monitor"` >> /bin/echo $PID >> DATE=`date +%Y%m%d` >> /bin/mv /var/log/flow/flowd.bin /var/log/flow/$DATE-flow.bin >> /bin/kill -HUP $PID >> >> with a cron like that: >> >> 0 0 * * * /usr/local/bin/flowd-logrotate.sh >> > > > In the man page (flowd.8) you can find this: > flowd may be terminated by sending it a SIGTERM signal. Upon > receipt of a SIGUSR1 flowd will close and reopen its logfile. This is > useful for > rotating logs. When flowd receives a SIGHUP it will re-read its > configuration and re-open its logfile. Some basic runtime statistics > will be > logged when flowd is signalled with SIGUSR2 or SIGINFO. > > Sending the HUP signal will work, but maybe some flows are lost (not > sure). The best thing is to replace HUP with SIGUSR1. > > > gr, > > gijs > > From gijs at looze.net Thu Sep 14 19:00:25 2006 From: gijs at looze.net (Gijs Molenaar) Date: Thu, 14 Sep 2006 11:00:25 +0200 Subject: [netflow-tools] logrotation In-Reply-To: <450905CB.7080300@paniert.org> References: <4507E3C3.10207@paniert.org> <4508F069.5050107@looze.net> <450905CB.7080300@paniert.org> Message-ID: <45091A29.4010703@looze.net> memic wrote: > ah, ok, somehow i have read over this, thx for poiting out. > > my rotation script now looks like this: > > #!/bin/bash > PID=`pgrep -f "flowd: monitor"` > /bin/echo $PID > DATE=`date +%Y%m%d` > /bin/mv /var/log/flow/flowd.bin /var/log/flow/$DATE-flow.bin > /bin/kill -SIGUSR1 $PID > > are you sure no flows can get lost this way? Yep. When you move the file, flowd still writes data to the it, because the inode doesn't change. After flowd receives the SIGUSR it directly starts to log to the new file. - gijs From phatbuckett at gmail.com Fri Sep 15 00:30:16 2006 From: phatbuckett at gmail.com (Darren Spruell) Date: Thu, 14 Sep 2006 07:30:16 -0700 Subject: [netflow-tools] logrotation In-Reply-To: <45091A29.4010703@looze.net> References: <4507E3C3.10207@paniert.org> <4508F069.5050107@looze.net> <450905CB.7080300@paniert.org> <45091A29.4010703@looze.net> Message-ID: <839aec700609140730w2c23dce7uced611bf20c6d138@mail.gmail.com> On 9/14/06, Gijs Molenaar wrote: > memic wrote: > > #!/bin/bash > > PID=`pgrep -f "flowd: monitor"` > > /bin/echo $PID > > DATE=`date +%Y%m%d` > > /bin/mv /var/log/flow/flowd.bin /var/log/flow/$DATE-flow.bin > > /bin/kill -SIGUSR1 $PID > > > > are you sure no flows can get lost this way? > > Yep. When you move the file, flowd still writes data to the it, because > the inode doesn't change. > > After flowd receives the SIGUSR it directly starts to log to the new file. Wouldn't the use of an automated log rotation utility like logrotate or newsyslog work well for this as well? DS From gijs at looze.net Fri Sep 15 00:33:27 2006 From: gijs at looze.net (Gijs Molenaar) Date: Thu, 14 Sep 2006 16:33:27 +0200 Subject: [netflow-tools] logrotation In-Reply-To: <839aec700609140730w2c23dce7uced611bf20c6d138@mail.gmail.com> References: <4507E3C3.10207@paniert.org> <4508F069.5050107@looze.net> <450905CB.7080300@paniert.org> <45091A29.4010703@looze.net> <839aec700609140730w2c23dce7uced611bf20c6d138@mail.gmail.com> Message-ID: <45096837.9010707@looze.net> Darren Spruell wrote: > On 9/14/06, Gijs Molenaar wrote: > >> memic wrote: >> >>> #!/bin/bash >>> PID=`pgrep -f "flowd: monitor"` >>> /bin/echo $PID >>> DATE=`date +%Y%m%d` >>> /bin/mv /var/log/flow/flowd.bin /var/log/flow/$DATE-flow.bin >>> /bin/kill -SIGUSR1 $PID >>> >>> are you sure no flows can get lost this way? >>> >> Yep. When you move the file, flowd still writes data to the it, because >> the inode doesn't change. >> >> After flowd receives the SIGUSR it directly starts to log to the new file. >> > > Wouldn't the use of an automated log rotation utility like logrotate > or newsyslog work well for this as well? > just a matter of taste... - gijs From matthew.wycoff at baesystems.com Fri Sep 15 02:06:06 2006 From: matthew.wycoff at baesystems.com (Wycoff, Matthew (US SSA)) Date: Thu, 14 Sep 2006 12:06:06 -0400 Subject: [netflow-tools] v9 -> v5 conversion Message-ID: <6DED202D454D3B4EB7D98A7439218D61070CF9EA@vahqex2.gfgsi.com> Is there a good way to do conversion from netflow v9 to v5? Similar to the redirect function of nfdump, but converting the flows as well. Sincerely, -Matt- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20060914/6d1768c6/attachment.html From mwlucas at blackhelicopters.org Mon Sep 25 06:30:24 2006 From: mwlucas at blackhelicopters.org (Michael W. Lucas) Date: Sun, 24 Sep 2006 16:30:24 -0400 Subject: [netflow-tools] Making flows expire Message-ID: <20060924203024.GA17916@bewilderbeast.blackhelicopters.org> Hi, My major use of softflowd is for generating pretty pictures with CUFlow. As CUFlow is RRD-based, flows that last a long time show up as spikes on the graph. This is increasingly becoming a problem for a variety of administrative (people admin, not tech admin) reasons. I'd like to have softflowd expire each flow after 300 seconds, and restart with a new flow. Yes, this is rather dumb, I know, but I can reassemble these flows more easily than I can explain problems with the pictures. One thought that comes to mind is to do a softflowctl expire-all every five minutes, but this strikes me as rather brute-force. Is there a cleaner way to do this, or am I just stuck with using an ugly hack to implement my ugly requirements? Thanks, ==ml -- Michael W. Lucas mwlucas at FreeBSD.org, mwlucas at BlackHelicopters.org http://www.BlackHelicopters.org/~mwlucas/ Latest book: PGP & GPG -- http://www.pgpandgpg.com "The cloak of anonymity protects me from the nuisance of caring." -Non Sequitur From djm at mindrot.org Mon Sep 25 13:40:33 2006 From: djm at mindrot.org (Damien Miller) Date: Mon, 25 Sep 2006 13:40:33 +1000 (EST) Subject: [netflow-tools] Making flows expire In-Reply-To: <20060924203024.GA17916@bewilderbeast.blackhelicopters.org> References: <20060924203024.GA17916@bewilderbeast.blackhelicopters.org> Message-ID: On Sun, 24 Sep 2006, Michael W. Lucas wrote: > I'd like to have softflowd expire each flow after 300 seconds, and > restart with a new flow. Yes, this is rather dumb, I know, but I can > reassemble these flows more easily than I can explain problems with > the pictures. > > One thought that comes to mind is to do a softflowctl expire-all every > five minutes, but this strikes me as rather brute-force. Is there a > cleaner way to do this, or am I just stuck with using an ugly hack to > implement my ugly requirements? You can set the "maxlife" timeout to achieve exactly this: > maxlife > This is the maximum lifetime that a flow may exist for. All > flows are forcibly expired when they pass maxlife seconds. To > disable this feature, specify a maxlife of 0. so, "softflowd -tmaxlife=300 ..." should do what you want. -d From mwlucas at blackhelicopters.org Mon Sep 25 22:53:41 2006 From: mwlucas at blackhelicopters.org (Michael W. Lucas) Date: Mon, 25 Sep 2006 08:53:41 -0400 Subject: [netflow-tools] Making flows expire In-Reply-To: References: <20060924203024.GA17916@bewilderbeast.blackhelicopters.org> Message-ID: <20060925125341.GA22694@bewilderbeast.blackhelicopters.org> On Mon, Sep 25, 2006 at 01:40:33PM +1000, Damien Miller wrote: > On Sun, 24 Sep 2006, Michael W. Lucas wrote: > > > I'd like to have softflowd expire each flow after 300 seconds, and > > restart with a new flow. Yes, this is rather dumb, I know, but I can > > reassemble these flows more easily than I can explain problems with > > the pictures. > > > > One thought that comes to mind is to do a softflowctl expire-all every > > five minutes, but this strikes me as rather brute-force. Is there a > > cleaner way to do this, or am I just stuck with using an ugly hack to > > implement my ugly requirements? > > You can set the "maxlife" timeout to achieve exactly this: > > > maxlife > > This is the maximum lifetime that a flow may exist for. All > > flows are forcibly expired when they pass maxlife seconds. To > > disable this feature, specify a maxlife of 0. > > so, "softflowd -tmaxlife=300 ..." should do what you want. You know, I must be blind. Thank you for reading the man page at me. :-) ==ml -- Michael W. Lucas mwlucas at FreeBSD.org, mwlucas at BlackHelicopters.org http://www.BlackHelicopters.org/~mwlucas/ Latest book: PGP & GPG -- http://www.pgpandgpg.com "The cloak of anonymity protects me from the nuisance of caring." -Non Sequitur From stan at holmes.nl Wed Sep 27 17:18:44 2006 From: stan at holmes.nl (S.M.Madge) Date: Wed, 27 Sep 2006 09:18:44 +0200 Subject: [netflow-tools] pfflowd 0.7 will not run as daemon Message-ID: <1159341524.3265.11.camel@localhost> Hi all, Somehow pfflowd 0.7 doesn't want to run as a daemon on an openbsd 3.9 box. It does run with the debug option (-D) enabled, but that generates a huge amount of extra output/syslog stuff etc. Any ideas? From stan at holmes.nl Wed Sep 27 23:52:05 2006 From: stan at holmes.nl (S.M.Madge) Date: Wed, 27 Sep 2006 15:52:05 +0200 Subject: [netflow-tools] pfflowd 0.7 will not run as daemon In-Reply-To: <20060927133022.GA30494@yamamoto.kisoku.net> References: <1159341524.3265.11.camel@localhost> <20060927133022.GA30494@yamamoto.kisoku.net> Message-ID: <1159365125.2441.17.camel@localhost> I found the problem already by looking at /var/log/messages: i didn't create a _pfflowd user and as such it couldn't chroot....maybe an idea to put that in the accompanying README file in future releases. On Wed, 2006-09-27 at 22:30 +0900, Mathieu Sauve-Frankel wrote: > On Wed, Sep 27, 2006 at 09:18:44AM +0200, S.M.Madge wrote: > > Hi all, > > > > Somehow pfflowd 0.7 doesn't want to run as a daemon on an openbsd 3.9 > > box. It does run with the debug option (-D) enabled, but that generates > > a huge amount of extra output/syslog stuff etc. > > > > Any ideas? > > How are we supposed to reproduce your problem when you provide absolutely > no details about your environment other than which version of openbsd you > are running it on ? > -- Met vriendelijke groeten, Stanford Madge Nederlands Forensisch Instituut - Digitale Technologie T +31(0)70 888 6 519 F +31(0)70 888 6 559 E s.madge at nfi.minjus.nl / stan at holmes.nl From msf at kisoku.net Wed Sep 27 23:30:22 2006 From: msf at kisoku.net (Mathieu Sauve-Frankel) Date: Wed, 27 Sep 2006 22:30:22 +0900 Subject: [netflow-tools] pfflowd 0.7 will not run as daemon In-Reply-To: <1159341524.3265.11.camel@localhost> References: <1159341524.3265.11.camel@localhost> Message-ID: <20060927133022.GA30494@yamamoto.kisoku.net> On Wed, Sep 27, 2006 at 09:18:44AM +0200, S.M.Madge wrote: > Hi all, > > Somehow pfflowd 0.7 doesn't want to run as a daemon on an openbsd 3.9 > box. It does run with the debug option (-D) enabled, but that generates > a huge amount of extra output/syslog stuff etc. > > Any ideas? How are we supposed to reproduce your problem when you provide absolutely no details about your environment other than which version of openbsd you are running it on ? -- Mathieu Sauve-Frankel