[netflow-tools] compressed flowd files
mworld at twbc.net
Wed Dec 7 05:12:05 EST 2011
Hi. Earlier this year I built a system to process netflow data plans
(flow-tools) using pthreads in c++. On a dual quad it can process ~6GB
of compressed flow data in 3 minutes including processing the data plans
and uploading the results back into an API. Currently, I'm working on
the same thing for flowd processing.
It would be cool to have a way to stream data from bz2 directly into one
of the functions in the flowd code. Like :
int store_get_flow(int fd, struct store_flow_complete *f, char *ebuf,
int elen) But not sending it a file pointer. This way, I could have all
the files compressed when my roll over script runs. So far, I've gone
ahead without compression, but am unsure how to proceed, or if it's
easily achievable. Any thoughts ?
More information about the netflow-tools