[netflow-tools] compressed flowd files

[Otto]

Hi. Earlier this year I built a system to process netflow data plans 
(flow-tools) using pthreads in c++. On a dual quad it can process ~6GB 
of compressed flow data in 3 minutes including processing the data plans 
and uploading the results back into an API.  Currently, I'm working on 
the same thing for flowd processing.

It would be cool to have a way to stream data from bz2 directly into one 
of the functions in the flowd code. Like :

int store_get_flow(int fd, struct store_flow_complete *f, char *ebuf, 
int elen)  But not sending it a file pointer. This way, I could have all 
the files compressed when my roll over script runs. So far, I've gone 
ahead without compression, but am unsure how to proceed, or if it's 
easily achievable. Any thoughts ?

Regards,
Otto.