Having dealt with custom binary formats for large amounts of data quite a bit, I tend to recommend against doing that if standard options can instead be used: there are a large number of pitfalls in going binary, both when writing the data, as well as when reading it back;

Hi Abhishek, On Fri, 2019-12-27 at 20:11 +0530, Abhishek Goyal wrote: Is that so? Color me surprised! While certainly any dictionary-based compressor could find the bytes that make up the individual digits and compress them to an average of a little less than 4b, that'd still be worse than the 8b you need to represent any number 0-255, for example. And if your dictionary allows for variable-length words, like an LZ(W) kind of algorithm, the compression ratio should saturate pretty early. Now, I haven't worked with the specific text data coming out of rtl_power, so I'd be very interested in the results! Bill, could you compress a few of your textual rtl_power output files (using gzip --best, and maybe xz) for us and tell us the how many numbers were in the original files and how many bytes are the resulting files in length? (zstd: would be very interesting to have a detached dictionary, because I presume the dictionary overhead to be non-negligible with large numbers of smaller observation files) (BTW, tar is the worst format under the sun to compress many small files; it pads every file to 512B; of course, zeros compress nicely, but suddenly your shortest codeword is a useless padding symbol and that has a measurable compressed file size effect) Indeed, that's what I'd have to bring forward: A "compressor" based on simply converting the tabular text data to binary format would be not so far away, worst case, from an actual entropy encoder, but allow for random seeks, AND be faster. I honestly don't see the downsides of that! Yep; or other formats. GNU Radio, for example, simply uses raw binary numbers packed end-to-end; there's the SigMF project which strives to provide metadata (sample format, acquisition time, and other parameters) in a separate file. It's JSON lying next to your data file. Whether or not that's useful to you... Yeah, but a protobuf that's mostly a buffer of ints really is only binary numbers right after each other, plus a header that you define yourself. It's a good idea to let some library like protobuf handle that, I agree! Not quite sure how well sqlite handles compression of BLOBs, or are you suggesting you insert samples as values individually? By the way, I think "so much data it becomes a burden to my server" is actually not covered by what sqlite is designed to do. There might be more optimized databases for that. if each "row" of data, in case a few bytes got corrupted on disk, thus allowing the rest to still be recovered) Which should be no problem, seeing that rows are fixed-length, reading code still remain able to read the older format) Important point, imho, but this feels like a one-off format, so really, might be a bit of overengineering. Anyways, never hurts to simply have a header field that says "version". Do that! intended to write, typically there will be no error messages from the code and no crashes - just wrong and misleading results due to miswritten/misread data), etc. In my experience, writing textual data is way harder to keep consistent, due to the non-fixed amount of bytes required per word. Best regards, Marcus

I will be starting by defining by format then writing a converter to convert between text csv and the binary format. I will also try to write a stream based converter that you can pipe the output of rtl_power into the write the binary format directly. I figure this is more universal then trying to make a binary format in the application itself. On Thu, Jan 2, 2020 at 7:30 PM Hayati Ayguen <h_ayguen(a)web.de> wrote: