anon is reading about communications protocols.
wants to record these thoughts, somewhere.
-
size of message can be dynamic, and size of size can also be dynamic; e.g. in header, size section could be defined as each bit is twice the size as the previous, size stops when a 0 is reached.
-
then while processing the final one it can be reduced since e.g. if you need 9 bytes, don't need to "waste" 7 (since it doubles 1, 2, 4, 8, and then 16). anon still needs to write the algorithm for the later reduction. perhaps: if first bit (of data block) is 0, then data block is full (and that first bit is "extra" e.g. outside the data such that it can fill the space e.g. "16 bytes worth"). each 1 cuts it in half, until a 0 is reached. actually that should do it; call that "data header" so "data" has complete space allocated.
-
anon is considering a merging between transfer and storage. basically, if the two communicating systems have compatible filesystems, could just send the file contents "as-is" on the disk, rather than converting on sending end, and de-converting on receiving end. (not sure that's enough words for the concept…)
-
anon has added encryption to a protocol so has some experience in this area.
taking 2. above further, let's see how large the envelope would be for a few examples, start with the "9" example above.
header: 111110 (1, 2, 4, 8, 16 bytes, and 0 terminator)
data header: 11110 (from above 16, reduce by half each so 8, 4, 2, finally 1 and 0 terminator; meaning, the final block which was allocated to have 16 maximum, will really only have 1, for a total of 8+1=9)
data: hh hh hh hh hh hh hh hh hh (e.g., 9 bytes, hh being hex values, 00 through FF)
that's 11 extra bits, to send 9 bytes of data, or 11/72 bits (9*8=72) of overhead, or 15.28%.
"1" example:
header: 10 (1, and 0 terminator)
data header: 0
data: hh (1 byte)
that's 3 extra bits to send 1 byte of data, or 3/8 bits of overhead, or 37.5%.
"99" example:
header: 111111110 (1, 2, 4, 8, 16, 32, 64, 128, and 0 terminator)
(note: we want remaining 99-64=35, so there's still going to be some waste, need to change the algorithm to account for this which I'll leave as an exercise to the reader which could be my later self; idea is to use "00" as terminator, and can then "shave down" and "shave up" to the precise number of bytes required; and measure this also, perhaps "wasting" one or more bytes in places may be less expensive than attempting to account for them all, like the "1" example above has more than a third of overhead)
data header: 10 (from above 128, reduce to 64, and 0 terminator; meaning, the final block which was allocated to have 128 bytes, will have 64, for a total of 64+64=128)
data: 99 bytes of actual data, then 29 bytes of "wasted space"
that's 11 bits of overhead, like the "9" example above; but also an extra 29 bytes (232 bits) for 243 bits overhead for 99*8=792 bits sent, or 232/792 = 29.2929%.
Make the protocol change and redo "99":
change is: after a 0, each 1 means "cut in half" until a 0; then, each 1 means what it originally meant e.g. add that amount of space.
header: 111111110 (1, 2, 4, 8, 16, 32, 64, 128, and 0 terminator)
data header: 11011100 (from above 128, reduce to 64, reduce to 32, and 0 to indicate direction change; 1, 2, 4 and we can stop with one byte wasted, and 00 terminator)
data: 99 bytes of actual data, then 1 byte of "wasted space"
that's 17 bits of overhead; and also an extra 1 byte (8 bits) for 25 bits overhead for 99*8=792 bits sent, or 25/792 = 3.16%.
it appears that "dialing it in" might save more space. redo most recent "99" with no bytes wasted:
header: 111111110 (1, 2, 4, 8, 16, 32, 64, 128, and 0 terminator)
data header: 110111010 (from above 128, reduce to 64, reduce to 32, and 0 to indicate direction change; 1, 2, 4 and 0 terminator to add (now total is 64+32+4=100), and finally 1 to remove, and 00 terminator)
data: 99 bytes of actual data
that's 18 bits of overhead; for 99*8=792 bits sent, or 18/792 = 2.27%.
also I can move the "data header" to be included as part of the "header" and just have it walk up, walk down, walk up etc as much as it needs for each. in fact can write a program to tell me how much overhead every number of bytes would require, in a range, to better fine-tune the algorithm; it's been a while, am rusty and this is helping oil those lesser-used joints so thank you for listening.