Yes, I did spend time on this

You would have done well with this kind of thinking in the mid-80s when you needed to fit code and data into maybe 16k!

As long as you were happy to rewrite it in Z80 or 6502.

Another alternative is arithmetic encoding. For instance, if you only needed to store A-Z and space, you code those as 0-26, then multiply each char by 1, 27, 27^2, 26^3 etc, the add them.

To unpack them, divide by 27 repeatedly, the remainder each time is each character. It's simply covering numbers to base-27.

It wouldn't make much difference from using 5 bits per char for a short run, though, but could be efficient for longer strings, or if encoding a smaller set of characters.

After all.. Why not?

Why shouldn’t I ignore the 100+ cultures whose character set couldn’t fit into this encoding?

Oh god that switch statement. Here, let me come up with something better:

if (pChar >= 'a' && pChar <= 'z') {
  return pChar - 'a' + 10;
} else if (pChar == ' ') {
  return 36;
} else if (pChar == '.'){
  return 37;
}
return 0;

Back in the day those tricks were common. Some PDP-11 OS's supported a "Radix-50" encoding (50 octal = 40 decimal) that packed 3 characters into a 16 bit word (40 codepoints=26 letters, 10 digits, and a few punctuation). So you could have a 6.3 filename in 3 words.

CPU still pulls a 32kb block from RAM...

Interesting idea but type conversion and parsing is much more slower than wasting 1 byte. Nowadays memory is "free" and the main issue is the execution speed.

I'm not sure if this is the right setting for technical discussion, but as a relative elder of computing I'd like to answer the question in the image earnestly. There's a few factors squeezing the practicality out of this for almost all applications: processor architectures (like all of them these days) make operating on packed characters take more operations than 8 bit characters so there's a speed tradeoff (especially considering cache and pipelining). Computers these days are built to handle extremely memory demanding video and 3d workloads and memory usage of text data is basically a blip in comparison. When it comes to actual storage and not in-memory representation, compression algorithms typically perform better than just packing each character into fewer bits. You'd need to be in a pretty specific niche for this technique to come in handy again, for better or for worse

Oh god, please don't. Just use utf8mb4 like a normal human being, and let the encoding issues finally die out (when microsoft kills code pages). If space is of consideration, just use compression, like gz or something.

You could save 0.64 bit per char more if you actually treated you output as a binary number (using 6 bits per char) and didn't go through the intermediary string (implicitly using base 100 at 6.64 bits per char).
This would also make your life easier by allowing bit manipulation to slice/move parts and reducing work for the processor because base 100 means integer divisions, and base 64 means bit shifts. If you want to go down the road of a "complicated" base use base 38 and get similar drawbacks as now, except only 5.25 bits per char.

Not useless -- you have a future in tiny, embedded systems.

I do this kind of thing everyday as a firmware engineer :)

It’s all fun and games until the requirement changes and you need to support uppercase letters and digits as well.

Does the efficiency of storage actually matter? Are you working on a constrained system like a microcontroller? Because if you’re working on regular software, supporting Unicode is waaaaaaaaaaay more valuable than 20% smaller text storage.

unsigned int turn_char_to_int(char pChar)
{
    switch(pChar)
    {
    case 'a':
        return 10;
    case 'b':
        return 11;
    case 'c':
        return 12;
    case 'd':
        return 13;
    case 'e':
        return 14;
    case 'f':
        return 15;
    case 'g':
        return 16;
    case 'h':
        return 17;
    case 'i':
        return 18;
    case 'j':
        return 19;
    case 'k':
        return 20;
    case 'l':
        return 21;
    case 'm':
        return 22;
    case 'n':
        return 23;
    case 'o':
        return 24;
    case 'p':
        return 25;
    case 'q':
        return 26;
    case 'r':
        return 27;
    case 's':
        return 28;
    case 't':
        return 29;
    case 'u':
        return 30;
    case 'v':
        return 31;
    case 'w':
        return 32;
    case 'x':
        return 33;
    case 'y':
        return 34;
    case 'z':
        return 35;
    case ' ':
        return 36;
    case '.':
        return 37;

    }
}

Are you a monster or just stupid?

We have a binary file that has to maintain compatibility with a 16 bit Power Basic app that hasn't been recompiled since '99 or '00. We have storage for 8 character strings in two ints , and 12 character string in two ints and two shorts.

dammit yesterday was too long i thought this was a dnd joke at first

I was hopping it was somehow badly implemented in python and each char ended up occupying 2Gb

Yes I know, the code is probably bad, but I do not care

That's why we love it.

In typical C fashion, there's undefined behavior in turn_char_to_int. xD

I mean… you’d get better results for large data sets by just using a known compression algorithm. This is only viable for situations where you only have a small amount of data, enough computation to run this conditional, but not enough computation to run compression/decompression.

That's bootleg gzip.

Mostly because compilers do this kind of stuff if you optimize for space, iirc. Not that you should never do it or something, but it kinda looks like premature optimization to me.

My colleague said he didn’t see the point in storing enums as shorts or bytes instead of a full word, so I retaliated by storing them in their string form instead, arguing that it’ll be compressed by the db engine.

And here I was wasting time with bit fields to make my bools smaller.

At first I thought, "How are they going to compress 256 values, i.e. 1 Byte sized data, by "rearranging into integers"?

Then I saw your code and realized you are discarding 228 of them, effectively reducing the available symbol set by about 89%.

Speaking of efficiency: Since chars are essentially unsigned integers of size 1 byte and 'a' to 'z' are values 97 to 122 (decimal, both including) you can greatly simplify your turn_char_to_int method by just substracting 87 from each symbol to get them into your desired value range instead of using this cumbersome switch-case structure. Space (32) and dot (46) would still need special handling though to fit your desired range.

Bit-encoding your chosen 28 values directly would require 5 bit.

I'm a simple man. Here's my list of variable names:

Var1

Var2

Var3

Var4

Var5

...

Use strings for everything and use a single universal method to convert some to floats only when you absolutely have to.

C lets you do this by putting text in single quotes:

int foo = 'Abcd';

works

I have a coworker who does stuff like this and it's always low-benefit optimizations that cost the team time to interface with - but I do still kind of love it

Hey, this is awesome for saving space when writing things to NFC tags! Every bit still matters with those suckers

in the same vein (storing more data in less bits) you should check out tagged pointers as well!

I don't think that's a useless implementation at all. code looks relatively clean, and it definitely has its uses in the embedded systems world.

now use the intchar to store prefixes for a smart string , or pack them to make a simd optimized b tree with 40 string prefixes per node instead of 32

It is neither useless nor funny. It's optimization for storage capacity. If everyone in the world put in that level of effort into compression, computer storage and processing would actually be faster than the previous generation.