• Knusper@feddit.de
    link
    fedilink
    arrow-up
    10
    ·
    1 year ago

    That Rust function returns the number of codepoints, not the number of graphemes, which is rarely useful. You need to use a facepalm emoji with skin color modifiers to see the difference.

    The way to get a proper grapheme count in Rust is e.g. via this library: https://crates.io/crates/unicode-segmentation

    • Djehngo@lemmy.world
      link
      fedilink
      arrow-up
      9
      ·
      1 year ago

      Makes sense, the code-points split is stable; meaning it's fine to put in the standard library, the grapheme split changes every year so the volatility is probably better off in a crate.

      • Knusper@feddit.de
        link
        fedilink
        arrow-up
        7
        ·
        1 year ago

        Yeah, although having now seen two commenters with relatively high confidence claiming that counting codepoints ought be enough…

        …and me almost having been the third such commenter, had I not decided to read the article first…

        …I'm starting to feel more and more like the stdlib should force you through all kinds of hoops to get anything resembling a size of a string, so that you gladly search for a library.

        Like, I've worked with decoding strings quite a bit in the past, I felt like I had an above average understanding of Unicode as a result. And I was still only vaguely aware of graphemes.

        • Turun@feddit.de
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          For what it's worth, the documentation is very very clear on what these methods return. It explicitly redirects you to crates.io for splitting into grapheme clusters. It would be much better to have it in std, but I understand the argument that Std should only contain stable stuff.

          As a systems programming language the .len() method should return the byte count IMO.

          • Knusper@feddit.de
            link
            fedilink
            arrow-up
            2
            ·
            1 year ago

            The problem is when you think you know stuff, but you don't. I knew that counting bytes doesn't work, but thought the number of codepoints was what I want. And then knowing that Rust uses UTF-8 internally, it's logical that .chars().count() gives the number of codepoints. No need to read documentation, if you're so smart. 🙃

            It does give you the correct length in quite a lot of cases, too. Even the byte length looks correct for ASCII characters.

            So, yeah, this would require a lot more consideration whether it's worth it, but I'm mostly thinking there'd be no .len() on the String type itself, and instead to get the byte count, you'd have to do .as_bytes().len().