> At that point, you don't know the encoding, so the only thing available to you is heuristics (including needing to guess the byte order).
That's emphatically not the case though. I explained how you could handle it here without requiring BOM or byte order knowledge or heuristics: https://news.ycombinator.com/item?id=20198208
> Either way, I don't think it's accurate to claim that ripgrep doesn't support UTF-16.
Having UTF-16 text in a file doesn't imply the file has have a BOM, and when I tried it rga didn't work on UTF-16 that didn't have a BOM. If that's still "ripgrep supports UTF-16" in your view then I'm not sure how else to word it, but the wording is hardly my concern. At the end of the day I was just trying to convey a particular fact, not argue over its wording.
> I explained how you could handle it here without requiring BOM or byte order knowledge or heuristics:
Yes, that's an absurd amount of development effort and would result in a serious performance regression. (To the point that it's likely nobody would use ripgrep at all, so your approach would need to be put behind a flag, which seriously hinders the feature since it's no longer automatic.) Moreover, that only covers match detection, but does not actually cover output. Once you find the match, you have to determine how to print it, and the device you're printing to very likely does not support things like UTF-32 or even UTF-16 in many cases. Moreover, there are many operations that ripgrep does in a post-processing step (like limiting the output to a certain number of characters per line) that require knowing the presumed encoding (which is always UTF-8 by that point, since the data will have been transcoded to UTF-8 if UTF-16 were detected).
> UTF-16 doesn't require BOMs
You cannot decode UTF-16 without knowing its byte order. The BOM tells you that. If there is no BOM, then you need to get the byte order from some other source (or guess it). ripgrep requires the user to tell it what it is. This seems entirely reasonable to me, especially since most or all UTF-16 files I've seen include a BOM. Notably, ripgrep's support for UTF-16 is good enough for VS Code, which has a pretty sizable Windows user base.
> your view then I'm not sure how else to word it, but the wording is hardly my concern. At the end of the day I was just trying to convey a fact, not argue over its wording or semantics.
At the end of the day, my concern is to correct misleading claims about what ripgrep can and can't do. ripgrep clearly has support for UTF-16, and this is actually one of its marquee features that sets it apart from other search tools. For example, grep doesn't (and literally can't) support UTF-16 at all. The only way to search UTF-16 encoded files with grep is to transcode the file to UTF-8 first or to set the locale to C, and search for the binary encoding directly. ripgrep does a lot better than that, so to lump it in with "pretty much every text search tool" is pretty misleading from my perspective.
I'm not saying you were intentionally misleading anyone. What I'm saying is that I'm trying to correct something that I saw as misleading. Criticism is totally fair, but criticism of criticism should be fair game too. I totally appreciate that we shouldn't take these things too personally, but that cuts both ways. I wasn't saying you were trying to be misleading; I was trying to point out an inaccuracy. Given that ripgrep is my project, and myths spread easily, I try to stay on top of that.
> If you don't care or it's too much work
I mean, I do care. Windows users and the prevalence of UTF-16 is why I added the automatic transcoding in the first place. But it's not just that it's too much work; as I said, the performance regression would be so serious that people would literally stop using ripgrep unless it was disabled by default. (In addition to the fact that printing the results puts you in a precarious situation.)
That's emphatically not the case though. I explained how you could handle it here without requiring BOM or byte order knowledge or heuristics: https://news.ycombinator.com/item?id=20198208
> Either way, I don't think it's accurate to claim that ripgrep doesn't support UTF-16.
Having UTF-16 text in a file doesn't imply the file has have a BOM, and when I tried it rga didn't work on UTF-16 that didn't have a BOM. If that's still "ripgrep supports UTF-16" in your view then I'm not sure how else to word it, but the wording is hardly my concern. At the end of the day I was just trying to convey a particular fact, not argue over its wording.