Multibyte characters ignored on OS X

[cpsdqs]

Characters with large code points are ignored for some reason. Possibly because they use multiple bytes.

[miestasmia]

I'm not sure what you're trying to write but afaik u+F09F does not exist.

[cpsdqs]

Each of the first six characters that aren't a space use four bytes, not two: e.g. F0 9F 90 A1, which is u+1F421 (blowfish)

[miestasmia]

Sorry, my bad. I'm afraid I can't reproduce it.

[miestasmia]

@cpsdqs Can you please confirm if this is still an issue in the latest version? Some of the internals have been changed.

[cpsdqs]

sure is

[miestasmia]

I still cannot reproduce this, which is leading me to think it's a Mac-only issue. Can you try run this on a Linux machine by any chance?

[miestasmia]

Additionally, could you try prepending export PYTHONIOENCODING=utf-8 prior to running unilookup on OS X?

[cpsdqs]

Doesn't work either ¯\_(ツ)_/¯
['D83C', 'DF29', '000A'] vs. ['1F329', '000A']
seems to be an issue with python itself

[miestasmia]

Could you try manually setting the input string instead of reading from stdin so we can try figure out where the issue arises?

[cpsdqs]

replacing sys.stdin on line 31 with ['🌩'] and adding # coding=utf-8 on line 2 (because it doesn't even run without: SyntaxError: Non-ASCII character '\xf0' in file ./unilookup on line 31, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details) doesn't really work

[miestasmia]

Hm, can you try add print char in the second for loop (at :33)?

[cpsdqs]

prints:

���
���

(all of them u+FFFD) … (side note: unilookup works fine for echo '���' | unilookup)

[miestasmia]

Okay so that seems to be where it breaks, because I'm getting the individual characters on Linux. I'll look into it

[miestasmia]

From what I've been able to find this is an issue with how Python 2 (on some platforms) handles unicode. The solution here would be to read the byte stream and manually determine byte length (as done here) and then split the byte stream as appropriate. This'll require quite a bit of refactoring to do, but I'll try to get it done soonish.