File created by BinaryStream throwing \u0000 on import

I’m having issues where a consumer (NetSuite) doesn’t like the files I’ve created with a simple BinaryStream. I’m using UTF-8 encodings, and when I open my file in any editor (Mac or PC) the bytes look perfect. However, the folks at NetSuite are claiming they see offending characters.

If I open this file in BBEdit, or even read it back it works just fine. Perhaps someone on the forum has run into something similar, perhaps this is an encoding issue(?) I’m a bit stuck. Here is a shot they sent me today. And below that is some text dump from their “import wizard”. Any help is appreciated. Thanks!!

Character error on Line# 1 Column# 4 (Byte # 4). ??I >>>\u0000<<< \nCharacter error on Line# 1 Column# 5 (Byte # 6). ??I >>>\u0000<<< n >>>\u0000<<< \nCharacter error on Line# 1 Column# 6 (Byte # 8). ??I >>>\u0000<<< n >>>\u0000<<< t >>>\u0000<<< \nCharacter error on Line# 1 Column# 7 (Byte # 10). ??I >>>\u0000<<< n >>>\u0000<<< t >>>\u0000<<< e >>>\u0000<<< \nCharacter error on Line# 1 Column# 8 (Byte # 12). ??I >>>\u0000<<< n >>>\u0000<<< t >>>\u0000<<< e >>>\u0000<<< r >>>\u0000<<< \nCharacter error on Line# 1 Column# 9 (Byte # 14). ??I >>>\u0000<<< n >>>\u0000<<< t >>>\u0000<<< e >>>\u0000<<< r >>>\u0000<<< n >>>\u0000<<< \nCharacter error on Line# 1 Column# 10 (Byte # 16). ??I >>>\u0000<<< n >>>\u0000<<< t >>>\u0000<<< e >>>\u0000<<< r >>>\u0000<<< n >>>\u0000<<< a >>>\u0000<<< \nCharacter error on Line# 1 Column# 11 (Byte # 18). ??I >>>\u0000<<< n >>>\u0000<<< t >>>\u0000<<< e >>>\u0000<<< r >>>\u0000<<< n >>>\u0000<<< a >>>\u0000<<< l >>>\u0000<<< \nCharacter error on Line# 1 Column# 12 (Byte # 20). ??I >>>\u0000<<< n >>>\u0000<<< t >>>\u0000<<< e >>>\u0000<<< r >>>\u0000<<< n >>>\u0000<<< a >>>\u0000<<< l >>>\u0000<<< >>>\u0000<<< \nCharacter error on Line# 1 Column# 13 (Byte # 22). ??I >>>\u0000<<< n >>>\u0000<<< t >>>\u0000<<< e >>>\u0000<<< r >>>\u0000<<< n >>>\u0000<<< a >>>\u0000<<< l >>>\u0000<<< >>>\u0000<<< I >>>\u0000<<<

Have you looked at the file in a hex editor yourself yet?

Absolutely! Every byte looks perfect, there are no “nulls” or other characters that don’t belong there.

Can you share the file (or even a portion of the file) somewhere so others can look at it?

Appreciate your time @Kimball_Larsen , I’ve zipped it just to maintain its integrity. Thanks!

File name: Auto_Upload_100_2021_7_29_10_53_38_688641071.csv.zip

Seems like they are converting it to UTF-16 or similar prior to their import. Here is a screenshot of the original portion you sent, which is in UTF-8:


Note there are no spurious nulls. The file looks fine (just as you described).

I opened the file in a plain text editor, changed the encoding to UTF-16 and then saved it. Here is what that looks like now:

Given that the nulls are showing up more or less in every other byte for them, this seems to match up pretty well with a borked encoding.

The super odd thing is that this must be happening somewhere between when you send the file and when they try to import it… something appears to be converting the encoding…

This looks exactly what their error report shows - your 2nd set of eyes really nailed it!

My guess? Someone received your file, opened it in a text editor to check it looked right, then saved it … but their text editor is set to save in UTF-16 by default, thus converting the encoding and messing up the file.

But that’s just wild speculation. Most text editors don’t behave that way… though if they are downloading it from some URL perhaps their browser is messing with the encoding when saving the file, or an anti-malware somewhere along the line, etc.

Best of luck convincing them, though! :slight_smile: