I need to read some data from a file that contains mostly string data on each line, but each line includes a social security number which is encrypted into binary data. The problem is, I need to read the data after the encryption and the encrypted data (SSN) takes up a different length for each line of data in the file, so I can’t rely on the data after the SSN being in the same location for each employee.
I have the file layout and all the data before the social security number is in the same location, but after the social security number, the location of the data varies.
My guess is that there is a way to determine where the encoded SSN ends on the line from the data on the line. If not, no one will be able to read and decrypt the SSN.
For sure you need to read each line as binary data. Do you know where the SSN starts and the encryption used ?
Kem, you are correct. I am hurting with this one because everything I’ve tried has failed to work. Oddly, I’m converting this from a Visual Foxpro application that DID work.
I know where the binary data starts, that’s a constant. But the only way I know to get past the binary data is to read each successive character until I get back to an ASCII character, then continue reading from there.
Are you dealing with fixed size records? Can you share some sample? You can use some hex/binary editor to cut a sample with few records and even change some confidential contents before sharing. Like “Mary Jane” → “Mary XXXX”
Having had to reverse-engineer a few binary formats for use in Xojo just in the last few months, I can confirm you are in for a world of hurt. A hex editor will be your best friend, and I found the Chilkat BinData and FileAccess plugins to offer a few beneficial shortcuts over using Xojo MemoryBlocks (which are still necessary in some cases).
But the only way I know to get past the binary data is to read each successive character until I get back to an ASCII character, then continue reading from there.
The catch with this approach is there’s no way to tell if the ASCII character is part of the binary data or not. The program that wrote these files must have some way of determining the binary length.
Often times the binary data has a 2 or 4 byte header containing the length of the record. Pull that apart in a hex editor and see if you can find any binary numeric data, checking both big and little endians for values that correspond with the length of the binary data as you can see it.
To be honest, seeing the data in this form is not really helpful, as the forum software changes the formatting.
Can you upload a binary file containing a record or two with the surrounding data? You might need to use a hex editor to change the ascii data and protect the privacy of the individuals involved.
No guaranties. But some are really simple, as fixed size records in one file and indexes in other file. If it is variant, it must have control fields, and you guys may suffer to find how to read the records correctly ( I won’t have time for that). Some could have separators, control chars, but having a binary field in the middle of the record would be weird, because coincidences could occur and data bytes could be confused with control chars. That’s why I’m curious about those unusual records said made of “lines”.
Are you sure about that? Length in characters is not the same as size in bytes; it’s not something you can see in a text editor.
The size wouldn’t need to be given if it’s always going to be the same, and SSNs are always the same size. So, unless the encryption is randomly adding padding I see no reason why it should be variable-length.
I won’t have time for that). Some could have separators, control chars, but having a binary field in the middle of the record would be weird, because coincidences could occur and data bytes could be confused with control chars. That’s why I’m curious about those unusual records said made of “lines”.