I need to run through a file that includes long string messages. The messages can possibly be repeated. I only want to keep one. I think using the message as the key and value in a dictionary would be my best option, but is there a string length limit on the key?
There may be other options depending on what you intend to do
What are you doing with the distinct strings once you have them all ?
If all you need to do is gather them then write them out you could md5 the strings as you encouter them, and just keep a dictionary of md5’s
Every time you find an md5 that is not in the list write the string out (to a file or db or whatever) and record the Md5
Please provide more information what you want to do. If you have data in the lower MB range then a dictionary may be okay. For larger amounts of data a different approach might be needed, for instance a database.
These are error messages. There could be anywhere from 0 to n messages, but I highly doubt it goes over 10 max. Each message could possibly be a few sentences long. They will be displayed to the user using a custom dialog window.