I have a programme that parses third party .txt files and loads the content into the database.
This all works fine, but every now and then it does not. A single file is not interpreted correctly.
A timer checks the download folder every 10,000 ms
the .txt file is structured, each line has a marker that indicates the type of information that follows
Before I work with the file I save a copy to a special folder
I need to read the entire text file, before I can load the lines into the db
I assemble the information with the help of a dictionary (59 values) and SQLite with 4 tables (up to 40 fields)
Generally there are only a few records for each text file, hardly ever more than 10
After the push to the db, the loop ends
The next loop starts with resetting the dictionary values to "" and deletes all records in SQLite
Every now and then, say three or four times a week, a file is not properly interpreted. Some data is missing.
There is no pattern to the missing data.
There are no error messages.
Originally I thought of a Memory leak.
But when I re-feed the file from the back-up into the running programme, it works properly.
Does anyone have a suggestion on where I should start looking for the bug
or do I have to live with it?