My Web application needs to constantly poll a directory on its sever for new files.
If the files contents haven’t changed for like 5 seconds it processes that file, then deletes it and goes on to the next file…
This loop goes on and on for ever in the background.
Should this be setup as a thread in the main application or should I have two applications running on the server, one to serve the user and another to polls and process this directory?
It depends how much you want/need this to scale. I’d imagine that just running in a thread is easier and gives easier opportunities to integrate feedback about the thread’s progress with the rest of the app. So if the app doesn’t need to scale much, and the work of the thread is negligible, that argues for keeping it in the app.
But remember, when your app is working on this thread, it’s not working on other things. Xojo’s thread model is basically single core cooperative. If the thread bogs down, it’s bogging down the web server, the UI, etc. So if you need it to perform well and scale, make it a separate process. That process could also be a web app, if that makes things convenient. For example, just create a web app to run this process and display status. Deploy as stand-alone so it’s always running.
I would say separate app because WE threading is cooperative. If anything blocks in the thread for any reason, your web server will stop responding until the blocked function clears.
Absolutely, I concur with the distinguished gentlemen above. One of the key pieces of advice I have for people with WE is to keep the main application laser focused on the UI interaction, and just the data and logic required for that. It makes a big different in user experience.
Even for “single server scaling” you’d want this a separate app- let alone the day where you might find yourself needing to scale out to multiple web/file servers… 
I would move the work to a separate app. A Web app blocked for a second will annoy your users a lot!
And you can block it for a second with opening a big image file for example.