I have been put in charge a project that I’m not sure I will be able to pull off. I’m looking to see if those much more knowledgable than I have any ideas.
I’ve been working on a project to read data from several different machines over a network and storing it in a database (currently a simply SQLite database). This is all working well. I have now been told (this was not part of the original requirement) that I need to have a “server” application that handles reading the data and storing in the database (basically, what already exists now) and “client” applications that can read the data from the master database over a network.
Here are specific requirements that I have been given. These, I have been told, are non-negotiable:
Must be Xojo-based since all previous work was done in Xojo.
I have to use a freely available database that has to be entirely contained within the application (in other words, not a separately installed package). To my limited knowledge, this seems to limit me to the built-in SQLite database.
Application will run on Windows.
I will not be able to run a server of any kind, which means faking a multi-user SQLite database with a server application with ServerSockets is a no-go.
The only network access I will be allowed will be a shared directory residing on a the “server” computer that can be accessed as a mapped drive on the “client” computers.
The only application that can write to the database will be the “server”/master application. The client applications would only be able to read from the database.
There will be a maximum of five “client” computers.
I have been told that the built-in SQLite database has a pseudo-mutli-user aspect to it, but that it’s really not that robust and probably not a good fit for what I need it to do.
I think with these restrictions in place, there may not be a good solution here. Should I just go tel the higher-ups that either some of the restrictions need to be loosened or this just won’t happen?
Using sqlite over a network on a shared volume is a recipe for a really crappy experience
They tell you that right up front
See http://sqlite.org/faq.html#q5
The restrictions are “make it multi user but we’re not going to let you use anything that would make it behave nicely for multi user”
what you are given is a bit limiting and can see few places where it would fail, but it’s possible to use the file share as a means of treating the share folder as a hot folder. the way it would work is:
client machines that you wish to acquire data from will have an app that runs in the background at x intervals and will create a unique xml or sqlite file that it would drop in this folder.
The machine that runs the master application could pick up these files and then process them into the main DB. But I recommend that the db be local to this machine.
Norm beat me too it, sqlite over a network share is a really bad idea.
As the clients are “read only” you could have a timer in the “server” app that dumped the tables to json files in the shared directory. These files could be read by the client apps into an in memory SQLite database in the client apps (again on a timer). Obviously not real time, but the clients would be snappy after they’d stored the data.
Just to be clear, in case I wasn’t or I’m not understanding what you’re saying here, the “clients” would be read-only. Only the “server” get data and insert it into the database.
Oh… and 7 Red Lines is classic corporate America!
Hmmm… Possible, but this could potentially be for data going back a year or more. Quite a lot.
I think you all have simply reinforced my original thoughts on this. I think I’m going to go back to them and tell them it’s not gonna happen unless they let me use a multi-user database or let me run a server app to handle the SQLite interaction.
Run, don’t walk, away. Now. While you still can. It’s not that you can’t convince them to modify these “rules”, it’s that they put them into place to begin with.
Here are specific requirements that I have been given. These, I have been told, are non-negotiable:
Must be Xojo-based since all previous work was done in Xojo.
I have to use a freely available database that has to be entirely contained within the application (in other words, not a separately installed package). To my limited knowledge, this seems to limit me to the built-in SQLite database.
Application will run on Windows.
I will not be able to run a server of any kind, which means faking a multi-user SQLite database with a server application with ServerSockets is a no-go.
The only network access I will be allowed will be a shared directory residing on a the “server” computer that can be accessed as a mapped drive on the “client” computers.
The only application that can write to the database will be the “server”/master application. The client applications would only be able to read from the database.
There will be a maximum of five “client” computers.
…
Anyone have any thoughts?[/quote]
Do you really want to go back to your higher up and tell them this is no go? I understand the requirements are non-negotiable.
This may affect your job appraisal down the road when that time of the year come.
Take this as a challenge to show your higher up your capabilities. This is a chance to impress that you can and will try all that you can to give the higher up what they want, with the necessary gotcha made known to them.
Requirement (6): 6. The only application that can write to the database will be the “server”/master application. The client applications would only be able to read from the database.
The “server”/master application does all the writing to the database, so make it this way then. Use your creativity to design some process that trigger the “server”/master application to proceed to write. Maybe it monitors a particular folder/directory for any new change, new files, updated files and then update according to what is in the files. These files written to this folder/directory by the “client” computers.
They may be other ways. Suggest you explore further.
Requirement (5): 5. The only network access I will be allowed will be a shared directory residing on a the “server” computer that can be accessed as a mapped drive on the “client” computers.
This shared folder can be the place for the “client” computers to write files to.
Requirement (7): 7. There will be a maximum of five “client” computers.
This is clear why higher up do not want to set up a server for this purpose. At least this is not the time yet. You can always tell them things will be more efficient when the time come for them to implement a real database server. You may secure for yourself the next phase of this project.
A thought. My inclinaton is to tell you to run far away. But if that is not possible, can you not design it around sockets and a request queue in a single app that does all the db accessing?
State very clearly that a db Server is a must for Multi-User Environments. Period.
A Synology Diskstation f.e. can be such a Server hosting a mySQL db for less than $250 or so. Easy to setup and easy to manage. Can be setup by anyone who can switch on a Computer and use a Browser.
Non-negotiable? As long as we are alive, there is no such thing. Maybe the need of the bosses here is to deal carefully with resources of the firm. Maybe it is something else. Try to empathize and then make sure that they know that you have heard their good reasons. In a next step you can ask whether they would be ready to hear your concerns because it might save them from a lot of long term troubles. Then tell them why there are database servers and why multi-using of the same data absolutely requires a multi-user database solution.
[quote=184862:@Cho Sing Kum]
This may affect your job appraisal down the road when that time of the year come. [/quote]
You just postpone the troubles. If you create something on shaky ground, right from the beginning, then the bosses will fire you just later, when everything breaks down.
[quote=184862:@Cho Sing Kum]
Take this as a challenge to show your higher up your capabilities. This is a chance to impress that you can and will try all that you can to give the higher up what they want, with the necessary gotcha made known to them. [/quote]
Bad advice. Wrong way to go. Improve your communication skills and negotiate, but do not work around foolish specifications. And don’t try to impress anyone.
Here are specific requirements that I have been given. These, I have been told, are non-negotiable:
Must be Xojo-based since all previous work was done in Xojo.
I have to use a freely available database that has to be entirely contained within the application (in other words, not a separately installed package). To my limited knowledge, this seems to limit me to the built-in SQLite database.
Application will run on Windows.
I will not be able to run a server of any kind, which means faking a multi-user SQLite database with a server application with ServerSockets is a no-go.
The only network access I will be allowed will be a shared directory residing on a the “server” computer that can be accessed as a mapped drive on the “client” computers.
The only application that can write to the database will be the “server”/master application. The client applications would only be able to read from the database.
There will be a maximum of five “client” computers.
Anyone have any thoughts?[/quote]
1 Yup
2 Ask your customer back: “Free” in means of “without any additional costs”? With “entirely contained” you mean a single distributed package? Of course you will need a 3rd party server-based DBMS like MySQL or MariaDB - but heck as far as the customer does not say “no 3rd party” everything is fine In order to avoid the handling with different packages put everything in a single installer file and make a couple of post-installation scripts.
3 Yup!
4 No Server Sockets needed… Yup!
5 No Shares nor mapped window drives needed… Yup! (as Alternative you can start up the Client App from the Share directly, so no client installation needed either)
6 Yup!
7 No problem!
I know I simplify… the secret is, take your customer literally.
But of course communicate… ask him back… thats far better than to make silly thoughts about 7 red lines
[quote=184874:@Oliver Osswald]Non-negotiable? As long as we are alive, there is no such thing. Maybe the need of the bosses here is to deal carefully with resources of the firm. Maybe it is something else. Try to empathize and then make sure that they know that you have heard their good reasons. In a next step you can ask whether they would be ready to hear your concerns because it might save them from a lot of long term troubles. Then tell them why there are database servers and why multi-using of the same data absolutely requires a multi-user database solution.
You just postpone the troubles. If you create something on shaky ground, right from the beginning, then the bosses will fire you just later, when everything breaks down.
Bad advice. Wrong way to go. Improve your communication skills and negotiate, but do not work around foolish specifications. And don’t try to impress anyone.[/quote]
If all bosses and higher ups are like you, everything will be okay and every employee will be happy. Working life will be stress free.
Sounds like you have encountered some difficult cases in your work-life, right? And that you have strong doubts whether negotiation would even be possible in such cases. Then you’d rather opt for facing the difficulties and do the best out of it? Not giving up, but find solutions despite the unfavorable conditions?
I have no problem going back and negotiating. If the demands are unreasonable, then it is foolish to try to chase those demands. Instead, my initial effort should be to help them understand why those demands won’t work.
Oh, and I should point out that this isn’t for a paying customer. It’s for my place of employment. I get no extra pay than my normal salary. Plus, this is a side-project and not part of my day-to-day responsibilities. They approached me about it because I wrote the original tool to get the machine readings. This is a “nice to have”, not a “need to have”. If I want to walk away from it, I can.
This was my initial thought and, for the most part would meet all their requirements except for the fact that they won’t allow me to run a server. The firewall is configured to not allow this and they aren’t willing to open the firewall for this. The moment I set a ServerSocket to Listen, the firewall gets very grumpy. IT is a bit paranoid in this case, I think, with their security requirements, but I also understand the need for security in our environment. But, this is where I’m going to make my case. Open up one port for me to listen on and I can probably make this work. But, they may say “non-negotiable” is “non-negotiable”. If that ends up being the case, then I won’t go any further with it.
Oh, and a further clarification, because I’m reading some things here that make wonder if some people have misunderstood:
Only one application would modify the database. The “clients” would be read-only. They would only view the data in the database, based on user-selected criteria. I don’t think that makes much of a difference in this particular situation, but I wanted to be clear about that in case it does make a difference.
For a sqlite database, I agree with Norm, if you host it on a share there is a probability that there will be issues at some point which is why I was recommending that the main db be hosted on the machine that ingestion the information from the hot folder and store it in this db. The next challenge is how do the other machines access this data? The suggestion that Wayne provided is a good one.
I think the main issue i see here is not that it can’t be done but the criteria sets a very bad framework for how much extra work you have to do to make it work and how supportable it will be. In other words, what you are doing as a favor might become the thing you loath the most.
I agree with others here, what they are requesting has requirements that contradict their scope. I would re-explain what you require to meet their objective and if they still do not budge, walk away and tell them good luck.
