An interesting and surprising article!
1 Like
Clever methodology: People using AI subjectively thought they were about 20% faster, but objectively measured about 20% slower.
As someone who has used AI in coding, what they didnāt measure was # of bugs generated, completeness of documentation, code coverage of unit tests, etc. But I do agree that using AI can be very slow - especially with complex tasks. And then you still have to review ever line and review logic.
Knowing how to prompt seems to be more of an art form than a science at the moment. And Iāve seen AI do the right thing one day and the next be completely stupid doing the same thing. YMMV.
Honestly, this just adds to my general sense of despair surrounding AI in coding. If it was a tool that makes you genuinely more productive, that might overcome my feelings that it is turning the industry further and further away from a challenging set of problems that are a pleasure to solve and more towards a grim task to complete by plugging part A into slot B. Where is the joy of figuring out an elegant solution to a problem? I doubt youāll get the same degree of satisfaction and fulfillment from crafting a prompt and cleaning up the output.
1 Like
Iām sure farmers, when tractors became available, felt that they were going to miss the satisfaction of knowing every piece of ground and knowing practically every plant they work with. Engineers, before calculators, Iām sure felt joy in solving complex calculations that took a while.
When I was a much younger person I took great satisfaction going to a library and finding information via their card catalog. It meant searching for a topic or author, and then finding the physical material. Donāt get me going on microfilm searching. I donāt miss that at all.
Iām not saying youāre wrong, but I am saying that the tools change over time (I mean, auto complete is rather new and we all take it for granted now). Rather than fear it maybe look at the tasks you donāt like (for me itās documentation) and use the tool to enhance that part of it. Using AI to do a code review (given a proper prompt) is a superb way to make you code better and more efficient and maybe finding some edge cases. Having AI create unit tests has been awesome because it WILL find edge cases that you didnāt think about.
3 Likes
IMO Claude hasnāt been as good generating answers for us as Grok has been. We found ourselves having to prompt Claude a lot more. Using it to code large blocks of code is inefficient to us, we mainly get a quick take on something or use it for an reference and examples.
Tractors replaced animal-drawn implements. The work stayed the same: the farmer rode the tractor and hauled the plow. His mind was still engaged in the process of planning the planting, watering the crops, worrying about the winter.
The old physical card catalog was replaced by the computerized card catalog, followed by electronic magazine indexes, followed by global search engines (before they were infected by AI), all of which widened and accelerated access to information. The searcher was still responsible for taking the search results and synthesizing them into something greater.
Auto-complete saves you a trip through the documentation if you donāt remember a method signature, but doesnāt provide you with context or understanding.
Consider the transition from manual telephone switching, to mechanical telephone switching that replicated the manual process of connecting physical circuits, to electronic switching that simply routes packets. At every stage, the process was radically transformed, but it is never outside of human conception and control. The system is understandable because it was conceived to be so.
AI removes, or at least distances, the human being from the critical thinking process, and this is where it is fundamentally different from other technological advances.
Itās true that tools change over time, and many industries have been completely transformed by changing methodologies. However, I cannot think of another industry where the creation or manufacturing process has been completely turned over to computers, and that is rapidly where we are heading with AI in many fields.
AI is the very exponent of the black box, where one types in an arbitrary request and through unknown (and unknowable) internal machinations, a result is produced. This is fundamentally different from any prior transformation of an industry or trade. There are no factories, devoid of humans, where you can make an arbitrary request and receive something in return; Star Trekās voice-driven replicator is nowhere on the horizon.
Perhaps the starkest analogy I can draw is that of historic tool making versus AI. Almost all tools are extensions of the human body or human senses: a hammer is a longer, harder arm and hand; binoculars are super-powered eyes; a gun throws a stone faster and harder than any human arm.
In contrast, AI is a tool that aims to replace the human mind in several important ways that have no precedent. There is no traditional tool that has internal workings which we do not control and understand; no screwdriver that lies to us with complete confidence; no hand mixer that sincerely reinforces our psychotic delusions. Our traditional tools do not masquerade as thinking beings, are not designed to appear to be what they are not. AI is different.
Iām not saying we shouldnāt create new tools. Iām pointing out that AI has no precedent in human history, and while it does have the potential to improve some parts of human existence, it has, I think, much greater potential to increase misery, especially in the hands of those would wield it against human creativity in an effort to reduce costs and increase productivity. And make no mistake, it is those malevolent hands that currently hold the reins.
5 Likes
+1000 ! You need to be very precise with your qestions, and then the AI tool provides the required code.
When talking with his AI tool, one can spend a lot of time without noticing, and I guess this is where the partticipants of the study spent most of their +20% time. When AI will be able to figure out what you are looking for, like a human asking questions, then the game may change.
Those who fire people to replace them with AI agents - hey Microsoft, I am talking to you - are out of their mind IMO. Just look how painfull are patch Tuesdays, now figure out what it will be when AI takes care of that.
No, AI helps me think. Tools always eliminate parts of thinking. I have a list of tasks where AI helps me to get through the items faster. AI is especially helpful with stuff where Iām not an expert. Like writing php or CSS. For my last Xojo project I needed a lot of CSS finetuning. It would have taken me a lot of time to research that.
Iām not sure what the ārandomized controlled trialā is supposed to measure. There are nice graphs which donāt mean very much. If you prompt then you canāt write code.
2 Likes
Exactly. And thatās why anti-AI rants happen. The main reason people complain AI is not good (enough), is because 99 out of 100 they did not provided enough input.
Iāve been very satisfied and somewhat surprised using Grok recently to help with my new Xojo app. The last time I used AI was a year ago using ChatGPT to generate DAX code for PowerBI, and it was terrible. This time, Iāve been asking Grok for assistance on some topics I didnāt know anything about or how to even begin. And I gave it some complex tasks. Iāll say that it got about 90% of the way there. There were some bugs I kept finding and when I asked it to fix them, it would apologize and give me some code that was slightly wrong in a different way. But it would describe the problem and the solution it was trying to implement, and I eventually understood it well enough to finish the last 10%.
Iād agree that using an AI does slow down raw code developement.
What Iām less sure about is the quality of the code generated.
Iāve been using GitHub Copilot[1] for about 8 months to generate some Xojo code.
Itās āsatisfactoryā most of the time. The usual failure is referencing extension methods on types that donāt exist in Xojo, but do in other places; e.g. Double.MaxValue.
Along with the usual logic errors - some have been quite creative! 
Where Copilot has been most useful is generating Unit Tests. Particularly finding edge cases. But you have to ask for them in the prompt. So Iād agree that crafting appropriate prompts is going to be a new skill that developers can learn. Hmm, can an AI help with crafting prompts? 
[1]When talking about Copilot itās important to identify the variant used. GitHub Copilot is the only variant tuned for programming use. Copilot 365 is tuned for Office 365 document creation and management. Desktop and Internet Copilot (might not be accurate names here) are general purpose AIās.
Note: Yes, I know these are not true AIās. Just completed a Masterās in Machine Learning - well, my daughter did the course and passed the exams (thesis still to done). But if you have had a child go through university you know what I mean!!!
1 Like
Yes - absolutely. I had a prompt that was failing yesterday and I gave an example of what I wanted to check for in a Code Review and it gave a much better prompt.
1 Like