Analysis of Xojo’s Bug Tracking System

It sounds like a report generated by an AI.

5 Likes

What can Xojo as company do what hasn’t been tried, yet?

The old bugs need to be weeded out - that is Robin’s job. But there are way too many bugs. There is the voting system. Has that improved anything? Less vanity features - if we as users say “we don’t want that” it’s implemented anyways.

The core problem is the lack of resouces. And this can only be fixed by growth.

5 Likes

Most tools state that this is 80% likely. Which would indicate a very high probablity.
Nevertheless it’s probably good to have a discussion about this.

But what i see its that thing currently are being solved fast, at least much faster than previously. And some bugs (if not most) that are over 2 year old are probably read for the bin, or they have too less information to be fixable (or interpreted wrongly). Anyway progress is good, there is one way and that’s forward.

I wonder if @Geoff_Perlman can verify these numbers in the first topic to be actually the case currently.

Although I generally agree with the report as Xojo has a serious bug problem for as long as I can remember and it is well know the lack of resources is their main problem (although they keep denying this but this is just the ostrich effect at play), it was also my initial thought: this report is generated by A.I.

It just may be the audit company does use it to generate their final reports. Many companies do nowadays.

Xojo is a small company with a small user base, so I doubt any report will have any effect on how they will operate in the future. It doesn’t report anything new and after 25+ years you can not expect the company to suddenly grow now because of something like this. If Xojo was meant to go big, it would’ve happened a long time ago. Users mature and learn, or change jobs, and many move on because higher standards are required. Technology changes faster and faster and not every company is able (or wants) to follow.

And if this is ok for Xojo Inc., it is their rightful choice.

1 Like

The free market usually decides what a product’s level of acceptability is whether that be functionality, price, customer service, etc. Different things are important to different people, a difficult balancing act for any company.

Everyone would like to see fewer bugs, but that doesn’t mean Xojo hasn’t found the right balance of product features, price and ultimately value for its customers. That’s why they are still in business. Xojo could put all its resources on fixing bugs, but then we’d be whining about new features.

6 Likes

The age of a bug is irrelevant. It’s the impact surface that is relevant and that’s a measure of how big a problem it is (no workaround for example) and the number of users affected by it. To use two extreme examples, a bug that caused every app built in Xojo to crash on launch would be the highest possible priority. Everything else would come to a stop until we fixed that bug and released an update. This is of course a thought experiment as no bug like that would ever get past the internal testing let alone beta testing by users. Another bug only affects one user and there’s an easy workaround. This bug is likely to linger for years because there are simply more important things for us to work on.

Like all companies we have limited resources so we have to prioritize. We don’t look at bug reports and feature requests in isolation because users are asking for both. But one thing is certain: the age of a bug report or a feature request is not relevant. Only the impact surface is relevant.

We have a system where issues that have been untouched (no one has commented on them or modified them in any way) for two years get reviewed. We do this because sometimes a bug gets fixed as a side effect or other work or a feature gets implemented and we didn’t notice that another duplicate feature request had been made. When this happens, we want to close these issues. Our two year review process helps with doing that.

8 Likes

I will disagree with you a bit. Just because few people have reported a bug or signed on it to doesn’t mean that it doesn’t mean it’s important. If it’s a bug that’s keeping my app from being shipped then it’s a BIG DEAL. If it gives me poor reviews because it affects a small number of my users (but gives an overall poor impression) that hurts my overall sales and thus my income.

Back in college I went to a professors office to ask about a question I had. He asked me why I didn’t ask the question in class and I said, I didn’t think it was relevant for everyone (and I didn’t want to appear dumb). Of course it was relevant! He said. Because if one person has the question there’s a good chance others have the same question.

Bugs are the same way. Just because few people have submitted or signed on doesn’t mean anything. I would imagine that a very, very tiny percentage of users have actually submitted a bug report or signed on to one. Why? Because they probably feel it’s their fault it’s not working. Lord knows I’ve felt that way many times only to discover, yes, it really is a bug in the framework. And I consider myself to be pretty well versed in Xojo coding whereas many Xojo users do not so I can only imagine how someone new to software development, or to the product, in general, would feel about submitting a bug report.

I was asked a few months ago about a Web 1 bug that was over 7 years old. Seriously, why is it still in the system? The product doesn’t even support Web 1 any more and no bug fixes will ever by forthcoming - ever. So there is something fundamentally flawed in the system if a bug this old for an obsolete framework is still in the system and being discussed.

I don’t know want to presume any ‘solution’ to Xojo bugs but being a user for over 20 years I know I found it extremely frustrating and one of the (many) reasons why I stopped Xojo consulting. Do better.

10 Likes

After reading the “Analysis of Xojo’s Bug Tracking System,” I realized that my fierce criticisms of Xojo Web, which I made a while back, are a drop in the ocean compared to the results of this analysis. Oh, joy! I already paid for Xojo Pro, so now I have to play detective with the bugs and shortcomings of this tool, looking for solutions that, luckily, I have been able to solve almost completely. Thank you, Underwriters Technologies, for this post. My criticisms are no longer alone in this forum. How exciting! I hope I don’t get censored again for sharing my honest thoughts.

4 Likes

And there have been many times we have been contacted by a user who has a bug that is preventing them from shipping their product. We first see if we can find a workaround and often when that fails, we will fix the bug. That it was important enough to them to bring it up with us also has meaning. We want Xojo users to be successful. The point is that simply analyzing statistics from the bug base is an insufficient way to determine success.

3 Likes

Every development environment as large and sophisticated as Xojo will have bugs you occasionally have to work around. That’s simply reality. As I mentioned in my blog post, bug free software as large as Xojo would be prohibitively expensive. I don’t mean just for us. I mean for any organization. Microsoft is a $3T company and they have a bug base with thousands and thousands of bugs in it for Visual Studio.

@Geoff_Perlman I would generally agree with you that analyzing statistics from the bug base isn’t always a good way to determine success. But the fact that there 7.2K of entries in the bug tracker is a ball and chain everyone feels, whether its the Xojo team working on prioritization or us developers on the other side trying not to submit duplicates or figure out if what we’re seeing is really a bug.

To Bob’s point, maybe there’s some large buckets of tickets that can simply be closed/archived including those for Web 1, anything related to REALbasic, Real Studio, deprecations, etc. that don’t apply to any of today’s products and frameworks.

3 Likes

One very recent example was a long time user that contacted me about a bug that, while it has a workaround, it was also both easy to see how lots of users could run into it and was likely easy to fix. I asked an engineer to look into it and we got it fixed for the next release. So when a bug is a big enough problem for a user, enough so that they bring it to our attention (beyond just reporting it), we certainly consider that as well.

No system is perfect and I personally love automation but some thing cannot be fully automated.

3 Likes

In essence, then, you’re saying that the report is largely content-free. A fair summary.

I had a serious bug this year, which appeared with 2024r1: the layout no longer rendered properly under Linux Mint (bits of conatiner controls appeared in wrong places). I was unable after much work to reproduce it in anying less than my entire app. My Issue didn’t get much traction until I made it private (or maybe I requested that, I forget now) and submitted the app itself as the example, after which it was fixed for 2024r3.1 fairly quickly (by reverting a change introduced with 2024r1). OK, I had to push a bit, but the issue was diagnosed and then quickly fixed.

Overall I’m satisfied with the service.

4 Likes

There is a very old saying: “The misfortune of many is the consolation of fools.”

I’m supposed to get a development tool to make app creation easier, especially for business apps. But what happens when the tool has bugs and I have to dig deeper than the Xojo manual? Honestly, the manual lacks enough examples to really understand the language, which eats up time I should be spending on development.

When I buy a physical machine, there’s a warranty period; if there’s a factory defect, the manufacturer fixes it, and I get a solution. What about a software tool like Xojo Web? It has bugs, and I have to wait for a fix to show up on the roadmap. By the time they fix it (if they do), several versions will have passed, and I’ll have to pay for that new version, which will probably have new bugs too.

Microsoft Windows fixes its issues for free with patches. There’s no patch, let alone a free one, to fix the bugs in the version of Xojo Web I already paid for.

I buy software and, as a customer, get a discounted upgrade price. That doesn’t happen with Xojo either.

My criticisms are reasonable, and I haven’t insulted anyone. Yet, by expressing my opinion honestly and directly, I risk being censored.

2 Likes

Are there a lot of open reports in the Xojo issues tracker? Certainly. Is it a concern? Obviously, yes. But how big of a concern?

As has been pointed out, there are issues that are so old that they are likely no longer relevant and XojoBot doesn’t appear to be very adept at flagging all of these for review. There are other issues that are opened by users who simply have questions and either don’t know to or wish to ask on the forums. So there is a percentage of the issues analyzed that would not be relevant to such a report but almost definitely wouldn’t have been excluded.

As Geoff said, one thing that is overlooked is that Xojo is quick to fix showstopper bugs that have a massively large impact area, such as calculation bugs with intrinsic types (as we’ve seen a few times in the recent past), but it shouldn’t be ignored that sometimes you need to take other avenues in addition to the issue tracker to get your bug addressed by contacting them directly or getting in touch with an MVP (yes, we get these requests from time-to-time and try to help when we can by either drawing Xojo’s attention to the problem or providing solutions ourselves). While I’m sure that movement isn’t a result of a report being ignored by the team unless prodded by management, it could appear that way and result in users opening issues then contacting Xojo using outside channels en masse in order to get traction. Such a flood would certainly slow development on Xojo as a whole with the personal attention that’s been outlined, and in my opinion supports that there is likely a deeper issue with how bugs are handled after the initial review.

I also think it’s fair to say that the amount of open reports is tied to, regardless of how tangentially, the size of the team. However, that’s not something easily (or quickly) remedied nor would hiring more engineers be a firm solution if the process is truly broken. More workers doesn’t necessarily equate to higher quality work nor faster completion of goals, especially in the training period before a developer is comfortable enough to work unsupervised. I would like to see Xojo’s team expand further as the number of targets has increased significantly while the size of the team has been roughly stagnant, but there’s a lot of data that I don’t have access to that would make me feel comfortable saying that Xojo should immediately expand their payroll. Additionally, there have been obvious issues with new features that probably should never have been released in the state that they were in but Xojo works to correct these shortcomings even if it had to be brought to their attention after the fact. This, in my opinion, possibly points to the need for a dedicated in-house QA team that vets new releases extensively before going even to pre-release channels, but we have to consider the team size and those related fiscal factors to which only Geoff can attest.

In recent years there have been a number of improvements to Xojo’s handling of bugs, starting with the new tracker, and I don’t think that should be overlooked. They are making changes as they are able when a solution is available, and that’s an important point in my opinion. There are easy things to point at that we don’t like such as the new LR (and it’s infamously bad search at launch). While there is some distance to travel, I don’t personally feel that the team is just sitting on their hands and taking our money then ignoring our concerns and problems.

At the end of the day, it’s about balance. Not just about Xojo balancing its income with payroll for the team size, but for each user. Does Xojo do what you need, even if you have to workaround bugs sometimes? Is the hassle of working around those bugs greater than the benefit Xojo provides versus another language and development environment? Do you see the future of Xojo as being so bleak as to warrant a change to something else for future projects? Is Xojo even the right tool for the task at-hand? These are all questions we each need to ask ourselves and take appropriate action based on our answers. While such an analysis of the open bug reports may give some insight, it’s only as good as the data the preparer has analyzed (both from Xojo and its alternate “industry standard” sources), how it analyzed that data, and the impact of that data and its conclusions on each user.

Taken in context with what I know of Xojo from working with it for over 20 years, this report doesn’t surprise or greatly concern me. Xojo is still my preferred environment and language. While that may not always be the case – nor has it always been true in the past – it’s where I stand right now.

8 Likes

I’m wondering how many users are expecting the tracking system to have equivalent strength (if not more) than contacting Xojo privately.
Especially if I already have a workaround, I’d tend to report in Issues.

IMO, both these categories should be closed as soon as possible. Even just taking 10 minutes per day to close some no longer relevant would make things clearer.

5 Likes

If Xojo can’t have one of the staff do this (as they have other priorities), maybe some of the MVP or other volunteers can help, maybe?

For example this 14 year old ‘bug’ (that I think is a Feature Request):
#11024 - WeekOfYear not international savvy
after helping someone in the forum with DateTime.ToString I reviewed how .ToString format gives us the ‘Week of Year’ and if you provide a locale that uses the ISO standard, you can get the ISO week of the year. See my last comment on that issue.

No work around was provided by Xojo. The need for ISO week of year is important for people that use it. DateTime.ToString format offers this since (I don’t know how many) years. And no one pointed that out before (on the issue or on the forum). Maybe is ‘easy’ for Xojo to add ISOWeekofYear to DateTime?

What will happen with a “Not Actionable” bug? Should that be closed or reviewed after 2 years? Example:
#64255 - 2021r1 running web apps from IDE

1 Like

Still, a positive note: In the last 2 years, I do see a lot of efforts to fix bugs by Xojo Inc! This is also worth mentioning and is not directly reflected in the above statistics.
It’s fair to say that too.

Just my 2C offcourse.

6 Likes

We do have someone who each day reviews all cases that have gone 2 years without any attention so that we can determine if they are still relevant or not.

5 Likes