Best practice to check if connected to the internet. Desktop app and web service

[Patrick_Salo]

@Tim_Parnell

Just to clarify, the test showed that InternetGetConnectedState will tell you if the LAN is available, but does not actually test internet connection.

Actually I might be wrong, but I believe this does check to see if internet connectivity exists (as well as LAN). See the documentation remarks. But mind you MS’s wording across the entire article is very confusing between LAN and the Internet. Also it appears that this call is deprecated and we all should probably move over to INetworkListManager.

@Tim_Parnell

1. As I mention in the other thread, you really should just attempt your request and then handle the error. In most cases you’re just going to relay to the user what the NetworkException code and message are.

I disagree with just trying the request. Going back to Apple and iOS, I believe why they were mandating checking for the network before using it, boiled down to the user experience. There’s often lengthy timeouts involved, especially if you’re on a crappy network. This reflects poorly not just on the app, but at the time iOS which in the early days didn’t have the most resilient or robust cellular connectivity.

Maybe I’ve spent too much time on the other side in Apple-land (Xcode, Obj-C, Swift, etc.), but I sort of see this as taking a trip to the mountains in the winter. Ideally one would check for road closures and issues along the way before hopping in the car and hoping for the best. The entire premise here on the network side is to check for a good likelihood for success before actually trying the connection.

The dilemma here is that there’s no right answers on the connection. If you keep normal connection timeouts, a user could be waiting 60 seconds before things fail. And if you shorten this down to make things fail faster, then those on crappy network connections will fail to make the connection even though they are capable given enough time.

@Tim_Parnell

2. Network.LookupIPAddress(sDNSAddress as String) will resolve a FQDN

I think the problem here is that LookupIPAddress, from what I can tell from the docs, is just a DNS lookup and has nothing to do with actually checking the path from the client to the FQDN. Theoretically one could bake in a traceroute to check themselves, but you’ll end up in the same bucket as the connection timeout issue (e.g. adjusting timeouts, needing to handle say a switch dropping some but not all packets, making the connection problematic but still traversable, etc.).

@Tim_Parnell

2.1. URLConnection.Send("HEAD", url)

Yes, this is very doable but would need to be done strategically and as part of a larger and more wholistic solution.

@Tim_Parnell

3. URLConnection.Send is event driven, LookupIPAddress is not though. FWIW, making the attempt and then handling the error is also an event driven design.

See my comments above. Ultimately and not to be an Apple weenie here, but how Apple does this with Reachability and others with subsequent technologies, is exactly what I’d like to see across all OSes within Xojo.

Here’s where the rubber meets the road within my own code. In my Apple target, I’m rest assured that someone is on a network, on the Internet and can see my server before making any network calls. How Apple does this has never been detailed as they’ve never released their implementation details and hence it’s always been a blackbox that has just worked. But under Windows, I can only determine if they’re on a network and the Internet, not whether or not they’re traversable to my server or even if my server is even up before making the call.

@Tim_Parnell

This is meant to be helpful, not a critique. It seems to me that the tools exist to properly handle the situation. Am I missing a piece of the puzzle?

I’m not disagreeing that the pieces aren’t already out there and I’m more than capable to craft some solutions here myself, but it’s low on my own priority list versus other things. But as this is something that impacts by and large all Xojo users who use the network, that’s why I called out an enterprising plugin vendor although Xojo could do this themselves if they didn’t already have so much on their plate.

Not that I’m fond of blackboxes, but frankly that’s what I’m advocating for here. Some high level wrapper or class that hides all the nuances and intricacies we’ve been talking about this thread where a Xojo developer can be rest assured that the results will be the same regardless of the platform they’re targeting.

[Tim_Parnell]

@Patrick_Salo

Actually I might be wrong, but I believe this does check to see if internet connectivity exists (as well as LAN).

I have tested and found that it does not behave that way, that’s the whole reason this thing got bumped. I gave the machine a LAN but not internet connection and ran the sample project I linked above.

The boolean result >with no internet< was true

[Patrick_Salo]

You’re absolutely correct here Tim. I’m getting an 18 flag back regardless of Internet connectivity on a Windows system, which I think ends up being INTERNET_CONNECTION_LAN 0x02 & INTERNET_RAS_INSTALLED 0x10.

So this ultimately makes the situation worse for Xojo Windows devs in that InternetGetConnectedState can’t be relied upon (but maybe INetworkListManager would be a suitable replacement?).

So maybe this is a poorly named function by MS or there’s some kind of other bug. Like I said, the documentation is very confusing as the way they explain things is not very precise regarding LAN versus Internet.

[Jason_King]

@Patrick_Salo

how are the Xojo iOS folks addressing this requirement? Xojo code? Declares? Something else?

iOSKit has a Reachability class: iOSKit/Modules/Extensions/Reachability.xojo_code at master · kingj5/iOSKit · GitHub

[DerkJ]

One could use a urlconnection to test for a connection to your (or google) server for example.
With a short timeout.
Or use the event headersreceived.

Try
URLConnection SendSync(“HEAD”, url, 1)
Catch e as networkexception

End try

[Patrick_Salo]

Thanks @Jason_King and I probably should have flagged you as well as Jeremy. Good to know that Reachability is the way under Xojo iOS.

[Christian_Schmitz]

We just go the Network classes to check for availability of routes: NWPathMonitorMBS

But in general, I would recommend to just try a connection and see whether it works.

Like you may a dial-up connection to connect you to the internet if you try it.
You may have a proxy which includes DNS, so a normal DNS lookup fails, but the query through proxy works.

[Matthew_Stevens]

@Patrick_Salo

On a separate note, I’m still left scratching my head a bit though in that I haven’t stumbled across any truly cross-platform solutions here especially as Apple and Microsoft have engineered things differently…

Software engineers often scratch their heads struggling to accept a network can not be entirely controlled. Network engineers accept it as ineffable truth

From the perspective of network operational theory. Applications reside on top of network protocol layers. The presumption at every layer is the layers underneath are working as they should. This infers the only valid test is whether each layer can reach it’s corresponding endpoint. The only valid test of an application instance is whether it can reach the application instance it is trying to communicate with.

[application] <...........> [application]
     |                           |
[ transport ] <...........> [ transport ]
     |                           |
[  network  ] <...........> [  network  ]
     |                           |
[  physical ]---------------[ physical  ]

The entire premise here on the network side is to check for a good likelihood for success before actually trying the connection.

This is what TCP is for. Completion of the three way handshake assures a route to the endpoint, and is probably more efficient than anything you can come up with yourself. Should your application subsequently fail to reach the endpoint you might assume a road accident or emergency closure occurred after you left home.

If you keep normal connection timeouts, a user could be waiting 60 seconds before things fail.

TCP has been around a long time. If you are still supporting acoustic couplers from half way around the World you might need a 60 second timeout. There is no hard and fast rule. It is up to your application to decide how long it should wait before giving up and closing the connection. As a rule of thumb I expect most applications to survive a live cable re-patch completed in under 15 seconds.

Having a single high level solution to cover the following would be great…

Try reversing the proposition. Why is there not a single high level solution?

One compelling answer is as follows. Dynamic disruption was presumed at design time (nuclear war) and has become an inevitability due to the sheer scale the thing has grown to. At any given moment someone, somewhere, is pulling a cable, replacing a router, rebooting a server, tightening a firewall rule and so on. Any reachability test is then only valid for the exact moment and circumstance it is carried out. The problem turns out to be paradoxical. Solving any part of the problem creates dependencies that were deliberately avoided at design time.

[Patrick_Salo]

@Christian_Schmitz

We just go the Network classes to check for availability of routes: NWPathMonitorMBS

But in general, I would recommend to just try a connection and see whether it works.

I guess I missed NWPathMonitorMBS along the way and this is great solution. Any chance of adding Windows and Linux as targets down the road?

[Patrick_Salo]

@Matthew_Stevens

TCP has been around a long time. If you are still supporting acoustic couplers from half way around the World you might need a 60 second timeout. There is no hard and fast rule. It is up to your application to decide how long it should wait before giving up and closing the connection. As a rule of thumb I expect most applications to survive a live cable re-patch completed in under 15 seconds.

Agreed, 15 seconds should be plenty of time, especially if we’re talking about desktop here where broadband is more common. The rub though is that Xojo in its more recent history supports mobile where connections can potentially be literally half way around the world and over some high latency backhauls.

@Matthew_Stevens

Try reversing the proposition. Why is there not a single high level solution?

One compelling answer is as follows. Dynamic disruption was presumed at design time (nuclear war) and has become an inevitability due to the sheer scale the thing has grown to. At any given moment someone, somewhere, is pulling a cable, replacing a router, rebooting a server, tightening a firewall rule and so on. Any reachability test is then only valid for the exact moment and circumstance it is carried out. The problem turns out to be paradoxical. Solving any part of the problem creates dependencies that were deliberately avoided at design time.

I think you’re missing my point here. I’m not looking for a high level network solution per se, but instead a high level cross-platform solution in Xojo. Just as Xojo abstracts away much of the specific aspects of a platform and creates higher level abstractions that work across platforms regardless of target, the same would be helpful on the network side. Think of how DesktopWindow or even DesktopButton are a high level abstraction to their native counterparts. Sure you lose the nuance of each platform along the way, but you gain the capability of putting a button in your Xojo app and can be rest assured it’ll be a button that functions in a certain way whether it’s a Windows, Linux or macOS app. Right now at least on the network side of things, there’s no single cross-platform solution as everything is piecemeal and somewhat locked into each platform’s specs/nuances. As this is a very commonplace issue for Xojo developers who use networking, it make sense that this would be something covered at a high level abstraction so that people don’t have to be experts on Apple vs. Microsoft vs. Linux platform implementations.

Of course I’m not debating any of the technical aspects of networking, but instead I’m coming at this from the user experience side of things. Personally, and I’m likely biased towards Apple here, but as a user I much prefer:

1. Open app
2. App determines network viability
3. If the network resource isn’t traversable/available, the UI is adjusted to call this out and to try again later

…to the alternative of:

1. Open app
2. User or app makes a network call
3. Worst case scenario, user waits the entire network timeout (60 seconds or even 15 seconds if dialed back)
   3.1 This timeout delay is a near an eternity in user land, even at 15 seconds
4. UI is adjusted, user given network message and to try again later

Changing gears to the solution front…

Yesterday I briefly tried InternetCheckConnectionA but this wasn’t fruitful as I was getting back a true regardless of whether or not the Windows system was on the Internet. The next step and preferred course per the documentation is INetworkListManager::GetConnectivity but this is much more involved and I ran out of time to head down this road.

Although the Xojo team already has much on their plate, I put in a feature request into the tracker, 75885 - Extend the Xojo Network Class to Check for Internet Availability & FQDN Reachability.

[TimStreater]

@Patrick_Salo

I think you’re missing my point here. I’m not looking for a high level network solution per se, but instead a high level cross-platform solution in Xojo. Just as Xojo abstracts away much of the specific aspects of a platform and creates higher level abstractions that work across platforms regardless of target, the same would be helpful on the network side. Think of how DesktopWindow or even DesktopButton are a high level abstraction to their native counterparts.

No, I think you’re missing his point, which is that whether or not the fundamental question (Is the network up?) even makes sense to be asked. The question of it being nicely cross-platform is completely secondary to that. You can’t determine network viablity, as that varies from moment to moment, along with response. If an outage occurs somewhere, traffic is automatically rerouted. If you’re unlucky, 100Gbps worth of traffic may suddenly find itself going over a 1Gbps connection, so response times shoot up. If you’re lucky, there will be no more than a blip as a few packets are dropped as the rerouting occurs.

[Patrick_Salo]

@TimStreater

No, I think you’re missing his point, which is that whether or not the fundamental question (Is the network up?) even makes sense to be asked. The question of it being nicely cross-platform is completely secondary to that. You can’t determine network viablity, as that varies from moment to moment, along with response. If an outage occurs somewhere, traffic is automatically rerouted. If you’re unlucky, 100Gbps worth of traffic may suddenly find itself going over a 1Gbps connection, so response times shoot up. If you’re lucky, there will be no more than a blip as a few packets are dropped as the rerouting occurs.

No, I’m not missing his point but I’m intersecting it with the general user experience and real world use cases. Here’s a real-world scenario…

On my Mac, when using Apple Mail, if I lose Internet connectivity, within 5 seconds the app adjusts its UI and functionality as it can no longer traverse to the mail servers. But this is even more nuanced in that if I have Internet connectivity and if just one of my mail servers, say iCloud, goes down, then just that account/server gets flagged as offline until it comes back (e.g. this could be a network issue but in most cases is a server issue). From a user perspective I really don’t care what the problem is, but it’s helpful to know and see within the UI that say iCloud eMail is down/not working and I shouldn’t expect any mail to flow in/out to the Apple Mail app until the issue gets resolved by someone (me, Apple, my ISP, some intermediary third-party such as a Cloud vendor Apple uses, backbone provider, etc.). It’s also really useful to see this in advance, rather than have some 60 second network timeout when I try and send an email.

So I get that folks are saying just to send the network request. But this is a different user experience than Apple or even Microsoft is providing. Maybe a better way of discussing this is to ask how are others mirroring the Apple Mail experience above in Xojo in a cross-platform type of way?

Also maybe I’m missing something here, but if the best practice isn’t to determine network/server viability, then why for a decade plus have Apple and Microsoft provided such functions in their APIs? When and where should these be used then? Also why was Apple so diligently NOT approving iOS apps that just shipped off network requests without first checking for reachability?

I’m not trying to be argumentative with any of this. I’m just looking for the best cross-platform user experience and I’m hoping folks might have some better answers than what I have.

[TimStreater]

I expect the apps in question (such as Apple Mail) are just doing their normal thing of asking the remote server “Got any mail?”. If they can’t get through, then they possibly start probing the remote server by sending a NOOP (no-operation) request at a higher frequency that the normal requests for mail. This is part of the POP3 protocol and the server is supposed to rmake a defined, recognised reply if it receives a NOOP. If Mail gets a good reply back it drops back to normal behaviour.

But see, this relies on Mail and the servers doing the probing/responding and is specific to Mail and POP3 servers. For some other pair of apps, they’d do something different but, still, it’s a top level exchange, not something you can ask the operating system about.

So I’d say you need to build it into your app.

[Patrick_Salo]

Yes, this makes sense. At least in my own particular case, Mail is using IMAP, some accounts with push so this detection might be happening faster there.

So as a followup, it does appears that there is one camp where devs want to check network availability prior to making calls…

• “Detecting an iOS app’s internet connectivity is useful if you want to check if the internet is reachable prior to making a network request.”
• “Most apps use some type of networking in their code. Whether you’re handling user accounts or fetching data from a remote server, you need to handle network connection issues gracefully. Not just making sure your app doesn’t crash with a fatal error, but displaying a proper information back to the user.”

But then there’s others in the other camp…

• “Before iOS 11, if you started a URLSession to make a network connection it would fail at once if the network was not available. To avoid retrying a failed connection, the SCNetworkReachability API could check the reachability of a target host. In iOS 11, Apple recommended you stop using the reachability API to check network availability before attempting a connection. Instead you configure a URLSession to wait for connectivity. The wait for connectivity model works better for creating connections but I still sometimes want to show the user a network status indicator.”

…and of course there’s the “official” answer from the Eskimo…

“Just as an aside, I generally disagree with the notion of disabling functionality when there’s no network available. The problem is that reachability (both SCNetworkReachability and NWPathMonitor) is not perfect; it can result in both false positives (saying that something is reachable when it’s not) and false negatives (saying that something is unreachable when it is).”

Although he does acknowledge…

“Having said that, this does not obviate your question because it is still useful to provide feedback to the user as to whether something is likely to succeed.”

The most comprehensive resource I found including references on this subject is…

Working with an Internet Connection on iOS with Swift: Best Practices

To to postulate an answer to one of my questions about Apple mandating reachability calls in the early days of iOS, my guess is that there was a person/group of folks that were in the first camp mentioned above. Maybe they were too hopeful or optimistic that SCNetwworkReachability was better than it was. This then evolved into the second camp with a timeline around 2017 based upon Apple’s documentation.

Also maybe I’m reading too much into this, but as SCNetworkReachability originally tested reachability to a specific FQDN/IP Address. Under NWPathMonitor (from 2018 & iOS 12) that succeeds it, this class instead focuses more on network status, cellular, low data mode, etc.

The the preferred method from Apple themselves is to just try the network resource and using the new “wait for connectivity” that allows for connections to be made by default within 7 days (not at typo!).

Thanks to everyone for the discussion and guidance along the way.

[Matthew_Stevens]

@Patrick_Salo

No, I’m not missing his point but I’m intersecting it with the general user experience and real world use cases. Here’s a real-world scenario…

These were the points.

@"Matt Stevens

The only valid [reachabilty] test of an application instance is whether it can reach the application instance it is trying to communicate with.

The [reachability] problem turns out to be paradoxical. Solving any part of the problem creates dependencies that were deliberately avoided at design time.

Network engineers accept the [this] ineffable truth

Let’s take a closer look at your real world scenario.

On my Mac, when using Apple Mail, if I lose Internet connectivity, within 5 seconds the app adjusts its UI and functionality as it can no longer traverse to the mail servers.

I would say you are looking at it the wrong way. What is being reported by Mac Mail is not the failure of the internet connection. It is the failure of an application instance to reach the endpoint it is attempting to communicate with - As that is the only valid reachbility test for an application. The internet connection may have failed but that is not what the Mail app is reporting.

You can only abstract what exists in the first place. How a UI reports reachability is bound to the specifics of the protocol it is implementing. The Mac Mail UI can provide some abstration because, as @TimStreater says, the IMAP protocol includes a keep-alive message. The applicaiton must be able to reach the server before it can monitor the keep alive messages though, which is where the paradox comes in. If you try to add an IMAP account into Mac Mail when the IMAP server is unreachable, you will experience delays much longer than 5 seconds. The SMTP protocol does not support persitent connection and does not include a keep-alive message. The Mac Mail UI will not inform you the SMTP server is offline until the moment you attempt to send an email.

Also maybe I’m missing something here, but if the best practice isn’t to determine network/server viability, then why for a decade plus have Apple and Microsoft provided such functions in their APIs?

You are missing something. The API calls are reporting OS status. The best they can do is provide some confidence and inform your application the host can reach some other application. To be clear about that, some other application is not the endpoint your app is trying to reach.

• InternetCheckConnectionA, reports the state of the Internet Connection Wizard that was introduced with Windows 95. The function informs your application when a point to point interface (usually a modem) is connected to a point of presence (usually an ISP). If you are still supporting modems or ad-hoc VPN connections it may still be useful. Otherwise selecting LAN Connection in the ICW UI causes the funciton to always return true. The function was deprecated with Windows XP.
• INetworkListManager::GetConnectivity, was introduced with Vista and reports the state of the network connection icon that appears in the system tray.
• NWPathMonitor, serves a similar purpose for Apple devices, reporting the status of the configured interfaces.

The OS presumes being able to reach whatever service it is probing (gateways, dns, ntp, telemetry) implies there is a connection to an ISP. The dependancy causes the API functions to be prone to false negatives especially and false positive occasionally.

What these functions are useful for is up to the developer. Maybe a service needs to be restarted after wandering between WiFi access points. Maybe a VPN connection needs to be raised before attempting transfer to HQ. The point I am making is these functions do not inform your application the endpoint it expects to reach is reachable.

Also why was Apple so diligently NOT approving iOS apps that just shipped off network requests without first checking for reachability?

Only Apple can answer that. Maybe the Apple software developers struggled to acccept the ineffable truth. Less cynically. Testing for ISP reachability has some validity when there is a high statistical risk of the test failing. As connection becomes more ubiquitous and the test returns true more often than false, the justification dissapears. There is also an argument that the public at large has adjusted to the reality of the matter; they have learnt the ineffable truth.

I’m not trying to be argumentative with any of this. I’m just looking for the best cross-platform user experience and I’m hoping folks might have some better answers than what I have.

That would be the TCP three way handshake. Completing the handshake informs your application it is able to reach the host it is trying to reach using the port it expects to communicate on. Unless you are going into space a successful handshake should complete in no more than a few seconds. How long you want to wait for an unsuccessful handshake is up to you. Once you reach the server you are going to want to exchange data and that is where the lengthy delays come from. A busy host can be reachable but may not get around to serving your application immediately.

[Patrick_Salo]

Wow, what a great and in depth response. I really appreciate it Matthew.