You have carefully completed the design, build, and test phases of your new wireless IoT device. Your testing in the lab shows that it meets applicable FCC radio standards, it can connect to the wireless router at your test bench, and it even connects to the site wireless access point (AP) in the office area. By Chris Kelly, Keysight Technologies
But what if your device gets into customer hands and fails to live up to expectations? What did you forget to test? Why does it work so well in the lab, and so badly out in the “real
world”? Do you worry about this at night?There could be several reasons, all having to do with the difference between the test conditions in your lab and the environment in which the device finds itself at the customer site.
What are you already testing?
There are certain tests that you are already doing, the basic tests for regulatory compliance.
- CFR 47 Part 15 device radio emissions – measurements include frequency accuracy, RF power, ACPR, harmonics, and similar measurements. This ensures that the IoT device radio operation remains within certain regulatory limits for its class of service, and reduces the chance of creating interference. Similarly, you will want your device to have good receiver operation (sensitivity, selectivity, blocking, etc.), though in many cases the IC manufacturer has control of these specifications. You specify the chips and modules, however, and you should be aware that different chips vary in these basic performance measures.
- CFR 47 Part 15 electromagnetic compatibility (EMC) – measurements ensure that your IoT device does not create unintended signals and is not upset by external signals applied wirelessly or through the powerline. These may include signals leaking out from your switching power supplies, display controllers, attached wiring, and are not directly related to the intended radio signals your device uses to communicate.
- Certification test suites (from the WiFi Alliance, for example) ensure that your device recognizes and properly uses on-the-air protocols to connect, disconnect, send, and receive data properly from other devices using that standard. For example, if your device is an IEEE 802.11b device, can it establish and maintain communications with a matching wireless access point, correctly using the radio channel to communicate with other similar devices.
You can see that the testing above covers a range of measurements. But there are conditions and circumstances in the operating environment that can still cause your device to fail in some ways.
Making a truly robust IoT device requires a bit more than testing for ideal operating conditions. In software test, this ideal operation mode is sometimes called the “Happy Path” test, but you also need to test for the path including briars and weeds and maybe even a Big Bad Wolf or two. This means simulating actual RF environmental conditions during your testing phase.
Here are some of the reasons your device may experience malfunctions “in the wild.”
Reason 1: Real world traffic load.
When your new device is powered on, it will find itself in a world of billions of IoT devices, and the number of wireless IoT devices will continue to grow rapidly. There are probably dozens to hundreds of devices nearby in the home, office, coffee shop and hospital where your device must operate. Many or most of them will be using the same channel IEEE as your device, and many of them will use the same protocols (such as 802.11 a/b/g/n/ac) as your device. Congestion – also known as high medium utilization – is a big problem in real world deployments, and winning the medium in a congested deployment is critical to successful operation. Your testing should challenge your new device with a traffic load expected in the target environment, while carrying on normal communications itself. You may choose any number of key performance indicators for the impact of this congestion on your device: data throughput in bytes per second, jitter, packet latency, retransmissions, and medium utilization are typical metrics worth tracking. Keep in mind the minimum performance limits you need for your device to be considered a success by the end user. Many medical devices will alarm if they cannot communicate in a timely manner, and alarm fatigue is a big issue in medical settings. Don’t let high traffic loads cause false alarms because your IoT device cannot cope.
A different, but related topic is the types of traffic your device will encounter, especially if they are different than the types you are using. Did you test your new device against a traffic mix including IP Voice, Data, and Streaming Video? It is possible that both the volume and mix of traffic is different from what you assumed, and when your device does not recognize other traffic it should still be a “good citizen” on the air and get its task completed in a timely manner. Remember that as traffic demand rises above some level of channel utilization, collisions and retries will suddenly drive channel activity dramatically upwards, and your IoT device should be robust enough to cope with this scenario, or recover well in case of a failure.
To solve this possible problem, test your new product with intelligent test equipment, possibly mixed with other real wireless devices (AP’s, routers, etc.) to establish working connections. Then add some real or simulated traffic from signal generators (recorded or simulated is easier to replicate and control than spontaneous) and see how well your device copes with the crowded radio conditions. Be sure you test while simulating different traffic types like streaming video or voice. Can it still pass traffic at the expected rates, without error, and without draining your device’s battery? If so, you have eliminated one possible real world obstacle.
Reason 2: Interference from other wireless signals.
There are many wireless communication standards using the ISM bands, including several forms of IEEE 802.11 transmissions, a couple of types of Bluetooth, some ZigBee, and even custom formats. But if your new IoT device has poor tolerance for the variety of signals it will encounter in operation, the device will fail to operate properly in crowded radio bands. Not all these devices can detect the others, let alone cooperate in taking turns or sharing the airwaves.
Coexistence measurements (such as described in ANSI C63.17 standard and AAMI TIR-69 medical recommendation) ensure that a certain level of operation is still possible when your new device must operate in the presence of other radio protocols than that for which it was designed. For example, does your new IEEE 802.11b device tolerate some level of Bluetooth® traffic on that same 2.4 GHz ISM band? Coexistence test has only recently been defined, and it has a somewhat different character than the three familiar forms of tests above, in that it does not have predefined pass/fail numerical limits, but is more descriptive of the results of certain test conditions. It is also possible that the FDA will soon issue guidance for medical device manufacturers regarding coexistence testing of medical IoT devices.
The ANSI and AAMI documents describe at least four possible different styles of coexistence test setup, and using Keysight spectrum analyzers and signal generators playing signals created by Signal Studio software can challenge your device to cope with many amplitudes, data rates, and protocols you are likely to encounter in a real installation.
Reason 3: Roaming behavior in enterprise deployments
This is all about fast and robust device behavior as the user moves through a larger space with new new device. Portable patient monitors must be 100% reliable around the clock, while communicating alarms, events and recordings in real time. Even a few seconds of data loss is considered “outrage” and can have an impact. As patients move around the hospitals from room to corridors to public spaces, the device needs to pick the right AP and roam successfully without any data loss. This requires the device to implement robust roaming algorithms to minimize roam delay – defined as the time delta between last successfully ACKd packet on Source AP and first successfully ACK’d packet on Target AP. Testing under real network conditions like congestion and interference is critical. Ideal roam delays should be under 50ms.
Wireless Scanners in the Warehouse: Pickers in a consumer goods warehouse use handheld wireless scanners extensively to pick and ship items. Time is of essence here, and as they move from one area of the warehouse to another, the device needs to roam efficiently – with no loss of connectivity, no data loss and with minimal roam delay (under 50ms).
Suppose the user of your new device is in a coffee shop, and people nearby are also using their wireless devices. The users of competing devices are moving constantly, a new one coming in the door and others leaving, and your device is moving from place to place as you place your order, find a table, visit with friends, and so forth.
As you move, your device must “roam” from AP to AP in the presence of lots of traffic. And here comes a smartphone acting as a mobile hotspot, colliding with the café wireless access point! You know that you tested the roaming function in some test suite or other, but was that done while high traffic was simulated? What if your device loses its signal, looks for another AP, and makes a bad choice among the other three to six nearby APs, and your data feed drops out temporarily? For a phone loading a webpage, this may not even be noticed, but what if your infusion pump fails to meet a status reporting deadline, an alarm will sound, causing the alarm fatigue many medical staff experience.
Be sure you test for roaming handoff behavior in a variety of challenging conditions before signing off your quality test. The problem may be a software defect in the sense that the software was not designed to handle roaming while coping with the volume and mix of traffic found in the real world. At times this testing can be accomplished by adding new signal sources to the coaxial cables simulating your antenna, but at other times you may want to buy the complete turnkey testing solution to get your new product on the market in the timeliest manner. Either way, Keysight Technologies has the software, hardware, and systems to improve your present levels of IoT test, and even generate complete test reports and executive summaries.
Reason 4: Antenna testing
Your new IoT device works fine most of the time, but every time a cell phone user walks past, your device drops its connection. Or suppose it works fine in the middle of the room but drops out when it is placed near another wireless device?
Perhaps the RF chip or radio module you selected does not have sufficient adjacent channel rejection? Did you test for interference from different radio device types using the same or nearby frequency allocations? It is possible that the antenna in your device worked well when open on the bench, but it was detuned or focused in a different direction by the surrounding metal structures when your device was fully assembled, or the antenna has pattern nulls, but you didn’t measure this.
The antenna is a critical component of your radio module. Performance may vary by a factor of 10 or more if it is not used as designed. Many small IoT devices use electrically small antennas (ESAs) which are a compromise of size, bandwidth, Q and efficiency. ESAs tend to be omnidirectional, though somewhat inefficient. But if the antenna has metal structures nearby, including the location of ground planes on the PC board, they can change the antennas feed point impedance, detune the antenna to a different frequency, or change its efficiency or directionality so the signals are attenuated in some directions. It can also experience shielding and signal loss to nearby tissue and fluids. Transmitter power can be wasted as heat, or misdirected into the floor or out the window. The receiver may become more sensitive to adjacent frequency signals than to the desired radio channel if it has been detuned.
A leader in the field of ESAs stated, “the theory is not always a good guide to performance in practice.” This means that, however you thought the antenna should work, nothing beats actual measurements of the real object in situ.
To avoid antenna problems, simulation software will allow you to model the antenna performance before you commit to fabrication. Antenna frequency and bandwidth, impedance matching and even directionality should be modeled in the target device structure, including the location of ground planes or other conductive materials which can affect your results. Possible antenna problems can be diagnosed by checking the antenna using VSWR, and S11 parameter measurements in the complete product package using a network analyzer. If the antenna does not match well to the RF components in your IoT device, the losses affecting both the transmitter and receiver not only decrease operating range, but can make the device deaf in certain directions, or unable to be heard and unable to obtain a fair share of airtime, all the while wasting battery power and shortening the time between recharge cycles.
RF test instruments can measure antenna performance in many ways, and the Keysight FieldFox is a specialist in field measurements like this. Don’t overlook testing your antennas, the simple but critical components of your IoT device.
Reason 5: Interoperability with Network Infrastructure
Your new device starts out working well, and for three months everything is just peachy. But then you get a call from a customer saying the 50 units he purchased started working erratically, or even stopped connecting. Another customer calls with similar complaints. These customers are good solid companies that care about keeping their wireless infrastructure up-to-date, and by the way, they just updated the firmware on their wireless APs last week.
You discover that the AP firmware has now implemented a protocol feature that your device assumed would not be part of the target environment. Once you identify the problem, it looks like a fast patch to your device firmware is required, or you will face a recall. If your protocol compliance test suite (see certification test suites, above) had included testing against all defined functions of your wireless protocol, and not just a small subset, this costly event could have been avoided. Complete interactive protocol test solutions, including hardware, waveforms, report generators, and easy-to-configure control software are available to perform these tests. (See Ixia Wavedevice WLAN test solution at https://www.ixiacom.com/resources/connected-healthcare).
Reason 6: Security Breaches
You know that security is important to your customers. It is why they buy your equipment, because you have a good reputation for quality. But they found out last week that when your device moves from AP to AP as it is moved through the hospital, when it hands off (roams) to the next AP, it loses the security configuration, if only briefly.
One consideration is whether the roaming operation itself opens a security hole in your connection. IEEE 802.141 roaming is a complicated sequence, while your IoT device probes alternative connections, chooses a target AP, breaks a secure connection, establishes a new connection, and restarts security behavior. You find out it is worse if the wireless network is very busy. It seems your device is missing certain handshake messages with the new Access Point and going into a default mode that exposes a vulnerability to hacking.
Be sure that roaming while experiencing interference does not overwhelm your device and cause it to enter fault states that cause long delays in connections or worse, make it temporarily vulnerable to hacking. If your test suite includes simulated roaming behavior in a cluttered RF environment, you might have found this early in product test phase and you would not now be rushing out firmware patches, and explaining to worried customers that the problem is fixed.
The solution to all these possible problems is comprehensive testing with mixed traffic types, variable traffic loads, multiple APs, simulated handoffs, expected interference, and other situations that go beyond basic compliance testing. Keysight has the tools and solutions to help you ensure that when your new IoT device leaves the nest, it is prepared for the actual conditions it will find in the world. This is especially true of medical environments, but with billions of IoT devices everywhere in the world, and growing at over 30% per year, it’s a jungle out there!