Is it data or metadata? The Internet of Things requires both
Is it data or metadata? The Internet of Things requires both. Consider the most successful, large-scale networks that enable us to function in today’s environment. No, I’m not referring to internet service providers; I’m referring to the Really, Really Big Networks. Without them, our society would be radically different. The telephone network. Containerized intermodal transportation. Control of air traffic. And they all have one critical enabling ingredient that allows them to be scalable: a Control Layer that is not inherent to the electronic or physical streams that comprise network traffic but rather coexists with them. For decades, the phone networks built Signaling System 7, which has governed the world of voice communications. Container manifests are used in intermodal shipping. ATC stands for air traffic control. And they actually do rule the world.
What exactly is a Control Layer?
Telephone calls were “switched” in the analog era using tones conveyed along with the same cables that handled the conversations. This was referred to as in-band signaling. While they were usable with the technology of the day, they had flaws such as easy signal faking to receive free international calls (known as “phreaking”), the inability to modify the course that a call may follow once it was in progress, no direct relationship with billing or tariffing, and other concerns. When it came time to de-regulate the historically government-run phone companies, laying the groundwork for the myriad of businesses that now offer the majority of phone service, the commercial and technological aspects of call-handling had to develop.
Is it data or metadata? The Internet of Things requires both. This resulted in SS7. Among its fundamental concepts is the separation of call management, invoicing, and routing into a control layer separate from the calls themselves. This enables “on-the-fly” switching between carriers, inter-company invoicing, and a slew of other capabilities made possible by the change to out-of-band (as it’s known in the telecom world) signaling.
Out-of-band signaling was also installed in shipping containers.
With a growing global population, an increasing appetite for food and goods from international sources (largely fostered during World War II as troops deployed abroad), and the rise of computerized inventory and supply chain handling in the early 1960s, the idea of having vessels, trucks, and train cars built for specific types of cargo wasn’t going to scale with demand. So some creative individuals devised the standardized shipping container that we now know.
This shifted command and control over how and where products are routed away from ship captains and into a control layer based on paper and, later, completely computerized manifests. This resulted in considerable flexibility and cost savings. Without these, it’s difficult to see Walmart and subsequently Amazon growing at the pace they have. Is it data or metadata? The Internet of Things requires both.
So did the airlines
Another innovation spawned during WWII was the concept of worldwide jet travel. At the same time as the public experienced the beginning of globalization, pressurized aircraft (at first, high-altitude bombers) propelled by jet engines became conceivable. With the dramatic increase in speed, range, and passenger-carrying capacity, as well as the shift in passenger air travel from water-based to land-based terminals, it became clear that control of where and when the aircraft moved needed to “get out of the cockpit” and into a “network,” which became ATC.
Although it is not as efficient as it might be, ATC has proven essentially safe and dependable for many decades and has been continuously scaled (noting that it does have to evolve once again, to accommodate the upcoming plethora of autonomous passenger and cargo aircraft aka air taxis being developed and deployed now).
Is it data or metadata? The Internet of Things requires both. It is now time for the Internet of Things to establish its own Control Layer.
The amount of information currently created by the IoT, or, to put it another way, machine-generated data (MGD), is absolutely astonishing and shows no indications of abating. In reality, when photography, particularly user-generated video, is removed from the equation, MGD already exceeds what humans have produced from the beginning of time. We are rapidly reaching the “needs to scale” tipping point experienced by telecommunications, global logistics, and aviation.
MGD has properties that make it more analogous to these “real world” networks than what we’ve seen with the advent of, say, social media. For starters, this is the data that may kill someone: if handled wrong, road sensor data could force all of a city’s traffic lights to turn red. Alternatively, a dam’s floodgates can be opened. Or send a train down the wrong track. Furthermore, the same data might have potentially positive applications. In addition to more effectively adjusting traffic lights, the same stream of MGD might be used in actuarial calculations to price insurance premiums or offer an advantage to a quantitative analyst in a hedge fund.
When it comes to IoT data, context is everything.
Given the gravity of MGD/IoT data use-cases such as those described above, it is vital that the user (whether a person, enterprise, government agency, or a fleet of robots) be given as much contextual information as possible. This can include provenance in terms of identifying the source in characteristics such as the brand and kind of sensors, literal context (e.g., inside a moving vehicle, attached to power poles, etc.), and if the source devices are regularly maintained and calibrated.
Is it data or metadata? The Internet of Things requires both. Outside of the data streams themselves, all of this “bounding information” must be identified and, ideally, presented. Tracking provenance and keeping the control layer distinct also provide significant security benefits, such as spoofing detection. However, most IoT devices and implementations do not have a control layer.
Metadata is evolving becoming the IoT’s control layer.
De facto standards are evolving for the IoT equivalent of SS7/Container Manifests/ATC. In contrast to those successful conventions, attaining this for MGD involves significantly more variables than practically anything else: early examples of ‘out-of-band signaling’ or metadata standards for MGD exist exclusively inside vertical industrial sectors. In a control layer, they must give way to a common standard for expressing sensors, data provenance, regulatory and compliance problems, and, of course, monetization. This is specified and stored as metadata in software. Metadata must be layered on top of MGD developed and used inside and across all sectors in order to expand and become an enabler.
Otherwise, sector-specific metadata would have to be ‘translated’ across formats, which nearly never delivers useable results. It is important to note that having a broad metadata schema means that the underlying data does not have to conform to any specific file or streaming format — this is critical for IoT data because converting it from one format to another is inherently ‘lossy,’ which can be dangerous when the data is used to control machines or infrastructure systems. The metadata layer provides the basic information required to identify and use the underlying data – if conversion is required, it may be done at or near the point of use, preserving the integrity of the original data.
This is especially handy when the same source data is sent to many applications, each of which uses a different section of the raw data file or stream. Is it data or metadata? The Internet of Things requires both.
The addition of a metadata layer will give significant benefits in terms of scaling and encouraging new applications, while also boosting data safety, security, and even liquidity. As it combines with technologies such as 5G, edge computing, and blockchain, metadata integration with most sectors’ processes and customer experiences will grow rapidly. IoT is affecting nearly every company and government agency in the world; it’s time to consider how this will affect you and your organization.