INTERNET OF THINGS
Editor: Amit Sheth, Kno.e.sis and Wright State University, amit@knoesis.org
Internet of Things
to Smart IoT Through
Semantic, Cognitive,
and Perceptual Computing
Amit Sheth, Kno.e.sis and Wright State University
A
ny time a technology enters a hype cycle,
there are promises of massive growth and
visions of immediate impact. But time and again,
it takes 10 to 15 years as the technology makes its
way slowly through different stages of a hype cy-
cle before it matures and we figure out the tech-
nology’s broad-based market presence. In my own
experience, the gap between the time when I began
working on federated databases, distributed work-
flow management, and a semantic search engine1
to the time when a credible and mature product or
service came about was 12 to 15 years.
One might argue, however, that the Internet of
Things (IoT) is seeing a much more rapid adop-
tion. There were an estimated 5 to 9 billion con-
nected devices with a market size around 700 bil-
lion in 2015. With estimates of more than 25 to 50
billion IoT devices deployed and 17 to 32 percent
annual growth, the IoT could be over a trillion-
dollar market before this decade is over.2,3
From a scientific and technological view-
point, the more important number is related to
the amount of data the IoT already generates. In
2008, social (or Web 2.0) data overtook all other
types of data on the Internet, and recently, IoT
data became the largest type of data on the In-
ternet. An estimated 2.5 quintillion bytes of new
data are being generated daily.4 Now, consider
that by 2008 we generated more data than it was
possible to store, and near the turn of the last de-
cade, less than 1 percent of all data was being ana-
lyzed. This means that data that is not analyzed
in real time or soon after it is generated is never
looked at. In this context, let’s look at the need to
108 1541-1672/16/$33.00 © 2016 IEEE
Published by the IEEE Computer Society
progress along the data, information, knowledge,
and wisdom (DIKW) path to support more intelli-
gent processing of IoT data.
Targeted and Limited Use
of IoT Data
Much of the current use of the IoT is point-to-point
for expected, targeted, or predetermined use of
data. This is the case with most of the use cases in
which IoT deployments are finding extensive appli-
cations5: energy control and HVAC maintenance in
commercial buildings, maintenance and field ser-
vice, manufacturing, transportation logistics, insur-
ance telematics, and in-store contextual marketing
are currently the leaders in IoT deployments. Such
applications use limited types of sensors, and the
data usage they supply is well defined and targeted.6
Integrated and Knowledge-
Enhanced Analytics of IoT Data
The next state of the evolution of IoT data will
involve integration of a wider variety of IoT data
with more advanced data analytics. Example ap-
plications that are likely to deploy such use of IoT
data include smart buildings and connected ve-
hicles. IoT data has found its use in fleet health
monitoring of aircrafts and in maximizing the op-
erational efficiency.7 Achieving efficiency requires
knowledge of the process (for example, schedul-
ing, demand, pricing, and logistics) and access to
near-real-time data facilitated through IoT.
Physical, Cyber, and Social (PCS) Data
Some of the more advanced applications in consum-
ers, smart city, and healthcare (including personalized
iEEE iNTElliGENT SYSTEmS

digital health) require a broader variety
of sensors of different modalities for col-
lecting data of interest. Furthermore,
the IoT data will be complemented by
social and Web data, such as event in-
formation, collective intelligence (such
as Wikipedia), open data, and curated
knowledge (in the form of taxonomies,
knowledge graphs, or ontologies).
With such progress in IoT appli-
cations, we will face all the classic
big data challenges in more extreme
forms—increasing volume, broader
variety, increasing complexity, rapid
changes (velocity), and more verac-
ity challenges encompassing trust, se-
curity, and privacy. This would then
imply a need to move from raw data
processing to more intelligent data
processing. Applications simply will
not be able to deal with data, so we
will have to sift through it to iden-
tify data that is higher-quality and
more relevant (in terms of appropri-
ate knowledge of context and degree
of personalization). We will have to
look at the data through layers of ab-
stractions. The real opportunity and
challenge related to IoT is not with
the devices and wearables, nor the
networking and infrastructure, nor
even collecting and storing the data—
it is, instead, in making sense of the
data. Let’s explore.
IoT Big Data to Smart IoT
The advancement of IoT usage will likely
be along the following dimensions:
• from single modality or a few types
of sensors to a broad variety of
multimodal sensors and physical-
cyber-social data;
• from simple, predefined, and ex-
pected events, with even specific
solutions to unexpected events and
situations not defined or planned a
priori, to event-agnostic solutions;
• from monitoring and data analy-
sis to predictive and intelligent
march/April 2016 www.computer.org/intelligent 109
Hyperthyroidism
l
efu
us
e
or
m
... Elevated blood pressure ...
...
l…
efu
us
Systolic blood pressure 150 mmHg
ss
Le
“150”
Figure 1. From data to decisions and actions: climbing the data, information,
knowledge, and wisdom (DIKW) ladder. Starting with a sensor reading of “150” to
being able to decide what medication is right for the patient exemplifies a series of
abstractions along the DIKW ladder to make data actionable.
processing, shifting from rules- accurately measured (as estimated),
based maintenance to more predic- intelligent processing implies convert-
tion-driven intervention; and ing massive amounts of raw data into
• from aggregating IoT data on the something contextually relevant or
cloud for intelligent processing to meaningful for situational awareness,
help make decisions and take actions, decision making, or taking actions.
to divesting or downscaling com- Consider the example shown in Fig-
putation to the edge of the network, ure 1. The first (bottom) level shows
gateways, and devices—that is, “in- “150,” which is a blood pressure
telligence at the edge”8 (see http:// reading (sensor and device data). The
wiki.knoesis.org/index.php/PCS). second level shows labeled (semanti-
cally annotated) data or information.
An IoT ecosystem that supports The third level represents knowledge
making sense of all the IoT big data based on the latest National Insti-
is what the research community calls tutes of Health guidance used by cli-
smart IoT (see www.youtube.com/ nicians; this information represents a
watch?v=skX7DjB2lyg). It enables medical condition of “elevated blood
intelligent applications that support pressure.” And yet, this knowledge
higher-quality and timely decision is not actionable—the clinician must
making and actions. What does this decide whether this is due to hyper-
mean? Here is a rough analogy: “The thyroidism or hypertension before a
human body sends 11 million bits per proper medication can be prescribed.
second to the brain for processing, Making sense of physical (includ-
yet the conscious mind seems to be ing IoT), cyber, and social big data
able to process only 50 bits per sec- will involve an intertwined trio of
ond” (http://j.mp/Brain101). Of the computing paradigms (see Figure 2). I
400 billion bits of information per take a pedagogic view for conceptual
second that reach the brain, we are clarity—for example, what we char-
aware of only 2,000 bits (possibly for acterize as a feature of cognitive com-
situational awareness or conscious puting based on our interpretation of
decision making).9 Whether the level contemporary dominant view might
of information compression can be have been considered to be aspects of
 semantic computing in some litera-
ture. If these three forms of comput-
ing are seen as key to intelligent com-
puting, the distinction of one form
to another at their respective interac-
tions will be somewhat arbitrary.
Acquisition and processing of IoT
data offers several challenges:
• heterogeneity, which leads to mul-
tiple sensors, devices, and textual
reports;
• incompleteness, which can hap-
pen when sensors fail or there is
partial coverage due to factors
such as communication or network
limitations;
• uncertainty, because veracity is a
major issue in a distributed setting
like IoT; and
• dynamism (or velocity), which hap-
pens when observations arrive con-
tinuously, reflecting changes in the
physical world.
Addressing these challenges is cru-
cial for a deeper understanding of
the physical world with numerous
applications, including manufactur-
ing, healthcare, sustainability, public
safety, and mobility. Next, let’s ex-
amine the role of semantic, cognitive,
and perceptual computing in dealing
with these IoT challenges and further
define the notion of smart IoT.
Role of Semantic Computing
Semantics is about associating mean-
ing with data. It lets us deal with data
so that we can interpret data in con-
text, despite differences in syntax and
representation or structure. Semantic
computing encompasses technologies
for representing concepts and their
relations that loosely mimic the in-
terrelation of concepts in the human
mind. This conceptual knowledge,
represented formally in an ontology
or as part of a structured knowl-
edge base (increasingly identified as a
110 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS
knowledge graph) can be used to an-
notate data and infer new knowledge
from interpreted data. Additionally,
semantic computing plays a crucial
role in supporting interoperability at
various system architecture levels and
in sensor/IoT data fusion, which in-
cludes integrating multisensory and
multimodal observations from di-
verse sources. Semantic computing
deals with the heterogeneity chal-
lenge of IoT and further enhances in-
terpretation of IoT data.
Some near-term challenges that se-
mantic computing can address can
support interoperability at the device,
networking, and data-exchange lev-
els. These issues can be addressed on
the basis of experiences with similar
challenges from the past. For exam-
ple, Samsung and Google’s collabora-
tion on a low-power wireless network
called Thread uses Bluetooth Smart
to connect devices. Samsung, Dell,
and Intel’s efforts on the Open Inter-
connect Consortium are working to
connect any device with another, re-
gardless of the operating system, con-
nection provider, or form factor.
To make data meaningful, seman-
tic techniques are needed at various
levels in a system architecture. On a
lower level, the Semantic Gateway as
Service effort allows for translation
among various IoT messaging proto-
cols, such as the Extensible Messag-
ing and Presence Protocol (XMPP),
Constrained Application Protocol
(CoAP), and MQTT.10 At a some-
what higher level, an important in-
teroperability standard is provided
by W3C’s Semantic Sensor Net-
work (SSN) ontology and annota-
tion framework,11 pioneered in the
Semantic Sensor Observation Service
and conceived in the Semantic Sen-
sor Web framework (http://knoesis.
org/projects/ssw). It is useful to de-
scribe any sensor or device and its
data in a standard form and support
semantic annotations of sensor data,
making that data more meaningful.
In essence, this provides semantic
interoperability among messages
carrying IoT data. W3C has paired
with the Open Geospatial Consor-
tium to make an international stan-
dard on Spatial Data on the Web,
with SSN as the primary input, whose
outcomes will further aid semantic
computing.
Role of Cognitive Computing
In 2002, DARPA defined a cognitive
system as one that can “reason, use
represented knowledge, learn from
experience, accumulate knowledge,
explain itself, accept direction, be
aware of its own behavior and capa-
bilities as well as respond in a robust
manner to surprises.”12 Cognitive al-
gorithms interpret data by learning
and matching patterns in a way that
loosely mimics the process of cogni-
tion in the human mind. Cognitive
systems learn from their experiences
and then get better when performing
repeated tasks. Through data min-
ing, pattern recognition, and natural
language processing, cognitive com-
puting is progressing rapidly toward
developing technology to support our
ability to answer complex questions.
Cognitive computing acts as pros-
thetics for human cognition by ana-
lyzing a massive amount of data and
being able to answer questions hu-
mans might have when making cer-
tain decisions.
With the success of IBM Watson
in defeating human champions on
Jeopardy! in 2011, IBM has used the
term “cognitive computing” for a sys-
tem that can ingest multimodal data,
understand natural language, gen-
erate and evaluate hypotheses, and
learn through interactions with hu-
mans. With IoT being pervasive across
various domains and having the po-
tential to deliver value to businesses,

cognitive IoT extends the idea of cog-
nitive computing to IoT. Cognitive
IoT aspires to add additional “senses”
comprising various sensors, devices,
and unstructured data on the IoT to
IBM Watson to help leverage value
from IoT data.4 IBM and its partners
have explored cognitive IoT in fitness
and well-being scenarios.
Although semantic computing ad-
dresses interoperability and integration
aspects of heterogeneity, cognitive com-
puting can address complementary as-
pects, such as hypothesizing correlations
and validating them through evidence.
Combining multiple data sources can
support dealing with incompleteness.
The degree of confidence the cognitive
computing system has in its observa-
tions quantifies uncertainty. The cogni-
tive computing system does not explic-
itly address the dynamism challenge of
IoT. Specifically, with IoT, the changes
in the physical world are reflected in
the observations being collected from
the physical world. Understanding the
current state of the world (that is, situ-
ational awareness) plays a crucial role in
deriving value from data. Some, includ-
ing IBM, consider natural-language-
based interaction with IoT data to be a
key feature of cognitive IoT.
Role of Perceptual
Computing
Socrates taught that knowledge is at-
tained through the careful and delib-
erate process of asking and answer-
ing questions. Through data mining,
pattern recognition, and natural lan-
guage processing, cognitive computing
is rapidly progressing toward develop-
ing technology to support our ability
to answer complex questions. To this,
perceptual computing provides an im-
portant complementary capability for
deriving intelligence from data by pro-
viding a technology to support our
ability to ask contextually relevant and
personalized questions. Making sense
march/April 2016 www.computer.org/intelligent 111
of observations from the physical world
is formulated as an iterative abductive-
deductive reasoning process13; focusing
and explaining based on the observed
data is also critical in ascending ab-
straction levels such as those shown in
Figure 1.
Context is better captured through
continuous observations from sensors
around us. For example, a mobile phone
has an accelerometer, microphone, tem-
perature, and location sensors that pro-
vide contextual information. Personal-
ization deals with learning preferences
and choices by people while interacting
with the physical world. The use of con-
textualization and personalization to
ask the right questions in various sce-
narios is an important aspect of percep-
tual computing.
A perceptual computing system’s
contextual and personalized explora-
tion of the physical world addresses
the dynamism challenge of IoT. Un-
derstanding contextual information
is mostly a bottom-up process. Based
on the contextual understanding, ap-
plying personalized recommenda-
tions is mostly top down. This inter-
leaving of top-down and bottom-up
processing complements cognitive
IoT systems. A personalized digital
medicine example of SmartIoT that
is empowered by complementary ca-
pabilities of semantic, cognitive, and
perceptual computing is the use case
of asthma in children.14
I oT has rapidly become a source of
massive growth in data and is comple-
mented by other Web and social data
to build exciting applications for con-
sumers and enterprise users. There is
an equally strong business appetite for
solving new challenges to improve ser-
vices and increase revenue in a grow-
ing number of industries and mar-
kets. This is reflected in numerous
ways, including exciting new product
Smart data:
Get insight, take action
Computing
for intelligent
systems
Perceptual
Cognitive Semantic
Multimodal PCS data
Figure 2. Semantic, cognitive, and
perceptual computing: converting
physical, cyber, and social (PCS) big data
into smart data (http://j.mp/SmData),
giving deeper insights and making
data actionable. Emerging computing
paradigms for building intelligent
systems, including intelligent machines,
will rely on synergistic use of the three
types of computing.
demonstrations at events such as the
2016 Consumer Electronics Show,
coverage at the 2016 World Economic
Forum, the astounding IoT device
growth and market size predictions in
the business press, and numerous ar-
ticles in technical media covering IoT-
powered intelligent machines and ap-
plications. This growth and potential
have energized research and technol-
ogy development centered on aspects
of artificial intelligence that will be
used to build future intelligent sys-
tems. This article provides a concep-
tual framework covering three impor-
tant components: semantic, cognitive,
and perceptual computing. During
the past decade, we have seen early
promise, and we can expect to see tre-
mendous growth during the coming
decade.
Following this first installment, the
IoT department will continue to explore
various topics for building IoT-based in-
telligent systems. The next article in the
series will deal with device- and net-
working-level challenges.
 Acknowledgments
This article benefited from brainstorming
with my PhD students Pramod Anantharam
and Cory Henson and captures some of the
broad themes explored in more detail in
their papers and dissertations. This research
is partly supported by NSF and NIH grants,
including NSF Award EAR 1520870. Opin-
ions are solely those of the author.
References
1. A. Sheth, “15 Years of Semantic
Search and Ontology-Enabled
Semantic Applications,” blog, 27
Sept. 2014; http://amitsheth.blogspot.
com/2014/09/15-years-of-semantic-
search-and.html.
2. B. Marr, “17 ‘Internet Of Things’ Facts
Everyone Should Read,” Forbes, 27
Oct. 2015.
3. “Internet of Things Market
Statistics–2015,” blog, 5 Mar. 2015;
www.ironpaper.com/webintel/articles/
internet-things-market-statistics-2015.
4. “IBM Opens Watson IoT Global
Headquarters, Extends Power of
Cognitive Computing to a Connected
World,” press release, IBM, 15 Dec.
2015; www-03.ibm.com/press/us/en/
pressrelease/48443.wss.
5. “Internet of Things Spending
Forecast to Reach Nearly $1.3
Trillion in 2019 Led by Widespread
Initiatives and Outlays Across Asia/
Pacific,” press release, IDC, 10
Dec. 2015; www.idc.com/getdoc.
jsp?containerId=prUS40782915.
6. T. Smith, “Problems Solved by IoT,”
blog, 24 Sept. 2015; http://dzone.com/
articles/20-real-world-problems-solved-
by-iot.
7. D. Robb, “Will The Internet Of Things
Revolutionize The Aircraft Industry?,”
Forbes, 23 June 2014.
8. C. Henson, K. Thirunarayan, and
A. Sheth, “An Efficient Bit Vector
Approach to Semantics-Based
Machine Perception in Resource-
Constrained Devices,” The Semantic
Web–ISWC 2012, LNCS 7649, 2012,
pp. 149–164.
112 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS
9. J. Dispenza, Evolve Your Brain: The
Science of Changing Your Mind, HCI,
2007.
10. P. Desai, A. Sheth, and P. Anantharam,
SUBMIT
“Semantic Gateway as a Service
TODAY
Architecture for IoT Interoperability,
Semantic Gateway as a Service
Architecture for IoT Interoperability,”
IEEE TRANSACTIONS ON
IEEE Int’l Conf. Mobile Services, 2015,
pp. 313–319. BIG DATA
11. M. Compton et al., “The SSN Ontology
of the W3C Semantic Sensor Network SUBSCRIBE
Incubator Group,” J. Web Semantics, AND SUBMIT
vol. 17, 2012, pp. 25–32.
12. R.C. Johnson, “Darpa Puts Thought For more information
on paper submission,
into Cognitive Computing,” EE Times, featured articles, call-for-
9 Dec. 2002; www.eetimes.com/ papers, and subscription
document.asp?doc_id=1227413. links visit:
13. C. Henson, A. Sheth, and K.
www.computer.org/tbd
Thirunarayan, “Semantic Perception:
Converting Sensory Observations
to Abstractions,” IEEE Internet
Computing, vol. 16, no. 2, 2012, pp.
26–34.
14. A. Sheth, P. Anantharam, and C.
Henson, “Semantic, Cognitive, and
Perceptual Computing: Advances
Toward Computing for Human
Experience,” to be published in
Computer, Mar. 2016.
TBD is financially cosponsored
by IEEE Computer Society, IEEE
Amit Sheth is an educator, researcher, and Communications Society, IEEE
Computational Intelligence Society,
entrepreneur. He is the executive director IEEE Sensors Council, IEEE Consumer
Electronics Society, IEEE Signal
of Kno.e.sis, the LexisNexis Ohio Eminent Processing Society, IEEE Systems,
Man & Cybernetics Society, IEEE
Systems Council, IEEE Vehicular
Scholar at Wright State University, and an Technology Society
IEEE Fellow. Contact him at amit@knoesis. TBD is technically cosponsored by
IEEE Control Systems Society, IEEE
org; http://knoesis.org/amit. Photonics Society, IEEE Engineering
in Medicine & Biology Society, IEEE
Power & Energy Society, and IEEE
Biometics Council
Selected CS articles and columns
are also available for free at
http://ComputingNow.computer.org.