MAY 2019 | WWW.THE-SCIENTIST.

COM

AI TACKLES
BIOLOGY
HOW MACHINE LEARNING WILL
REVOLUTIONIZE SCIENCE AND MEDICINE

BUILDING BRAINS
WITH SILICON

ALGORITHMS
FOR CANCER

DATA DIVING
IN BIOLOGICAL
IMAGES

PLUS
ARTIFICIALLY INTELLIGENT
DRUG DEVELOPMENT
HUNT FOR 10-PLEX
BIOMARKERS
WITH UNMATCHED
SENSITIVITY
With Quanterix SP-X™, unparalleled Simoa™ 10-plex
detection of circulating protein biomarkers is now possible
at the earliest stages of disease progression – even at
healthy baseline levels

Introducing the Quanterix SP-X Imaging and Analysis System,

detection at both acute and baseline levels. With the new SP-X,
oncology and immuno-oncology researchers and others who
rely on robust multiplexing capabilities now have access to
next generation Simoa planar technology in an easy-to-use

research, and ultimately accelerate drug approvals.

Visit quanterix.com/SP-X
for more information.

Quanterix.com | © 2018 Quanterix, Inc. SP-X™ is a registered trademark of Quanterix, Inc.

For research use only. Not for diagnostic procedures.
 Vi-CELL BLU
Cell Viability/Counting Analyzer
does more work
in less time, using
fewer resources

50% faster
sample
processing
speeds in FAST
uses sample mode
volumes as
low as 170 μL
in FAST mode

improved
greater sample
instrument-
capacity-choose
to-instrument
between a
comparability
24-position
carousel and
a 96-well
plate loader

21.2"

16.5"

Different by (your) design

The new Vi-CELL BLU Cell Viability Analyzer from Beckman Coulter Life Sciences.
Before we upgraded our Vi-CELL XR, we asked labs around the world what improvements they wanted
to see. As a result, our new Vi-CELL BLU is a next-generation cell viability analyzer with faster processing
speeds, expanded sample capacity (choose between a 24-position carousel and a 96-well plate), minimized
sample test volumes (170 μL in FAST mode), improved instrument-to-instrument comparability, and a
smaller footprint with on-board computer and monitor (just 16.5” x 21”, 42 cm x 54 cm)
Learn more about Vi-CELL BLU now, at beckman.com/blu

© 2019 Beckman Coulter, Inc. All rights reserved. Beckman Coulter and the stylized logo are trademarks or
registered trademarks of Beckman Coulter, Inc. in the United States and other countries.
PART-3390ADV02.18
 be INSPIRED
drive DISCOVERY
stay GENUINE

You’ll be
thrilled to pieces.
NEBNext Ultra II FS DNA Library Prep Kit
® ™

Novel Enzymatic Fragmentation System

Do you need a faster, more reliable solution for NEBNext Ultra II FS DNA produces the highest yields,
DNA fragmentation and library construction? from a range of input amounts
The NEBNext Ultra II FS DNA Library Prep
140

Kit meets the dual challenge of generating NEBNext Ultra™ II FS

®

Kapa™ HyperPlus
120
Covaris
®

high quality next gen sequencing libraries from 100

Illumina Nextera
® ®
Library Yield (nM)

ever-decreasing input amounts AND simple 80

scalability. Enzymatic shearing increases library 60

yields by reducing DNA damage and sample loss.

40

20

Our novel fragmentation reagent is combined 0

with end repair and dA-tailing reagents, and a PCR cycles

DNA input
13
100 pg
10
500 pg
9
1 ng
5
50 ng
4
100 ng
3
500 ng

single protocol is used for a wide range of input

amounts (100 pg – 500 ng) and sample types.
You’ll be thrilled to pieces with the result – a reliable,
flexible, high-quality library prep that is fast
and scalable.
Libraries were prepared from Human NA19240 genomic DNA using the input amounts and
numbers of PCR cycles shown. For NEBNext Ultra II FS, a 20-minute fragmentation time
was used. For Kapa HyperPlus libraries, input DNA was cleaned up with 3X beads prior to
library construction, as recommended, and a 20-minute fragmentation time was used. Illumina
recommends 50 ng input for Nextera, and not an input range; therefore, only 50 ng was used
in this experiment. “Covaris” libraries were prepared by shearing each input amount in 1X TE
Buffer to an insert size of ~200 bp using a Covaris instrument, followed by library construction
using the NEBNext Ultra II DNA Library Prep Kit (NEB #E7645). Error bars indicate
standard deviation for an average of 3–6 replicates performed by 2 independent users.

Visit NEBNextUltraII.com One or more of these products are covered by patents, trademarks and/or copyrights owned or controlled by New England Biolabs, Inc.
For more information, please email us at gbd@neb.com.
to request your sample today. The use of these products may require you to obtain additional third party intellectual property rights for certain applications.
ILLUMINA® and NEXTERA® are registered trademarks of Illumina, Inc.
KAPA™ is a trademark of Kapa Biosystems. COVARIS® is a registered trademark of Covaris, Inc.
© Copyright 2017, New England Biolabs, Inc.; all rights reserved.
 MAY 2019

Contents
THE SCIENTIST THE-SCIENTIST.COM VOLUME 33 NUMBER 05
© MOLLY MENDOZA; © HEIDELBERG UNIVERSITY; MODIFIED FROM © ISTOCK.COM

Features ON THE COVER: © ANDRIY ONUFRIYENKO, GETTY IMAGES

22
Run Program: Cancer
28
A Silicon Brain
36
A Deeper Look
Using input from images to -omes, Biology-inspired computer chips With the help of computer programs
artificial intelligence can find patterns in may help scientists better simulate that learn from experience, researchers
tumors and generate prognoses where neurological function. look for meaning in vast volumes of
human clinicians see only a jumble of data. BY SANDEEP RAVINDRAN image data.
BY AMBER DANCE BY JEF AKST AND CAROLYN WILKE

05. 201 9 | T H E S C IE N T IST 3

 MAY 2019

Department Contents
10 FROM THE EDITOR 52 LAB TOOLS
Cerebral Inception Automating the Fight Against
Our brains have evolved to the point Superbugs
where we can build artificial brains Antibiotic resistance is on the rise.
that might help us understand our Can AI help?
brains. Let that sink in. BY AMBER DANCE
BY BOB GRANT
56 BIO BUSINESS
12 THE BASICS Computing a Cure
Artificial Intelligence: An The pharmaceutical industry is look-
Introduction ing to machine learning to overcome
A brief history of AI, machine complex challenges in drug
learning, artificial neural networks, development.
and deep learning BY BIANCA NOGRADY

13 BY JEF AKST
60 READING FRAMES
13 Robots and Eureka Moments
46
NOTEBOOK
Going Through the Motions; Listening Artificial intelligence can be trained
MURTHY LAB AND SHAEVITZ LAB, PRINCETON UNIVERSITY; © ISTOCK.COM, JXFZSY; © JANE SHAUCK PHOTOGRAPHY

for Your Health; A Better Beehive; to recognize patterns and predict

Into the Cell results. But will machine learning
ever be able to make novel scientific
20 MODUS OPERANDI discoveries?
Microbiology Meets Machine BY KARTIK HOSANAGAR
Learning
Artificially intelligent software 64 FOUNDATIONS
augments the study of host-pathogen Learning Machine, 1951
interactions. BY JEF AKST
BY RUTH WILLIAMS
IN EVERY ISSUE
21 CRITIC AT LARGE 9 CONTRIBUTORS
AI Versus Animal Testing 11 SPEAKING OF SCIENCE
Machine learning could be the key 62 THE GUIDE
to reducing the use of animals in 63 RECRUITMENT
51 experiments. ANSWER
BY THOMAS HARTUNG
PUZZLE ON PAGE 11

46 THE LITERATURE S T I RRU P P A P E R

AI for filtering out sequence noise; A R O L A O A
algorithms predict CRISPR repairs L L AMA A NN E L I D
M Q R C S E I
OX I D I Z E Y UCC A
48 PROFILE N N B A L
Bob Murphy: Autopilot Advocate A NGKO RWA T
BY SHAWNA WILLIAMS J N E A A
ORD E R F OR T R A N
51 SCIENTIST TO WATCH V R O F T O G
Nick Turk-Browne: Pattern Seeker I SO T OP E H EGE L
BY CATHERINE OFFORD
A I T C O U E
NOD E S T I GR E S S

05. 201 9 | T H E S C IE N T IST 5

 MAY 2019

Online Contents

THIS MONTH AT THE-SCIENTIST.COM:

VIDEO VIDEO SLIDESHOW

Murphy on AI for Biology
See Robert Murphy of Carnegie Mellon
University discuss the revolutionary
application of machine learning to
biomedical research.
A Window on Memory
Watch Princeton University’s Nicholas
Turk-Browne describe his research on
how the human brain makes, stores, and
adjusts memories.
AI Photo Album
Peruse some images that artificial
intelligence is sifting through for clues
about the biology they depict.

AS ALWAYS, FIND BREAKING NEWS EVERY DAY ON OUR WEBSITE.

Coming in June

SNAPSHOT SERENGETI; © ISTOCK.COM, WACOMKA

HERE’S WHAT YOU’LL FIND IN NEXT MONTH’S ISSUE

• The new frontier of quantum biology

• How the body tolerates disease instead of fighting it

• Can microbiome-produced antigens lead to autoimmunity?

• Inclusivity in academia

AND MUCH MORE

6 T H E SC I EN TIST | the-scientist.com
 WE REMOVED
THE COMPRESSORS.
Y O U ’ R E W E L C O M E.

For decades, you’ve protected your life’s work with technology destined to fail. Like you,
we don’t accept the status quo. Which is why we invented the first and only compressor-free,
ultra-low temperature freezer. With the uncomplicated, free-piston Stirling engine, there’s virtually
nothing to fail. It’s an engine that has created breakthroughs in every aspect of ULT technology:
in performance, energy savings, sustainability, total cost of ownership and sample safety.

Learn more about breakthroughs in ULT technology at NoCompressors.com.

 415 Madison Avenue,
Suite 1508,
New York, NY
10017
E-mail: info@the-scientist.com

EDITORIAL DESIGN ADVERTISING, MARKETING, EDITORIAL ADVISORY BOARD

EDITOR-IN-CHIEF AND PRODUCTION ADMINISTRATION Roger Beachy
Bob Grant PRODUCTION MANAGER ASSOCIATE SALES DIRECTOR Donald Danforth Plant Science
rgrant@the-scientist.com Greg Brewer Key Accounts Center
gregb@the-scientist.com Ashley Haire Steven A. Bent
MANAGING EDITOR ashleyh@the-scientist.com Foley and Lardner LLP
Jef Akst ART DIRECTOR
SENIOR ACCOUNT Deborah Blum
jakst@the-scientist.com Erin Lemieux
EXECUTIVES University of Wisconsin
elemieux@the-scientist.com
SENIOR EDITOR Northeast US, Eastern
Annette Doherty
Kerry Grens INTERIM ART DIRECTOR Canada, Europe, ROW, Pfizer Global Research
kgrens@the-scientist.com Luiza Augusto Careers/Recruitment and Development
laugusto@labx.com Melanie Dunlop
ASSOCIATE EDITORS Kevin Horgan
melanied@the-scientist.com GE Healthcare
Catherine Offord VIDEO PRODUCTION
cofford@the-scientist.com COORDINATOR Western US and Steve Jackson
Ryan Kyle Western Canada University of Cambridge
Shawna Williams rkyle@labx.com Karen Evans
swilliams@the-scientist.com Simon Levin
kevans@the-scientist.com
Princeton University Center
Ashley Yeager CREATIVE SERVICES ACCOUNT EXECUTIVE for BioComplexity
ayeager@the-scientist.com Midwest and Southeast US
DIRECTOR Edison Liu
CONTRIBUTING EDITOR Elizabeth Young Anita Bell Genome Institute of Singapore
Alla Katsnelson eyoung@the-scientist.com abell@the-scientist.com
Peter Raven
SENIOR SCIENTIFIC DIRECTOR OF MARKETING Missouri Botanical Garden
COPY EDITOR
TECHNICAL EDITOR Alex Maranduik Joseph Schlessinger
Annie Gottlieb
Nathan Ni amaranduik@labx.com Yale University School
CORRESPONDENTS nni@the-scientist.com of Medicine
AUDIENCE DEVELOPMENT
Anna Azvolinsky SCIENTIFIC TECHNICAL EDITORS SPECIALIST J. Craig Venter
Abby Olena Kathryn Loydall Matthew Gale J. Craig Venter Institute
Ruth Williams kloydall@the-scientist.com mgale@labx.com Marc Vidal
Dana Farber Cancer Institute
INTERNS SOCIAL MEDIA EDITOR EVENTS MANAGER
Harvard University
Chia-Yi Hou Lisa Winter Cayley Thomas
Carolyn Wilke lwinter@the-scientist.com cayleyt@labx.com H. Steven Wiley
Biomolecular Systems Pacific
WEBINARS & SOCIAL MEDIA SALES AND MARKETING Northwest National Laboratory
MANAGEMENT
COORDINATOR COORDINATOR
AND BUSINESS Alastair J.J. Wood
Meaghan Brownley Katie Prud’homme Symphony Capital
PRESIDENT mbrownley@labx.com katiep@the-scientist.com
Bob Kafato CUSTOMER SERVICE
SOCIAL MEDIA SUBSCRIPTION RATES & SERVICES
bobk@labx.com info@the-scientist.com
COORDINATOR In the United States & Canada individual subscriptions:
$39.95. Rest of the world: air cargo add $25.
GENERAL MANAGER Catherine Rocheleau
Ken Piech crocheleau@labx.com For assistance with a new or existing subscription
please contact us at:
kenp@labx.com
Phone: 847.513.6029
Fax: 847.763.9674
MANAGING PARTNER E-mail: thescientist@halldata.com
Mario Di Ubaldi Mail: The Scientist, PO Box 2015, Skokie, Illinois 60076
mariod@the-scientist.com For institutional subscription rates and services visit
www.the-scientist.com/info/subscribe or
VICE PRESIDENT e-mail institutions@the-scientist.com
GROUP PUBLISHING DIRECTOR
LIST RENTALS
Robert S. D’Angelo Contact Statlistics, Jennifer Felling at
rdangelo@the-scientist.com 203-778-8700 or j.felling@statlistics.com

REPRINTS
Contact Katie Prud’homme at katiep@the-scientist.com
POSTMASTER: Send address changes to The Scientist, PO Box 2015, Skokie, Illinois 60076. Canada Publications Agreement #40641071 The Scientist is indexed in
Current Contents, Science Citation Index, BasicBIOS IS, and other databases. Articles published in The Scientist reflect the views of their authors and are not the official views PERMISSIONS
of the publication, its editorial staff, or its ownership. The Scientist is a registered trademark of LabX Media Group Inc. The Scientist® (ISSN 0890-3670) is published monthly.
For photocopy and reprint permissions, contact
Advertising Office: The Scientist, 415 Madison Avenue, Suite 1508, New York, NY 10017. Periodical Postage Paid at New York, NY, and at additional mailing offices. Copyright Clearance Center at www.copyright.com

8 T H E SCI ENT I ST | the-scientist.com

 MAY 2019

Contributors
As a kid, Molly Mendoza loved to watch cartoons such as Sailor Moon and Speed Racer and to read Japanese
comics. “From a really young age I wanted to tell stories with pictures,” she recalls. She didn’t really consider
making a career out of it, though, until her high school art teachers and college professors told her she had tal-
ent. With their encouragement, she decided to pursue a bachelor’s of fine arts degree at the Pacific Northwest
College of Art in Portland, Oregon. After graduating in the winter of 2014, Mendoza went straight into free-
lancing, creating banner images for different tutorials within the creative suite of Adobe, as well as illustrating
for publications such as Nautilus and The Scientist. In May 2015, her work graced the cover of The Scientist’s
special issue on HIV. And on page 22 of this issue, she illustrates how artificial intelligence (AI) technologies
are transforming masses of data on cancer into meaningful information for patients.
In addition to freelancing, Mendoza works part-time at Land Gallery, an online and brick-and-mortar
retail space for independent artists in and around Portland, where she has settled. And this July, she will pub-
lish her first graphic novel, Skip, about two people hopping between dimensions as they try to find their way
home. “It’s an adventure story,” she says, but it’s also more than that. “Along the way they realize each other’s
flaws and fears but also what about them makes them special and what they should be proud of.”

Kartik Hosanagar, professor of technology and digital business at the Wharton School of the University
of Pennsylvania, has had a front row seat for the artificial-intelligence revolution. In addition to studying
how analytics and algorithms affect the behavior of consumers and society, he’s launched his own compa-
nies, built around automating our lives. For example, in 2005 he cofounded Yodle, a marketing platform
for small businesses that was designed to take the place of traditional phonebook searches performed
by people looking for particular products or services. Powered by Hosanagar’s custom-built algorithms,
Yodle was acquired by web.com in 2016.
Although he doesn’t specifically research the interface of AI and biology, Hosanagar says that he has
an ongoing study that looks at how healthcare professionals incorporate AI into their work. He says his
research asks questions such as, “What does transparency or model interpretations do to an expert’s deci-
sion to adopt [AI technology]?”
Hosanagar took two years to write his first book, A Human’s Guide to Machine Intelligence. He says that his
impetus for penning the work was to demystify AI for a lay audience. “I felt like there’s a lot of excitement about
AI, but a lot of the discussion is either incomplete or often imprecise.” On page 60 he writes about the potential
for AI to make scientific discoveries, something he says could happen in the near future. “It’s a matter of maybe
two to three years before we start seeing the first good examples coming out.”
CELESTE NOCHE; THE WHARTON SCHOOL; SANDEEP RAVINDRAN

Though he had spent much of his early life with a foot in two worlds—science and writing— Sandeep
Ravindran thought he would have to eventually choose between them. Upon completing a bachelor’s degree in
biological sciences at Cornell University, Ravindran simultaneously considered PhD programs and master of fine
arts programs in creative writing. He ultimately decided to pursue his PhD in microbiology and immunology at
Stanford University. But a talk by a science writer late in his training made Ravindran realize he could combine
the best of both worlds. “I decided right then that [science writing was] kind of perfect for me,” he says.
After finishing his doctorate, Ravindran completed internships at Popular Science and Science Illustrated,
before pursuing additional training through the UC Santa Cruz science communication program. After com-
pleting internships at a host of outlets including the San Jose Mercury News and Science News, he worked for
two years as a science writer at PNAS. Now Ravindran works as a freelance science journalist based out of
Washington, DC, delving into any topics that capture his interest, which often lie in the realm of life sciences.
“I’m still fascinated by biology in general, both in terms of how the natural world finds ways, through evolu-
tion, to solve problems, and how humans emulate those ways using technology,” he says. Over the past 3 years,
Ravindran has written several features and Lab Tools pieces for The Scientist. Find his latest feature about
brain-inspired computing on page 28 of this issue.

05 . 2019 | T H E S C IE N T IST 9
 FROM THE EDITOR
Cerebral Inception
Our brains have evolved to the point where we can build artificial
brains that might help us understand our brains. Let that sink in.
BY BOB GRANT
Y
ou’d think that overseeing an entire issue of The Sci-
entist focused on artificial intelligence would cause
my mind to wander far into the future—robotic
researchers formulating digital hypotheses, whizzing about
in sleek, metallic labs. But immersing myself in stories
about the novel insights and deep analyses enabled by smart
instruments and machine learning did not transport me into
a vision of science in the 23rd century.
Instead, I found myself thinking of the distant past, of a
time when the first micro-vibrations of life were roiling the
raw muck of early Earth. Rather than the grand sweep of
what artificial intelligence may bring about—faster and more
economical data processing, new insights, novel discoveries,
and revolutionized workflows and transportation systems—
I thought of the original form of intelligence on our planet.
Intelligence on the molecular scale.
The Miller-Urey experiment of the 1950s began to
shed mechanistic light on the dark mysteries of how Earth
changed from an inanimate sphere to a planet bursting with
life. But those famed researchers could zap into existence
only amino acids, the building blocks of proteins. Later
experiments pushed the chemical evolution toward life fur-
ther by generating nitrogenous bases, the building blocks
of RNA and DNA. But researchers have not yet succeeded
in demonstrating a route from raw chemical materials to
those crucial macromolecules. And it was through RNA,
DNA, or both that life really burst from the starting blocks,
those plucky nucleic acids acquiring a sort of self-motivation
to replicate. That, I believe, was the dawn of intelligence.
RNA and DNA, past and present, harbor a chemical drive
to persist and reproduce that has woven a thread of life on
Earth through the eons, unbroken.
Fast-forward the evolutionary clock by 4 billion or so
years, and you arrive at today. In this issue, you’ll read of
fascinating insights into basic biology, land management,
and clinical practice facilitated by AI algorithms. Such mod-
els are on the precipice of helping physicians diagnose and
treat cancer (see “Run Program: Cancer,” page 22), pre-
dicting wildfire spread (see “A Deeper Look,” page 36), and
improving the performance of CRISPR-Cas9 genome edit-
ing (see “CRISPR Predictions,” page 46).
But it is the construction of advanced neural net-
works that attempt to mirror the architecture and func-
tion of the human brain (see “A Silicon Brain,” page 28)
that truly blows my meaty mind. Scientists have progressed
10 T H E SC I EN TIST | the-scientist.com
to the point of assem-
bling models of brains,
built with inanimate
components, that
simulate the func-
tioning of their bio-
logical counterparts.
These brain mimics
can perform impres-
sive feats of pattern
recognition, calcu-
lation, and decision
making. Some even
border on human-like
intelligence, learning and adapting their behavior based
on inputs they receive through sophisticated sensors. As
researchers and engineers develop AI machines, from
self-driving cars to self-driving microscopes (see “Autopi-
lot Advocate,” page 48), at a breakneck pace, it’s how this
technology informs our understanding of human biology
that interests me most.
The intellectual feedback loop involved is what really
gets me. Biology has evolved in fits and starts—species com-
ing and going, adaptations developed and obviated by natu-
ral selection. Eventually, curious apes appeared on the scene,
and, by dint of luck and environmental happenstance, their
lineage branched and changed until our species, Homo sapi-
ens, arrived, persisted, and spread. In the space of 1,000 or
so centuries, we became a singular force in the biosphere.
Although not the most numerous organism on Earth, our spe-
cies is certainly the one that has made the most forceful and
lasting changes to our planet and its inhabitants. And now
we’ve arrived at a point where a human brain, made from that
same muck that gave rise to the first life forms, can conceive of,
build, and manipulate models of itself that serve to teach that
same lump of biological matter about its own inner workings.
The master is becoming the student. g
ANDRZEJ KRAUZE
Editor-in-Chief
eic@the-scientist.com
 QUOTES

Speaking of Science
1 2 3 4 5 6 7 This is a technology with the
potential to change history for
all of us. The question is, ‘Can we
8 9 have the good without the bad?’
—Former Google AI chief Fei-Fei Li speaking in mid-March
at the opening of Stanford University’s new Institute for
Human-Centered Artificial Intelligence (March 19)
10 11

Will it be possible, in the

Note: The answer grid will include every letter of the alphabet.

foreseeable future, to build a

12 13
machine that can discover physics
14 15 or mathematics that the brightest
humans alive are not able to do
16 17 18 19
on their own, using biological
hardware? Will the future of
science eventually necessarily be
20 21
driven by machines that operate
on a level that we can never reach?
22 23
I don’t know. It’s a good question.
—Kevin Schawinski, an astrophysicist and
CEO/founder of artificial-intelligence company
BY EMILY COX AND HENRY RATHVON Modulos, talking to Quanta Magazine about how AI is
poised to change science (March 11)

ACROSS DOWN
1. Innermost ossicle of the middle ear 1. Swimmer using a ladder, maybe
5. Nest material for some wasps 2. Modern-day Mesopotamian
8. Camel’s South American cousin 3. Vocalizing like a 23-Across
9. Segmented worm 4. Relevant phenomenon in drug
10. Form rust, say testing (2 wds.)
11. Shrub pollinated by a moth 5. Flower in the violet family
12. Temple pictured on the 6. Fetid ferret relative
Cambodian flag (2 wds.) 7. Jellyfish’s kind of symmetry
16. Carnivora, for dogs, e.g. 12. Robot in human form
18. Programming language of the 1950s 13. Wild sub-Saharan pig
20. Variant of a chemical element 14. Like the Galilean moons
21. German thinker who wrote Science 15. Quadrilateral quartet
of Logic 17. Underground network
22. Tree-graft sites 19. Elephant loner
JONNY HAWKINS

23. Mother of striped cubs

Answer key on page 5

05.2019 | T H E S C IE N T IST 1 1
 THE BASICS
Artificial Intelligence: An Introduction
A brief history of AI, machine learning, artificial neural networks, and deep learning
BY JEF AKST
T
he term “artificial intelligence” dates back to the mid-
1950s, when mathematician John McCarthy, widely rec-
ognized as the father of AI, used it to describe machines
that do things people might call intelligent. He and Marvin Min-
sky, whose work was just as influential in the AI field, organized
the Dartmouth Summer Research Project on Artificial Intel-
ligence in 1956. A few years later, with McCarthy on the fac-
ulty, MIT founded its Artificial Intelligence Project, later the AI
Lab. It merged with the Laboratory for Computer Science (LCS)
in 2003 and was renamed the Computer Science and Artificial
Intelligence Laboratory, or CSAIL.
Now a ubiquitous part of modern society, AI refers to any
machine that is able to replicate human cognitive skills, such as
12 T H E SC I EN TIST | the-scientist.com
problem solving. Over the second half of the 20th century, machine
learning emerged as a powerful AI approach that allows computers
to, as the name implies, learn from input data without having to be
explicitly programmed. One technique used in machine learning
is a neural network, which draws inspiration from the biology of
the brain, relaying information between layers of so-called artifi-
cial neurons. The very first artificial neural network was created by
Minsky as a graduate student in 1951 (see “Learning Machine, 1951”
on page 64), but the approach was limited at first, and even Min-
sky himself soon turned his focus to other approaches for creating
intelligent machines. In recent years, neural networks have made a
comeback, particularly for a form of machine learning called deep
learning, which can use very large, complex neural networks.
ARTIFICIAL INTELLIGENCE
An attribute of machines that embody a
form of intelligence, rather than simply
carrying out computations that are input
by human users.
Early applications of AI included machines
that could play games such as checkers and
chess and programs that could analyze and
reproduce language.
MACHINE LEARNING
An approach to AI in which an algorithm
learns to make predictions from data that
is fed into the system.
From personalized news feeds to traffic pre-
diction maps, most people in developed coun-
tries use machine learning–based technolo-
gies every day.
NEURAL NETWORKS
A machine learning approach in which
algorithms process signals via intercon-
nected nodes called artificial neurons.
Because they mimic the architecture of bio-
logical nervous systems, artificial neural net-
works are the obvious method of choice for
modeling the brain.
DEEP LEARNING
A form of machine learning that often
uses a network with many layers of
THE SCIENTIST STAFF
computation—a deep neural network—
enabling an algorithm to powerfully
analyze the input data.
Deep neural networks are responsible
for self-driving vehicles, which learn to
recognize traffic signs, as well as for
voice-controlled virtual assistants.
 NEWS AND ANALYSIS
Notebook
Going Through
the Motions
MURTHY LAB AND SHAEVITZ LAB, PRINCETON UNIVERSITY
I
t takes an average of 17 minutes for a
fruit fly couple to move from meeting
to mating, says Talmo Pereira, a PhD
student studying neuroscience in Joshua
Shaevitz’s lab at Princeton University.
The encounter is marked by “lots of com-
plex stages, arguably more complex than
human courtship,” he says. A male and
a female Drosophila melanogaster first
size each other up through an exchange
of pheromones. If they’re compatible,
the male chases the female down and
woos her by “singing” with a wing that
he vibrates in particular patterns to form
the notes of his ballad. Then the partners
dance, running and circling each other.
Finally, the male attempts to copulate
and the female accepts or rejects.
Pereira is studying how the court-
ship song and dance are represented in
the brains of the flies. Along the way,
he and colleagues developed a power-
ful method to track animal behavior.
Their tool, LEAP Estimates Animal
Pose (LEAP), harnesses a type of arti-
ficial intelligence called a deep neural
network, essentially a “fancy machine
that can learn to do any . . . arbitrary
operation that you can train it to do,”
says Diego Aldarondo, currently a PhD
student at Harvard University, who
built the tool with Pereira during his
MAY 2019
A FLY’S TRACKS: An algorithm that auto-
matically tracks an animal’s limb movements
could help scientists study the link between
brain and behavior.
undergraduate studies at Princeton.
“We developed all this crazy artificial
intelligence just to try to understand
fly sex,” jokes Pereira. “Or not even sex
really, just what leads up to it.”
Traditionally, researchers have col-
lected data on animal movements by
clicking through video footage, frame
by frame, and labeling the body parts of
interest. It’s a laborious process that can
take graduate students or volunteers
hours upon hours, says Ben de Bivort,
a behavioral neuroscientist at Harvard
05 . 2019 | T H E S C IE N T IST 1 3
 NOTEBOOK
who was not part of the study. And such
herculean efforts produce small data-
sets compared to other methods that
researchers employ in their studies of
animal biology, such as genomics or
high-resolution neural recording, he
says. “So measurement of the behavior
was always kind of a bottleneck.”
Another option is to glue markers
onto an animal’s limbs and then use
computer software to track them from
video footage. “Imagine like the markers
you would put on Andy Serkis to make
Gollum in Lord of the Rings,” says Gor-
don Berman, a theoretical biophysicist
at Emory University who did a postdoc
with Shaevitz but was not involved in
the LEAP project. Unfortunately, ani-
mals are pretty good at grooming them
off, and “putting these markers on a fly
. . . is rather difficult,” he says.
We developed all this crazy
artificial intelligence just to
try to understand fly sex.
—Talmo Pereira, Princeton University
Pereira says that watching actors in
motion capture suits was in fact what
got him thinking about how to track the
flies. But as he dug into the literature,
he realized that scientists were already
starting to capture animal motion with-
out using markers. Aldarondo, mean-
while, was working on motion capture
algorithms in a computer science class,
and, after chatting about it with Pereira,
he decided to apply neural networks to
his lab’s footage of individual fruit flies
as a course project.
In the first attempt, Aldarondo and
another student in the course labeled
thousands of frames of video with points
denoting fly body part locations and
then used those frames to train the net-
work to recognize the body parts auto-
matically. After the course ended, he
and Pereira continued working on the
project, tweaking the algorithm to auto-
mate more of the process. At the end
of last year, they published a version of
14 T H E SC I EN TIST | the-scientist.com
the tool that needs far fewer frames—
around 100—to achieve up to 95 per-
cent accuracy in tracking 32 points on a
fly’s body. In their report, the researchers
used LEAP to track all six of a fly’s legs,
plus its wings, body, and head. They also
applied their tool to capture the limb
movements of a mouse (Nat Methods
16:117–25, 2019).
LEAP’s success comes from a com-
bination of human and artificial input.
After receiving a set of labeled video
frames, it uses them to learn how points
are placed according to each image’s fea-
tures, and then spits out labels for the
next set of frames, which a researcher
then reviews. The tool’s guesses may not
be great the first time, but correcting
the program helps it get smarter. After
a few rounds of back and forth between
LEAP and a human, the program has
learned enough to correctly identify the
parts—even over the course of less than
one day. De Bivort describes the process
as “using the algorithm to produce the
data to make a better algorithm.”
“It’s surprisingly easy. It’s obviated
a lot of my hard-won image processing
skills over the years,” says Berman, who
uses the tool in his own research on flies
and prairie voles. “What used to take
months and months of work now takes
a couple of weeks, if that.”
Another artificially intelligent
method for motion capture, DeepLab-
Cut, developed by a separate group of
Harvard researchers appeared around
the same time as LEAP, also for track-
ing mice and fruit flies. Each tool has
its advantages: LEAP requires less time
to train, but DeepLabCut uses a bigger
neural network and performs better
than LEAP on cluttered or lower-qual-
ity images, says Berman. But both have
the strength of being applicable beyond
the species they were first developed
for. So far, multiple research groups
have used LEAP to track the motion of
mice, rats, grasshoppers, spiders, ants,
fish, and more, says Pereira.
Both tools could have applications
in everything from behavioral ecology
to medical research, where they could
help study disorders such as autism
that are associated with stereotyped
movements, says de Bivort. They’ll also
help neuroscientists probe the connec-
tions between the brain and behavior,
he adds. “Maybe the biggest question
in neuroscience is: How does the brain
produce behavior? Because that’s what
the brain is for,” de Bivort says. “It’s not
exaggerating to say that these tools are
a big deal for our field now.”
—Carolyn Wilke
Listening for
Your Health
Speech is a window into our brains—and
not just when we’re healthy. When neuro-
logical issues arise, they often manifest in
the things we say and how we say them.
IBM computer scientist Guillermo
Cecchi came to appreciate just how
important language is in medicine as
a psychiatry postdoc at Weill Cornell
Medicine in the early 2000s. Despite
advances in brain imaging, “it’s still
[through] behavior, and fundamentally
through language, that we assess men-
tal health,” he says. “And we deal with it
through therapy. . . . Language is essen-
tial for that.”
In the digital age, hardware and
software are available for “natural lan-
guage processing”—a type of artificial
intelligence (AI) pioneered by IBM’s
Watson that extracts useful content
from written and spoken language. But
while companies such as Google and
Facebook use language processing to
evaluate our social media interactions,
emails, and browsing histories in order
to personalize the ads we see in our
news feeds, the tools have yet to be har-
nessed for medical applications. In the
clinic, “all that technology is completely
ignored,” says Cecchi. “We still judge the
language production of the patient on a
subjective basis.”
With recent advances in AI, that’s
starting to change, and Cecchi’s team at
IBM is one of several groups now devel-

oping machine learning algorithms to
analyze patient language. “I would say in
the last five years there’s been an explo-
sion of interest,” he says. The approaches
are all in the earliest stages of develop-
ment, with most models suffering from
small training and testing datasets. But
several studies have yielded promising
results across a range of psychiatric and
neurological conditions.
One area that Cecchi has explored
is the prediction of schizophrenia and
psychosis onset. Because in these con-
ditions “it’s your thought process that
is disordered,” as Cecchi explains, their
connection to language is intimate.
A few years ago, Cecchi and his col-
leagues developed a machine learning
algorithm to analyze two features of
speech known to be affected in psycho-
sis: syntactic complexity and semantic
coherence, a technical term for the flow
of meaning.
ANDRZEJ KRAUZE
In two small-scale validation stud-
ies, Cecchi’s team trained the algo-
rithm using transcripts of interviews
with patients, and showed that the
resulting model could predict, with
85–100 percent accuracy, if psychosis
onset was imminent (in the next two
years) in young, high-risk patients (npj
Schizophr, 1:15030, 2015; World Psy-
chiatry, 17:67–75, 2018).
The model is a long way from clinical
use, cautions Cecchi, noting that the stud-
ies included data from just a few dozen
subjects. “We need to reach sample sizes
[of ] several thousand to say we are abso-
lutely sure this is working.” But he sus-
pects that more work will support the use
of such AI-based approaches, not only for
helping psychiatrists diagnose psychosis,
but also for monitoring patients who suf-
fer from psychotic disorders.
And it’s not just psychosis, he empha-
sizes. “The major disorders affecting
our society—depression, PTSD, addic-
tion, and then neurological disorders,
Alzheimer’s disease, Parkinson’s disease
—all of them leave a mark in language.”
A few years ago, for example, his group
developed machine learning models that
predict Parkinson’s diagnoses and sever-
ity with about 75 percent accuracy based
on transcripts of patients describing their
typical day (Brain Lang, 162:19–28, 2016).
While the linguistic content of speech
can reveal a lot about how a person’s brain
is functioning, other aspects of spoken
language, such as voice, tone, and intona-
tion, could provide additional clues about
a person’s physical and mental health. “If
you have a cold, the sound of your voice
changes,” notes MIT computer scientist
James Glass, who has investigated AI
analyses of speech for detection of cogni-
tive impairment and depression.
Monitoring people’s health by listen-
ing to the sound of their voice is the focus
of researchers at Sonde Health, a startup
based in Boston, Massachusetts, that aims
to integrate voice-analysis technology into
consumer devices such as Google Home
and Amazon Echo. Company cofounder Jim
Harper says the team has already developed
machine learning models to predict more
than 15 conditions, including neurological,
respiratory, and muscular or cardiac dis-
orders, based on the acoustic properties of
short fragments of speech. The early mod-
els work “almost as well as existing measure-
ments,” Harper says, noting that the com-
pany is already in talks with the US Food
and Drug Administration about its model
for detecting depression and hopes to begin
a clinical study within the year.
While several groups are
investigating either linguis-
tic or acoustic elements of
spoken language, a com-
bination of the two often
yields the best results.
Examining qualities of speech, such
as the tone and expressiveness of a per-
son’s voice, can be particularly revealing
for identifying movement disorders such
as Parkinson’s disease, which can disrupt
the functioning of muscles involved in
speech. After being diagnosed, some Par-
kinson’s patients will recognize that one
of the first symptoms was a flat speaking
tone, for example.
Laureano Moro-Velázquez, a tele-
communications engineer at Johns
Hopkins University, and his colleagues
05 . 2019 | T H E S C IE N T IST 1 5
 NOTEBOOK

are using machine learning to analyze

phonemes—the discrete sounds that
compose speech—as a means of diagno-
sis. This February, the team published a
model, trained with recordings of sen-
tences recited by about 100 people with
Parkinson’s and 100 controls, that could
determine, with more than 80 percent
accuracy, whether or not someone had
the disease (Biomed Signal Process Con-
trol, 48:205–20, 2019).
While several groups are investi-
gating either linguistic or acoustic ele-
ments of spoken language, a combi-
nation of the two often yields the best
results. Cecchi’s group, for example,
recently used an analysis of recordings
of Parkinson’s patients’ speech—consid-
ering both acoustic and linguistic fea-
tures—to successfully identify who was
taking the drug levodopa (bioRxiv, doi:
10.1101/420422, 2018).
And Tuka Alhanai, a graduate stu-
dent in Glass’s lab at MIT, has devel-
oped a machine learning model to
extract clues about whether a person is
depressed from text and audio record-
ings of interviews. The model learned—
both from the words used to respond to
questions and from other features of
speech, such as its speed—to predict,
with 77 percent accuracy, depression in
142 patients whose data were held in a
public repository, according to results
Alhanai presented at the Interspeech
conference in September 2018.
For now, the use of speech analysis
is still in the proof-of-concept stage,
whatever aspects are analyzed. “I think
everything suffers from small data-
bases,” says Glass. “Do it on something
ten or a hundred times bigger, and I’ll
pay more attention.” But if validation of
the early-stage work proves as success-
ful as many in the field anticipate and
hope, he says, “I think it just opens up
new opportunities to complement exist-
ing techniques and maybe provide more
comprehensive [care].”
—Jef Akst
A Better
Beehive
When high school student Jade Greenberg
heard about what was happening to Amer-
ica’s bee populations, she decided to take
action. Last year, Greenberg, then a junior
at Pascack Hills High School in New Jer-
sey, learned from a local beekeeper about
a tiny reddish-brown mite that is posing
a serious threat to the honey bees (Apis
mellifera) used across the US for polli-
nating various crops. But what began as
a school project to build a hive that might
help boost bee health soon turned into a
collaboration with two tech companies to
use artificial intelligence (AI) and tracking
systems to tackle the problem.
The varroa mite (Varroa destructor)
was brought over to North America from
Southeast Asia decades ago and has been
decimating not only colonies tended by
beekeepers, but also feral colonies—those
that were started in the wild by bees orig-

YOU ARE NOT A ROBOT...

SO DON‘T ACT LIKE ONE
FREE YOURSELF FROM
ROUTINE PIPETTING

ASSIST PLUS Automating Multichannel Pipettes

Making hands-free serial dilutions, reagent additions and sample reformatting very
affordable for every lab. Compatible with all Integra‘s electronic pipettes from 4 to 16
channels for consistent results and unbeatable ergonomics.

VIAFLO - Electronic Pipettes VOYAGER - Adjustable Tip Spacing Pipettes www.integra-biosciences.com

 inally from hives kept by humans. The
pest reproduces within honey bee colo-
nies, latching onto the insects and feeding
off their fat body, a tissue similar in func-
tion to the mammalian liver. Over time,
the parasite weakens bees’ immune sys-
tems, making them more susceptible to
viruses and pesticides.
“Even if we solve the other problems
contributing to honey bee loss, like pesti-
cides and poor nutrition, colonies will still
be lost if varroa is not under control,” says
honey bee researcher Gloria DeGrandi-
Hoffman of the US Department of Agri-
culture’s Agricultural Research Service
(USDA-ARS) Carl Hayden Bee Research
Center in Tucson, Arizona.
What began as a school
project soon turned into a
collaboration with two tech
companies to use artificial
intelligence to tackle the
problem.
There are hints that the practice of
beekeeping itself may be contributing to
the problem. DeGrandi-Hoffman and her
colleagues, for example, recently found
ceptibility to mite infestations. “There
is research on environmental factors,
such as where the bees forage and how
they are affected by pesticides,” she
says. “But there is very little informa-
tion on whether or not the shape of the
commercial hive has an effect on [mite
infestation rates].”
Taking advantage of a school sci-
ence fair as an occasion for the proj-
ect, Greenberg set about designing her
own beehive, based on the so-called Sun
Hive. Invented in the 1980s by Günther
Mancke, a German sculptor who studied
natural beehives, Sun Hives “are hand-
made hives and not commercially via-
ble,” Greenberg says. “I decided to riff off
of Mancke’s design and make something
that could be both commercially viable
and also healthier for bees.”
Greenberg incorporated sensors to
monitor hive weight, temperature, and
humidity, along with video monitoring
technology, while still retaining the nec-
essary features for commercial beekeep-
ing, such as removable frames. After a few
months of honing the design on the com-
puter, Greenberg was considering building
a prototype of the 42-liter structure using
...TOBOR A TON ERA UOY
a 3-D printer when her father, a solutions
engineer at artificial intelligence company
Kinetica, made a suggestion.
Kinetica uses AI and other technol-
ogies to analyze extremely large data
sets on everything from financial mar-
kets to the movements of fleets of trucks
as they drive around the US. Jonathan
Greenberg had told colleagues about his
daughter’s project and, to his surprise,
found they were keen to get involved.
One of those colleagues was Jacci Cenci,
a solutions architect at NVIDIA, a tech
company focused on AI and graphics.
Cenci offered to mentor Greenberg on
machine learning technologies and pro-
vide her with real-world data to analyze
bee health.
Working with a beekeeper in Califor-
nia, the team placed sensors on a single
Langstroth hive containing a healthy bee
colony to monitor its temperature, humid-
ity, and weight, and installed a camera to
capture images. The researchers then cre-
ated an analytical tool using Kinetica tech-
nology to detect bees in the images and
BEES, DEBUGGED: Researchers trained
machine learning algorithms to distinguish
between a healthy honey bee and one covered in
parasitic mites.

that the current commercial beekeeping

practice of preventing swarming—when
ENO EKIL TCA T‘NOD OS
a queen bee and a group of worker bees
leave their original colony to form a new
one—may have exerted selection pressure
MORF FLESRUOY EERF
on the mites to find new ways to disperse
among bee colonies (Environ Entomol,
GNITTEPIP ENITUOR
46:737–46, 2017).
Greenberg was interested in the role of
another aspect of beekeeping: hive design.
Feral bees typically nest in tree cavities or
other structures, forming layers of wax
comb to fill the available area. By contrast,
the most widely used manmade beehive settepiP lennahcitluM gnitamotuA SULP TSISSA
in North America, the Langstroth hive, yrev gnittamrofer elpmas dna snoitidda tnegaer ,snoitulid laires eerf-sdnah gnikaM
is essentially a stacked, wooden file cabi- 61 ot 4 morf settepip cinortcele s‘argetnI lla htiw elbitapmoC .bal yreve rof elbadroffa
JONATHAN GREENBERG

net with removable frames on which bees .scimonogre elbataebnu dna stluser tnetsisnoc rof slennahc
build their combs, making it easy for bee-
keepers to collect honey and monitor the
colony’s health.
Greenberg wondered if the hive
design could be affecting the bees’ sus-

05 . 2019 | T H E S C IE N T IST 1 7
moc.secneicsoib-argetni.www settepiP gnicapS piT elbatsujdA - REGAYOV settepiP cinortcelE - OLFAIV
 NOTEBOOK
to analyze environmental factors such as
temperature, humidity, and weather con-
ditions that might be contributing to var-
roa mites’ ability to infect the colony. The
effort was an initial proof of concept, says
Jonathan Greenberg. “This was about fig-
uring out how the technology can work
to more effectively monitor parasites for
Jade’s hive experiments in the field.”
The group also came up with a better
way to monitor mite infection in honey
bee colonies. Usually, “the way beekeep-
ers assess whether there is a mite infes-
tation is invasive,” says Samuel Ramsey,
a research entomologist at the USDA
Bee Research Laboratory in Beltsville,
Maryland, who was not involved in the
project. “We take 300 bees, put them in
a jar, sprinkle powdered sugar on them,
shake the jar, and count how many mites
fall off of the bees.”
Instead, Greenberg’s team captured
images in the California hive every
10 seconds and then used NVIDIA’s
machine learning to detect the presence
of varroa mites on bees. To do this, the
team used individual frames from vid-
eos of the healthy beehive, and from a
different beehive where the varroa mite
had taken up residence, to train a com-
puter algorithm.
The data collection and analysis
tools that Cenci, Greenberg, and their
colleagues built are all still in the proto-
type stage. But Ramsey says that it has
been a step toward figuring out whether
Langstroth hives contribute to the mite
infestation problem—and if Green-
berg’s hive design could aid the bees in
staving off the mites with their natu-
ral defenses. “Designing a better honey
bee hive would be wonderful because
we are putting these bees in an unnat-
ural situation, the Langstroth hive,” he
says. “It’s an old technology that wasn’t
designed to address current issues like
mite infestation.”
The project won the engineering cat-
egory at the Nokia Bell Labs North Jer-
sey Regional Science Fair, and Greenberg
was a finalist at the Intel International
Science and Engineering Fair. She says
she now hopes to measure physical prop-
18 T H E SC I EN TIST | the-scientist.com
erties such as ventilation and heat dis-
tribution in her prototype hive, in the
Langstroth hive, and in wild beehives to
understand how hive design might help
or hinder mite infestations.
“I’m always excited when younger stu-
dents bring passion and a youthful, new
way to look into established systems,” says
Ramsey. “That is certainly a way to make
sure there is progress.” DeGrandi-Hoffman
agrees. “It’s great to see young students with
enthusiasm and confidence to tackle com-
plicated problems with creative solutions.”
For Greenberg, the biggest surprise
was how few researchers are studying
biological problems such as honey bee
health from an engineering perspective.
Her solution? To major in bioengineering
when she starts university this fall.
—Anna Azvolinsky
Into the Cell
For cell biologists, fluorescence micro-
scopy is an invaluable tool. Fusing dyes
to antibodies or inserting genes coding
for fluorescent proteins into the DNA of
living cells can help scientists pick out the
location of organelles, cytoskeletal ele-
ments, and other subcellular structures
from otherwise impenetrable microscopy
images. But this technique has its draw-
backs. There are limits to the number of
fluorescent tags that can be introduced
into a cell, and side effects such as photo-
toxicity—damage caused by repeated
exposure to light—can hinder research-
ers’ ability to conduct live cell imaging.
These issues were on biomedical engi-
neer Greg Johnson’s mind when he joined
the Allen Institute for Cell Science in
Seattle in 2016. Johnson, whose doctoral
work at Carnegie Mellon University had
focused on creating computational tools
to model cellular structures (see “Autopi-
lot Advocate” on page 48), was hired as
part of a group of researchers working
to build a 3-D model of a cell. Accord-
ing to Johnson, one of the key aims of
the project, dubbed the “Allen Integrated
Cell,” was to develop a tool to help visu-
alize changes in the spatial organization
of cells as they move from one state to
another—for example, from a pluripotent
stem cell to a differentiated heart cell.
“Because of technological limita-
tions, we can only see a few things in
the cells at once,” Johnson says. “So we
wanted to figure out ways that we could,
at the very least, predict the organization
of many more structures from the data
that we already have.”
Specifically, they wanted to develop a
method to identify a living cell’s compo-
nents in images taken using brightfield
microscopy. This technique is simpler and
cheaper than fluorescent microscopy, but
has a major disadvantage—it produces
images that appear only in shades of gray,
making a cell’s internal structures diffi-
cult to decipher. So the scientists decided
to create a computer algorithm that could
combine the benefits of both methods by
learning how to detect and tag cellular
structures the way fluorescent labels can,
but in brightfield images instead.
To do this, the team turned to deep
learning, an artificial intelligence (AI)
approach where algorithms learn to
identify patterns in datasets. They trained
convolutional neural networks—a deep
learning approach typically used to analyze
and classify images—to identify similari-
ties between brightfield and fluorescence
microscopy images of several cellular com-
ponents, including the nuclear envelope,
cell membrane, and mitochondria.
Seeing this work in a movie,
in a live cell, in 3D, was
really jaw dropping.
—Rick Horwitz, Allen Institute for Cell Science
After comparing many pairs of
images, the algorithm was able to pre-
dict the location of structures that flu-
orescent labels would have tagged, but
in 3-D brightfield images of live cells
(Nat Methods, 15:917–20, 2018). The
researchers found that the tool was very
accurate: its predicted labels were highly
correlated with the actual fluorescent
labels for many cellular components,
including nucleoli, nuclear envelopes,
 and microtubules. By applying the tech-
nique to a series of brightfield images
and merging the outputs, “we [were
able to get] this beautiful time-lapse of
all these cell parts moving around and
interacting with each other,” Johnson
tells The Scientist.
The cool thing about this
technology is that it can be
applied so broadly.
—Steve Finkbeiner, Gladstone Institutes
“Seeing this work in a movie, in a
live cell, in 3D, was really jaw dropping,”
says Rick Horwitz, executive director of
the Allen Institute for Cell Science, who
wasn’t directly involved in the project. “It
was really a bit like magic.”
Laura Boucheron, an electrical engi-
neer at New Mexico State University
who was not involved in the work but
coauthored an accompanying perspec-
tive article piece in the same issue of
Nature Methods, tells The Scientist that
the results were “shockingly impressive.”
She adds that the images generated by
the algorithm are “remarkably similar”
to those produced using fluorescence
microscopy. “The brightfield images are,
to a human, visually not particularly
interesting. They are not as clear—in
terms of the structures present—as flu-
orescent images,” Boucheron says. “But
based on the results, clearly there is infor-
mation [in the brightfield images] that
the network is learning to interpret.”
Johnson notes that a big upside to
his team’s method is that, contrary to the
common belief that deep learning algo-
rithms require thousands of images to
learn, this tool could be trained with just
dozens. “This is something that a gradu-
COURTESY OF THE ALLEN INSTITUTE
he adds. The researchers were also able
to use their deep learning algorithm to
identify the location of proteins that
make up myelin—the protective sheath
around neurons—in 2-D electron micro-
scope images.
Still, the method has some limita-
tions. According to Johnson, one key
issue is that the technique does not
work on all cellular structures, because
some simply do not appear in images
taken with certain forms of micros-
copy. In their recent study, for example,
the algorithm had difficulty identifying
a few structures in brightfield images,
including Golgi apparatuses and desmo-
somes, junctions that hold cells together.
Another limitation is that, while the tool
requires a relatively small training set,
a model trained on images from one
microscope might not work on images
gathered from another.
The team is now investigating some
potential applications of the technique.
Horwitz suggests that, in addition to
being able to make imaging studies faster
and cheaper, the tool could eventually be
applied in pathology to help identify sick
cells or to rapidly identify how cellular
structures change in diseased states.
Another group, which included Steve
Finkbeiner, a neuroscientist at the Glad-
stone Institutes and the University of Cal-
ifornia, San Francisco, and his colleagues
at Google, developed a similar AI-based
cell-labeling technique last year (Cell,
173:P792–803.e19). “I think the cool
thing about this technology is that it can
be applied so broadly, and I don’t think
we have a great feel yet for what the lim-
its are,” Finkbeiner says.
Boucheron notes that she is currently
investigating applications of these AI-
based approaches to image analysis in
both biology and astronomy—another
field where researchers rely on a vari-
ety of instruments to capture and ana-
lyze natural phenomena. “I work with
astronomy data quite a bit, and partic-
ularly with solar images,” she explains.
“I’ve been looking for several years for
ways to kind of translate between some
of the different [instruments] that are
used to image the Sun.”
Techniques that apply deep learning
to image analysis could be useful wher-
ever a microscope or telescope is used,
Horwitz says. This latest study is “just the
tip of the iceberg.”
—Diana Kwon

ate student can gather in an afternoon,”

NEW PERSPECTIVE: Researchers used an algo-

rithm to predict the location of structures in a
human cell from images taken with brightfield
microscopy. Cellular components include the
nuclear envelope and DNA (gray), mitochondria
(purple), actin (yellow), endoplasmic reticulum
(blue), and microtubules (tan).

05 . 2019 | T H E S C IE N T IST 1 9
The IsoLight:
Illuminating
The Small Things
That Make a Big
Difference
IsoPlexis, with industry leaders, has uniquely revealed differences
between Non-Responders and Responders in a variety of
immunotherapy contexts. Follow our data to see how the award winning
IsoLight is making a difference throughout immuno-oncology at
www.IsoPlexis.com/CellT-P
 Off-target Effects in CRISPR-Cas9 Genome Editing:
COMINGSOON Securing Specificity
A cause for concern regarding the popular CRISPR-Cas9 genome editing technology is the potential occurrence of off-target effects. RNA-guided
Cas9 may cleave DNA sequences that are not exact complements of the guide strand when either strand harbors bulges due to insertions or
deletions. Undoubtedly, a better understanding of the specificity of CRISPR-Cas9 editing can help develop strategies to minimize off-target cleavage.
To take a closer look at the specificity and possible causes of off-target effects in the CRISPR-Cas9 system, The Scientist is bringing together
researchers who will summarize their work on off-target effects limiting the applications of Cas9-mediated genome modification and discuss
strategies for predicting and preventing such off-target effects. Attendees will have the unique opportunity to ask experts about their experience with
off-target effects of the CRISPR-Cas9 system.

LAURYL NUTTER, PhD TUESDAY, MAY 7

Associate Director, Model Production and
Cryopreservation & Recovery
The Centre for Pharmacogenomics
The Hospital for Sick Children
2:30 – 4:00PM EASTERN TIME
REGISTER NOW!
www.the-scientist.com/crisprcas9offtarget
The webinar video will also be available at this link.

DAVID J. SEGAL, PhD TOPICS TO BE COVERED:

Professor, Dept. of Biochemistry • Whole genome sequencing to assess Cas9 off-target
and Molecular Medicine effects in genetically engineered mice
UC Davis School of Medicine • Will elimination of off-target effects be good enough?
Professor, Dept. of Pharmacology
UC Davis MIND Institute

WEBINAR SPONSORED BY:

Systems Immunology:
COMINGSOON Understanding Responses to Vaccination and Infection
Systems immunology is a new and rapidly developing field of research providing quantitative molecular profiles of patients’ immune responses,
made possible by recent advances in multiple technological platforms. The human-centered approach has invigorated vaccine research and
shown promise to enhance our knowledge of the systemic immune response to vaccination and infection that was not possible with classical,
animal-model based approaches. Techniques that make use of peripheral blood to survey systemic immune responses include next-generation
sequencing coupled to bioinformatic analysis, gene and protein microarrays, metabolomics, multiparameter flow cytometry, mass cytometry,
and multiplex cytokine and chemokine assays. The Scientist  is bringing together a panel of experts to present their research in systems
immunology and provide insight into this exciting new field.

KATHRYN MILLER-JENSEN, PhD TUESDAY, MAY 21

Associate Professor, Departments of
Biomedical Engineering and Molecular,
Cellular, and Developmental Biology
Yale University
2:30 – 4:00PM EASTERN TIME
REGISTER NOW!
www.the-scientist.com/systemsimmunology
The webinar video will also be available at this link.

TOPICS TO BE COVERED:
EDANA CASSOL, PhD • New insights into immune responses to infection
Assistant Professor through data-driven analysis of multiplexed
Department of Health Sciences single-cell data
Carleton University
• Using systems immunology to understand innate
immune dysfunction in HIV infection

WEBINAR SPONSORED BY:

 MODUS OPERANDI
Microbiology Meets Machine Learning
Artificially intelligent software augments the study
of host-pathogen interactions.
BY RUTH WILLIAMS
C
omputational systems called neural
networks—based on the learning
processes of biological brains—
enable a form of machine learning that has
the potential to help researchers interpret
biological and medical images. Scientists who
study how pathogens interact with host cells
are now beginning to harness such technology.
“Most people in the [pathogen-host
interactions] field were just manually
counting—literally sitting there and assessing
how many [parasites] per cell, how many
in one of these vacuoles,” and so on, says
parasitologist Eva Frickel of the Francis Crick
Institute in London. “My students were losing
hours and hours, days and weeks counting
these events.”
Neural networks are used for all manner
of image-processing tasks, such as face
recognition, diagnostics, and self-driving
cars, so Frickel thought such a system might
offer a solution to her team’s problem. She
teamed up with computational biologist Artur
Yakimovich of the Medical Research Council’s
Laboratory for Molecular Cell Biology to make
a human-like host-pathogen analyzer.
Yakimovich, Frickel, and colleagues
started with an existing open-source neural
network–based analytics platform called
KNIME (for Konstanz Information Miner)
and tweaked its algorithms to process
images of host cells and their pathogens.
AT A GLANCE
IMAGE ANALYSIS TYPES OF IMAGES PROCESSED
SOFTWARE AND HARDWARE REQUIRED
Harmony from Perkin Images captured on an Operetta CLS or
Elmer Opera Phenix  high-content screening
system.
HRMAn Images collected on any fluorescent
microscope or high-content screening
system.
22 T H E SC I EN TIST | the-scientist.com
Toxoplasma
gondii
Pathogen-derived
proteins
DETECTING INVADERS: A neural network–based machine learning program called HRMAn spots several features of
cells to determine whether they are infected with a pathogen (in this case, T. gondii), such as the number of vacuoles,
which changes with infection, and the presence of host defense proteins and of the pathogen and its proteins.
The system required training with thousands
of example images, and once it was up and
running, the team gave the system a name:
HRMAn, pronounced Herman, for Host
Response to Microbe Analysis.
They’ve used the software to analyze
Toxoplasma gondii and Salmonella enterica
infections in a variety of human cell lines. Other
high-throughput image-analysis software may
be capable of identifying which cells contain
pathogens, but HRMAn comes into its own,
says Frickel, in its ability to identify multiple
visual characteristics of pathogens and host
cells at once and detect patterns in the images.
USES AI
COMPONENT?
Analyzing a large variety of cell No
phenotypes, including host-pathogen
interactions
Currently trained to detect Toxoplasma Yes
gondii and Salmonella enterica interacting
with human cells, but could be adapted
for a variety of host-pathogen
interactions
Host defense
proteins
Vacuoles
Indeed, the team demonstrated that
HRMAn could simultaneously recognize
pathogen killing, replication, and a variety
of cellular defense processes—just as a
trained scientist might, but the computer
had far higher throughput and greater
statistical strength, and did not require
tea breaks and sleep.
“It’s a tool that goes beyond our
ability as humans to process and to
interpret image data,” says parasitologist
Adrian Hehl of the University of Zurich
who was not involved in the study (eLife,
8:e40560, 2019). g
COST
Comes free with the Operetta CLS  and Opera Phenix 
high-content screening systems, which themselves
© GEORGE RETSECK
cost several hundred thousand dollars each.
Free, open source
 CRITIC AT LARGE

AI Versus Animal Testing

Machine learning could be the key to reducing the use of animals in experiments.

BY THOMAS HARTUNG

T
here are more than 100,000 chemi-
cals in consumer products. For the vast
majority, there is very little information
about their toxicity. Traditionally, researchers
will test chemicals of interest in animals. As
an extreme example, a pesticide undergoes
about 30 animal tests, costing about $20 mil-
lion and consuming more than 10,000 mice,
rats, rabbits, and dogs over five years. About
20 kilograms of the chemical are needed for
this testing; obtaining such a volume can be
quite a challenge for a substance not yet on
the market. Other substances receive less
scrutiny, but even products with lower regu-
latory standards, such as industrial chemicals,
can require $5 million worth of animal test-
ing before entering the marketplace.
Our group, the Center for Alternatives
to Animal Testing (CAAT) at Johns Hopkins
University, sought a better way. As so many
biologists are doing these days, we turned to
intelligent computer programs for help. We
showed that artificial intelligence (AI) could
mine existing data on chemical toxicity and
generate new information. In 2016, we com-
piled a database of 800,000 toxicological
studies on more than 10,000 chemicals reg-
istered under the European REACH legisla-
tion for industrial chemicals, and used it to
feed an advanced predictive algorithm that
enabled us to predict the toxicity of any chem-
ical without setting foot in the animal lab.
The software takes advantage of the power
of big data and transfer learning, a machine
learning method that applies information
from one task or set of items to another. Simi-
MODIFIED FROM ©ISTOCK.COM, JXFZSY
In collaboration with Underwriters
Laboratories (UL), a global safety consult-
ing and certification company, the data-
base has expanded to more than 10 million
chemical structures, more than 300,000
of which are annotated with biological and
chemicophysical data and some 50,000
of those also include animal data. Using
an Amazon cloud server, it took two days
to analyze the similarities and differences
among the 10 million chemicals to place
them on a map. Applying this to 190,000
classified chemicals based on animal tests,
the computer correctly predicted the out-
come of toxicity studies 87 percent of the
time—exceeding the 70 percent probabil-
ity of animal tests to find a toxic substance
again in a repeat animal test. Traditional
testing for the nine different hazard clas-
sifications currently analyzed by the model
consumes 57 percent of all animals in safety tion will only work for toxic properties that
testing in Europe, or about 600,000 ani- are directly dependent on chemical structure.
mals per year. These include assays such as Where the underlying biology is complex—
skin and eye irritation/corrosion, skin sen- such as links between a substance and can-
sitization, acute toxicity, and mutagenicity. cer, or insults to a developing embryo—more
Some of the chemicals in the database information is required. Here, large stan-
have already been tested excessively, caus- dardized databases are rare, with the notable
ing unnecessary loss of animal life. For exception of some US agencies’ robot-testing
instance, two chemicals were tested more programs. These programs have subjected
than 90 times in rabbit eyes; 69 substances about 2,000 chemicals to more than 700
were tested more than 45 times in the same cellular assays, and nearly 10,000 chemicals
eye test. Our AI approach could drastically to about 60 assays. With computing power
reduce the need for these common animal now more accessible and far more affordable,
toxicity tests, and save quite a bit of money AI is the solution to many real-world prob-
in the process. Running the supercomputer lems, such as limiting the need for animals in
to create the map cost about $5,000, and safety testing and biological research. g

lar chemicals have similar properties. Based on
that principle, the software builds a map of the
chemical universe. Similar chemicals are put
close to each other, dissimilar ones more dis-
tant. Then, the model can place new chemicals
on the map, assess what is known about their
neighbors, and from that information surmise
their potentially harmful health and environ-
mental effects. The more data are fed into the
model, the more powerful it becomes.
computer costs per prediction are now
negligible. Most importantly, beyond the
regulation of chemicals being brought to
market, chemists can identify hazards before
the compounds are ever synthesized. And
when a toxic chemical needs to be replaced,
the AI approach can help ensure that less-
toxic substances are chosen.
The power of this approach depends on
the availability of data, and its implementa-
Thomas Hartung is chair for Evidence-
based Toxicology at the Johns Hopkins
Bloomberg School of Public Health in Bal-
timore, Maryland, where he is a professor
of molecular microbiology and immunol-
ogy. Hartung is also a professor of phar-
macology and toxicology at the University
of Konstanz in Germany. He directs the
Centers for Alternatives to Animal Testing
(CAAT) of both universities.

05 . 201 9 | T H E S C IE N T IST 23
 Run Program:
Cancer
Using input from images to -omes, artificial intelligence can find patterns in tumors
and generate prognoses where human clinicians see only a jumble of data.

BY AMBER DANCE

I
t’s the question on every cancer patient’s mind: How long
have I got? Genomicist Michael Snyder wishes he had
answers.
For now, all physicians can do is lump patients with simi-
lar cancers into large groups and guess that they’ ll have the
same drug responses or prognoses as others in the group.
But their methods of assigning people to these groups are
and Personalized Medicine at Stanford University. In 2013, he
and then–graduate student Kun-Hsing Yu wondered if artificial
intelligence could provide more-accurate predictions.
Yu fed histology images into a machine learning algo-
rithm, along with pathologist-determined diagnoses, train-
ing it to distinguish lung cancer from normal tissue, and two
different types of lung cancer from each other. Then he fed
© MOLLY MENDOZA

coarse and imperfect, and often based on data collected by in survival data for those slides, letting the system learn how
human eyeballs. that information correlated with the images. Finally, he added
“When pathologists read images, only sixty percent of the time in new slides that the model hadn’t seen before, and asked the
do they agree,” says Snyder, director of the Center for Genomics all-important longevity question.

24 T H E SC I EN TIST | the-scientist.com
 The computer could predict who would live for shorter or
longer than average survival times for those particular cancers—
something pathologists struggle to do.1 “It worked surprisingly
well,” says Yu, now an instructor at Harvard Medical School.
But Snyder and Yu thought they could do more. Snyder’s lab
works on -omics, too, so they decided to offer the computer not
just the slides, but also tumor transcriptomes. With these data
combined, the model predicted patient survival even better than
images or transcriptomes alone, with more than 80 percent accu-
racy.2 Today, pathologists normally make survival predictions based
on visual evaluations of tissue micrographs, from which they assess
a tumor’s stage—its size and extent—and grade, the likelihood that
it will grow and spread further. But pathologists don’t always agree,
and tumor grade doesn’t always predict survival accurately.
Snyder and Yu aren’t the only researchers who are recognizing
the power of AI to analyze cancer-related datasets of images, of
-omes, and most recently, of both combined. Although these tools
have a long way to go before they reach the clinic, AI approaches
stand to yield a precise diagnosis quickly, predict which treatments
will work best for which patients, and even forecast survival.
For now, some of those applications are still “science fiction,” says
Andrea Sottoriva, a computational biologist at London’s Institute of
Cancer Research who’s working on AI to predict cancer evolution and
choose the right drugs to treat a given tumor. “We aim to change that.”
INPUT: Images, OUTPUT: Diagnosis
Finding and treating cancer before it progresses too far can be key
to increased survival. When it comes to cervical cancer, for exam-
ple, early detection leads to five-year survival rates of more than
90 percent. Doctors can fry, freeze, or excise precancerous cells in
the top four millimeters of the cervix’s transformation zone, a ring
of tissue surrounding the cervix where cancer most often arises.
Once the cancer metastasizes, however, survival rates drop to 56
percent or lower over five years.
When pathologists read images,
only sixty percent of the time do
they agree.
—Michael Snyder, Stanford University
Early treatment is commonplace in developed nations, where
women get regular Pap smears to check for abnormal cervical cells
and tests for the human papillomavirus that causes the cancer. But
in the developing world, such screenings are rare. There is a cheaper
test—health care workers coat a woman’s cervix in acetic acid, look-
ing for telltale white areas that could indicate cancer—but “this tech-
nique is so inaccurate,” says medical epidemiologist Mark Schiffman
of the National Cancer Institute. As a result, some healthy women
undergo treatment while others might have their precancerous cells
missed, leading to cancer that requires more-radical treatments,
such as chemotherapy, radiation, or hysterectomy.
Schiffman and other groups have been trying to find a way to
make acetic acid screening more accurate—for example, by imag-
ing with spectra other than white light. Schiffman’s team had accu-
mulated thousands of cervix pictures from diverse sources in the
US and Costa Rica, including photos taken by health-care profes-
sionals with a magnifying camera called a colposcope or with a cell-
phone. But he was about to give up. “We couldn’t make it be really
as sensitive or as accurate or as reproducible as the other [tests].”
Then, near the end of 2017, a nonprofit associated with the
Bill & Melinda Gates Foundation called Global Good reached out.
The organization wanted to try machine learning on Schiffman’s
image collection, to see if a computer could provide diagnoses
when physicians could not.
So Schiffman teamed up with Global Good and other collabo-
rators to use a particular kind of machine learning, called a convo-
lutional neural network, to analyze the cervix images. The goal of
the algorithm was to identify features in the images—for example,
how similar or dissimilar side-by-side pixels tend to be—that help
it get the right diagnosis. At the start, its accuracy was no better
than chance. As it analyzed more and more images, it weighed
those features to help it find the answer. “It’s a process of getting
hotter, hotter, colder, colder, oh yeah, hotter, hotter . . . until it gets
as close as it possibly can,” explains Schiffman.
The team started with cervix images collected over seven
© MOLLY MENDOZA
years in Costa Rica from more than 9,000 women. Schiffman had
also amassed data from more-accurate screening tests in these
women, along with 18 years’ worth of follow up information on
precancer or cancer diagnoses. The researchers used 70 percent
of the complete dataset to train the model, then tested its perfor-
 HOW AI TAKES ON CANCER
Scientists have been using two main forms of clinical data to predict cancer outcomes: images (either photographs, as in
the case of skin cancer, or pathology slides) and -omes of various sorts. Applying ever-more sophisticated machine learning
approaches to these datasets can yield accurate diagnoses and prognoses, and even infer how tumors evolve (yellow arrows).
Now, scientists are finding that images can predict -omics (blue arrows). Combining the two data sources gives researchers
even better predictions of how long a cancer patient will live (thick purple arrows). The ultimate goal of these algorithms, cur-
rently under development in basic biology labs, is to help doctors select treatments and forecast survival.

DIAGNOSIS/TUMOR GRADE
DIRECT IMAGING
HISTOPATHOLOGY
SLIDES

PROGNOSIS/SURVIVAL

MACHINE
LEARNING
THEI SCIENTIST STAFF; © ISTOCK.COM, DZIKA_MROWKA, FROM2015

TUMOR EVOLUTION

GENOMES
TRANSCRIPTOMES
PROTEOMES BEST DRUG TREATMENTS
 mance on the images only from the remaining 30 percent. Schiff-
man couldn’t believe the results: machine learning distinguished
between healthy tissue, precancer, and cancer with 91 percent on
a standard measure of the accuracy of machine learning predic-
tions. A human visual inspection, in contrast, only scored 69 per-
cent.3 “I’ve never seen anything this accurate,” says Schiffman. He
was sure there was some mistake.
The group checked its work and asked collaborators at the
National Library of Medicine to independently verify the tech-
nique. There was no error: the machine really was that good
at identifying precancer and cancer. Armed with this new tool,
Schiffman hopes to develop a low-cost screening test for cervical
cancer coupling a cell phone–type camera with machine-based
image analysis. First, he wants to train his algorithm on tens of
thousands of cell-phone cervix images from all over the world.
He’s not the only one eyeing smartphones for cancer diagno-
sis. Skin lesions—which might be cancerous or benign—are right
on the surface, and anybody can snap a shot. Researchers at Stan-
ford University built a database of almost 130,000 photographs
of skin lesions and used it to train a convolutional neural network
to distinguish between benign bumps and three different kinds
of malignant lesions, with at least 91 percent accuracy. The algo-
rithm outperformed the majority of 21 dermatologists asked to
assess the same pictures.4
I’ve never seen anything this
accurate.
—Mark Schiffman, National Cancer Institute
A major challenge to creating predictive models of cancer is
acquiring enough high-quality data. When the Stanford team com-
piled images of skin cancer from Stanford Medical School and from
the internet, the angles, zooms, and lighting all varied. The research-
ers had to translate labels from a variety of languages, then work with
dermatologists to correctly classify the lesions into more than 2,000
diseases categories.
And, of course, most cancers require more than a smart-
phone camera to see what’s going on. Observing individual cells
in tumors requires microscopy. Scientists would also like to incor-
porate as much information as possible about a person’s clinical
treatments and responses, plus molecular data such as genomes,
but that too can be hard to come by, says Yu. “Rarely will we find
a patient with all the data we want.”
INPUT: Images + -Omes, OUTPUT: Survival
As Snyder and Yu have found, -omics data, when available, can
provide information about the molecular pathways involved in
a given cancer that may help identify cancer type, survival, or
likely response to treatment. In their initial, image-based stud-
ies, the researchers had 2,186 lung tissue slides, disease classi-
fications from human pathologists, and patient survival times.
28 T H E SC I EN TIST | the-scientist.com
The researchers used a computer algorithm to extract from those
images nearly 10,000 features, such as cell shape or size, which
they used to train several machine learning algorithms.
One approach that worked well is called Random Forest. It
generates hundreds of possible decision trees; then those “trees”
vote on the answer, and the majority rules. This algorithm was
more than 75 percent accurate in distinguishing between healthy
tissue and the two cancer types, and it could predict who fell into
the high- or low-survival group with greater accuracy than mod-
els based solely on the cancer’s stage.1 “This is something that goes
beyond the current pathological diagnosis,” says Yu.
In their follow-up study, the researchers ran their trained image
analysis algorithm on histopathology slides from 538 people with
lung cancer, then added transcriptomes and proteomes from those
same patients, and asked the “random forest” to vote on the grade
of their cancers. The expression levels of 15 genes predicted can-
cer grade with 80 percent accuracy. These genes turned out to be
involved in processes such as DNA replication, cell cycle regula-
tion, and p53 signaling—all known to play roles in cancer biology.
The team also identified 15 proteins—not the ones encoded by the
15 genes—involved in cell development and cancer signaling that
predicted grade with 81 percent accuracy. While the researchers
didn’t compare this to human performance, one study of patholo-
gists found 79 percent agreement on lung adenocarcinoma grad-
ing5—suggesting the machine and humans were equally accurate.
But the machine was going further, apparently homing in on the
specific gene-expression factors driving a cancer’s progression.
Finally, the researchers asked the computer to predict survival
based on gene expression, cancer grade, and patient age. With all
those data, the model achieved greater than 80 percent accuracy,
correctly sorting cases into long-term and short-term survivors
better than human pathologists, transcriptomes, or images alone.2
Inspired by Snyder and Yu’s work, Aristotelis Tsirigos and col-
leagues at New York University School of Medicine also sought
to link images to genetics in lung cancer, using 1,634 slides of
healthy or cancerous lung tissue. Based on images alone, their
convolutional neural network was able to distinguish adenocarci-
noma from squamous cell carcinoma with about 97 percent accu-
racy. Then, the team fed the algorithm data on the 10 most com-
monly mutated genes in lung adenocarcinoma, and the computer
learned to predict the presence of six of those mutations from the
pathology slides with accuracy ranging from 73 to 86 percent.6
“It works quite well,” commented Sottoriva, who was not
involved in the work. “As a start, it’s quite exciting.”
Of course, doctors and scientists don’t need to identify muta-
tions via imaging; other tests are more straightforward and more
accurate, with genetic sequencing providing a nearly perfect readout
of the cancer’s genome. This study, explains Tsirigos, serves to dem-
onstrate that genetics and image features are related in predictable
ways. Now, he’s working to combine histopathology and molecular
information to predict patient outcomes, as Yu and Snyder’s group
did. These kinds of methods should work for any cancer type, says
Tsirigos, as long as researchers have the right data to input.
 INPUT: -Omes, OUTPUT: Tumor Evolution
-Omics data are also useful on their own, even without images.
For example, Sottoriva and colleagues are using genomics to
understand tumor evolution. One tumor is typically made up
of multiple cell lineages all derived from the same original can-
cer cell. To effectively treat cancer, it’s important to understand
this heterogeneity and the manner in which a tumor evolved. If
a treatment works on only a portion of a tumor, the cancer will
come back. “It’s a real matter of life and death,” says Guido San-
guinetti, a computer scientist at the University of Edinburgh and
a collaborator on the tumor evolution studies.
By sampling multiple parts of an individual tumor, research-
ers can infer what evolutionary paths the cancer took; it’s akin
to sampling modern human genomes to trace various popula-
tions back to ancestral groups. (See “Tracking the Evolutionary
History of a Tumor,” The Scientist, April 2017). Tumors from dif-
ferent patients, even with the same kind of cancer, tend to have
wildly different evolutionary trees. Sanguinetti, Sottoriva, and
colleagues think that if they can find common pathways that can-
cer tends to follow, oncologists could use that information to cat-
egorize people who are likely to have similar disease progression,
or to respond similarly to drugs. that are clinically relevant. Moreover, by selectively eliminat-
To find those common evolutionary trees, the researchers ing certain pieces of data from the model’s input and seeing if
used a form of machine learning called transfer learning. The its accuracy drops, bioinformaticians are starting to figure out
algorithm looks at all the trees from patients’ genomes simul- what features the computers are using to distinguish those pat-
taneously, sharing information between them to find a solu- terns, says Tsirigos.
tion compatible with the whole group, explains Sanguinetti. Current AI applications for cancer research are just the begin-
They called their tool REVOLVER, for Repeated Evolution in ning. Future algorithms may incorporate not only -omes and
cancer. As a first test, they invented fictional tumor evolution- images, but other data about treatment outcomes, progression,
ary trees. When they fed REVOLVER genomics data based on and anything else scientists can get their hands on.
those made-up trees, it did spit out a phylogeny that matched “At the end of the day,” says Snyder, “when dealing with com-
the invented trees. plicated diseases like cancer, you want every bit of information.”
To validate the tool in a well-known form of cancer evolution,
the researchers turned to the transition to malignancy in colorec- Amber Dance is a freelance science journalist living in the Los
tal cancer. This happens as a benign adenoma accumulates muta- Angeles area.
tions in known driver genes: for example, in APC, then KRAS,
then PIK3CA. The researchers fed REVOLVER a set of genomes
from nine real benign adenomas and 10 malignant carcinomas.
Sure enough, the model drew phylogenetic trees that matched the References
1. K.H. Yu et al., “Predicting non-small cell lung cancer prognosis by fully
adenoma-to-carcinoma transition.
automated microscopic pathology image features,” Nat Commun, 7:12474,
The group then analyzed tumor samples for which the evolu- 2016.
tion was less well understood. In genomes from 99 people with 2. K.H. Yu et al., “Association of omics features with histopathology patterns in
non-small-cell lung cancer, REVOLVER identified 10 potential lung adenocarcinoma,” Cell Syst, 5:620–27.e3, 2017.
clusters of patients, based on the sequence of mutations tumors 3. L. Hu et al., “An observational study of deep learning and automated
evaluation of cervical images for cancer screening,” J Natl Cancer Inst,
accumulated. People in some of those clusters lived for less than
doi:10.1093/jnci/djy225, 2019.
150 days, while those placed in other clusters survived much 4. A. Esteva et al., “Dermatologist-level classification of skin cancer with deep
longer, suggesting the categories have prognostic value. Simi- neural networks,” Nature, 542:115–18, 2017.
larly, REVOLVER found six clusters among 50 breast cancer 5. Y. Nakazato et al., “Nuclear grading of primary pulmonary adenocarcinomas,”
© MOLLY MENDOZA

tumors with varying levels of survival between clusters.7 “We
didn’t expect to find groups, really,” says Sottoriva. “These results
tell us that evolution in cancer can be quite predictable.”
Medicine runs on those kinds of predictable patterns, says
Sottoriva. And AI is a powerful tool to help identify patterns
Cancer, 116:2011–19, 2010.
6. N. Coudray et al., “Classification and mutation prediction from non-small cell
lung cancer histopathology images using deep learning,” Nat Med, 24:1559–67,
2018.
7. G. Caravagna et al., “Detecting repeated cancer evolution from multi-region
tumor sequencing data,” Nat Methods, 15:707–14, 2018.

05 . 2019 | T H E S C IE N T IST 2 9
A Silicon
Brain
Biology-inspired computer chips may help scientists
UNIVERSITY

better simulate neurological function.

HEIDELBERG
LINE

BY SANDEEP RAVINDRAN
CREDIT
©
 SMART CHIP: A neuromorphic chip designed by the
Heidelberg group of physicist Karlheinz Meier. The chip
features 384 artificial neurons connected by 100,000
synapses, and operates approximately 100,000 times
CREDIT LINE

faster than the speed at which the brain computes.

 I It’s one of the most ambitious technological
n 2012, computer scientist Dharmendra
Modha used a powerful supercomputer to
simulate the activity of more than 500 bil-
lion neurons—more, even, than the 85 bil-
lion or so neurons in the human brain. It was
the culmination of almost a decade of work,
as Modha progressed from simulating the
brains of rodents and cats to something on
the scale of humans.
The simulation consumed enormous
computational resources—1.5 million proces-
sors and 1.5 petabytes (1.5 million gigabytes)
of memory—and was still agonizingly slow,
1,500 times slower than the brain computes.
Modha estimates that to run it in biological
real time would have required 12 gigawatts
of energy, about six times the maximum out-
put capacity of the Hoover Dam. “And yet, it
was just a cartoon of what the brain does,”
says Modha, chief scientist for brain-inspired
computing at IBM Almaden Research Center
in northern California. The simulation came
nowhere close to replicating the functional-
ity of the human brain, which uses about the
same amount of power as a 20-watt lightbulb.
Since the early 2000s, improved hard-
ware and advances in experimental and theo-
retical neuroscience have enabled researchers
to create ever larger and more-detailed mod-
els of the brain. But the more complex these
simulations get, the more they run into the
limitations of conventional computer hard-
ware, as illustrated by Modha’s power-hun-
gry model.
Modha’s human brain simulations were
run on Lawrence Livermore National Lab’s
Blue Gene/Q Sequoia supercomputer, an
immensely powerful conglomeration of tra-
ditional computer hardware: it’s powered by
a whole lot of conventional computer chips,
fingernail-size wafers of silicon containing
millions of transistors. The rules that gov-
ern the structure and function of traditional
computer chips are quite different from that
of our brains.
But the fact that computers “think” very
differently than our brains do actually gives
them an advantage when it comes to tasks
like number crunching, while making them
decidedly primitive in other areas, such as
understanding human speech or learning
from experience. If scientists want to simu-
late a brain that can match human intelli-
32 T H E SC I EN TIST | the-scientist.com
puzzles that we can take on—reverse engi-
neering the brain.
gence, let alone eclipse it, they may have to
start with better building blocks—computer
chips inspired by our brains.
So-called neuromorphic chips replicate
the architecture of the brain—that is, they
talk to each other using “neuronal spikes”
akin to a neuron’s action potential. This
spiking behavior allows the chips to con-
sume very little power and remain power-
efficient even when tiled together into very
large-scale systems.
“The biggest advantage in my mind is
scalability,” says Chris Eliasmith, a theoreti-
cal neuroscientist at the University of Water-
loo in Ontario. In his book How to Build a
Brain, Eliasmith describes a large-scale
model of the functioning brain that he cre-
ated and named Spaun.1 When Eliasmith
ran Spaun’s initial version, with 2.5 million
“neurons,” it ran 20 times slower than bio-
logical neurons even when the model was
run on the best conventional chips. “Every
time we add a couple of million neurons, it
gets that much slower,” he says. When Elia-
smith ran some of his simulations on digital
neuromorphic hardware, he found that they
were not only much faster but also about 50
times more power-efficient. Even better, the
neuromorphic platform became more effi-
cient as Eliasmith simulated more neurons.
That’s one of the ways in which neuromor-
phic chips aim to replicate nature, where
brains seem to increase in power and effi-
ciency as they scale up from, say, 300 neu-
rons in a worm brain to the 85 billion or so
of the human brain.
Neuromorphic chips’ ability to perform
complex computational tasks while consum-
ing very little power has caught the atten-
tion of the tech industry. The potential com-
mercial applications of neuromorphic chips
include power-efficient supercomputers,
low-power sensors, and self-learning robots.
But biologists have a different application in
mind: building a fully functioning replica of
the human brain.
—Mike Davies, Intel
Many of today’s neuromorphic systems,
from chips developed by IBM and Intel to
two chips created as part of the European
Union’s Human Brain Project, are also
available to researchers, who can remotely
access them to run their simulations.
Researchers are using these chips to create
detailed models of individual neurons and
synapses, and to decipher how units come
together to create larger brain subsystems.
The chips allow neuroscientists to test the-
ories of how vision, hearing, and olfaction
work on actual hardware, rather than just
in software. The latest neuromorphic sys-
tems are also enabling researchers to begin
the far more challenging task of replicating
how humans think and learn.
It’s still early days, and truly unlocking the
potential of neuromorphic chips will take the
combined efforts of theoretical, experimen-
tal, and computational neuroscientists, as
well as computer scientists and engineers.
But the end goal is a grand one—nothing less
than figuring out how the components of the
brain work together to create thoughts, feel-
ings, and even consciousness.
“It’s one of the most ambitious techno-
logical puzzles that we can take on—reverse
engineering the brain,” says computer engi-
neer Mike Davies, director of Intel’s neuro-
morphic computing lab.
It’s all about architecture
Caltech scientist Carver Mead coined the
term “neuromorphic” in the 1980s, after
noticing that, unlike the digital transistors
that are the building blocks of modern com-
puter chips, analog transistors more closely
mirror the biophysics of neurons. Specifi-
cally, very tiny currents in an analog circuit—
so tiny that the circuit was effectively “off”—
exhibited dynamics similar to the flow of ions
through channels in biological neurons when
that flow doesn’t lead to an action potential.
Intrigued by the work of Mead and his
colleagues, Giacomo Indiveri decided to do
 POWER GRID
Neuromorphic hardware takes a page from the architecture of animal nervous systems, relaying signals via spiking that
is akin to the action potentials of biological neurons. This feature allows the hardware to consume far less power and run
brain simulations orders of magnitude faster than conventional chips.

Conventional
hardware HOOVER
Neuromorphic DAM
THE SCIENTIST STAFF; INTEL CORPORATION; © ISTOCK.COM, JOLYGON; © ISTOCK.COM, VADIM ZHAKUPOV; PHOTO COURTESY UNISEM EUROPE LTD; © HEIDELBERG

hardware
Produces up to 2
SUPERCOMPUTER CHIP
gigawatts
Not analogous to neural function
Runs slower than biological neural
networks

Megawatts
WIND
TURBINE

Produces ~2-3
megawatts

BRAIN
~85 billion neurons DESKTOP
1 quadrillion synapses COMPUTER CHIP
Not analogous to neural function
Runs vastly slower than biologi-
20 watts cal neural networks

50–100 watts

LIGHT
BULB
SPINNAKER
Variable number of artificial neurons,
typically ~1,000 Uses ~20 watts
1 million synapses per 1,000 neurons
Runs at speed of biological neural networks
BRAINSCALES
1 watt 512 artificial neurons
128,000 synapses
10,000x faster than biological
neural networks

1 watt

LASER IN
CD/DVD
UNIVERSITY; INTEL CORPORATION; IBM RESEARCH

PLAYER LOIHI
~130,000 artificial neurons
Uses ~5-10 milliwatts 130 million synapses
Runs at speed of biological
neural networks

IBM’S TRUE NORTH Tens of milliwatts

~1 million artificial neurons
ER

256 million synapses

Runs at speed of biological
W

neural networks
PO

Tens of milliwatts
HEARING
AID

Uses less than 1 milliwatt

 his postdoctoral research at Caltech in the
mid-1990s. Now a neuromorphic engineer
at the University of Zurich in Switzerland,
Indiveri runs one of the few research groups
continuing Mead’s approach of using low-
current analog circuits. Indiveri and his team
design the chips’ layout by hand, a process
that can take several months. “It’s pencil and
paper work, as we try to come up with elegant
solutions to implementing neural dynamics,”
he says. “If you’re doing analog, it’s still very
much an art.”
Once they finalize the layout, they email
the design to a foundry—one of the same pre-
cision metal-casting factories that manufac-
ture chips used in smartphones and com-
puters. The end result looks roughly like a
smartphone chip, but it functions like a net-
work of “neurons” that propagate spikes of
electricity through several nodes. In these
analog neuromorphic chips, signals are
relayed via actual voltage spikes that can vary
in their intensity. As in the brain, information
is conveyed through the timing of the spikes
from different neurons.
“If you show the output of one of these
neurons to a neurophysiologist, he will not
be able to tell you whether it’s coming from
a silicon neuron or from a biological neu-
ron,” says Indiveri.
If you’re doing analog, it’s still very much
an art.
These silicon neurons represent an
imperfect attempt to replicate the nervous
system’s organic wetware. Biological neurons
are mixed analog-digital systems; their action
potentials mimic the discrete pulses of digital
hardware, but they are also analog in that the
voltage levels in a neuron influence the infor-
mation being transmitted.
Analog neuromorphic chips feature sili-
con neurons that closely resemble the physi-
cal behavior of biological neurons, but their
analog nature also makes the signals they
transmit less precise. While our brains have
evolved to compensate for their imprecise
components, researchers have instead taken
the basic concept into the digital realm. Com-
panies such as IBM and Intel have focused
34 T H E SC I EN TIST | the-scientist.com
on digital neuromorphic chips, whose silicon
neurons replicate the way information flows
in biological neurons but with different phys-
ics, for the same reason that conventional dig-
ital chips have taken over the vast majority of
our computers and electronics—their greater
reliability and ease of manufacture.
But these digital chips maintain their
neuromorphic status in the way they capture
the brain’s architecture. In these digital neuro-
morphic chips, the spikes come in the form
of packets of information rather than actual
pulses of voltage changes. “That’s just dramat-
ically different from anything we convention-
ally design in computers,” says Intel’s Davies.
Whatever form the spikes take, the sys-
tem only relays the message when inputs
reach a certain threshold, allowing neuro-
morphic chips to sip rather than guzzle power.
This is similar to the way the brain’s neurons
communicate whenever they’re ready rather
than at the behest of a timekeeper. Conven-
tional chips, on the other hand, are mostly
linear, shuttling data between memory hard-
ware, where data are stored, and a processor,
where data are computed, under the control
of a strict internal clock.
When Modha was designing IBM’s
neuromorphic chip, called TrueNorth, he
first analyzed long-distance wiring diagrams
—Giacomo Indiveri, University of Zurich
of the brain, which map how different brain
regions connect to each other, in macaques
and humans.2 “It really began to tell us about
the long-distance connectivity, the short-
distance connectivity, and about the neuron
and synapse dynamics,” he says. By 2011,
Modha created a chip that contained 256
silicon neurons, the same scale as the brain
of the worm C. elegans. Using the latest chip
fabrication techniques, Modha packed in the
neurons more tightly to shrink the chip, and
tiling 4,096 of these chips together resulted
in the 2014 release of TrueNorth, which
contains 1 million synthetic neurons—about
the scale of a honeybee brain—and con-
sumes a few hundred times less power than
conventional chips.3
Neuromorphic chips such as True-
North have a very high degree of connec-
tivity between their artificial neurons, simi-
lar to what is seen in the mammalian brain.
The massively parallel human brain’s 85 bil-
lion neurons are highly interconnected via
approximately 1 quadrillion synapses.
TrueNorth is quite a bit simpler—con-
taining 256 million “synapses” connecting
its 1 million neurons—but by tiling together
multiple TrueNorth chips, Modha has cre-
ated two larger systems: one that simulates
16 million neurons and 4 billion synapses,
and another with 64 million neurons and 16
billion synapses. More than 200 researchers
at various institutions currently have access
to TrueNorth for free.
In addition to the highly interconnected
and spiking qualities of neuromorphic chips,
they copy another feature of biological ner-
vous systems: unlike traditional computer
chips, which keep their processors and mem-
ory in separate locations, neuromorphic chips
tend to have lots of tiny processors, each with
small amounts of local memory. This config-
uration is similar to the organization of the
human brain, where neurons simultaneously
handle both data storage and processing.
Researchers think that this element of neu-
romorphic architecture could enable mod-
els built with these chips to better replicate
human learning and memory.
Learning ability was a focus of Intel’s
Loihi chip, first announced in September
2017 and shared with researchers in Janu-
ary of last year.4 Designed to simulate about
130,000 neurons and 130 million synapses,
Loihi incorporates models of spike-timing-
dependent plasticity (STDP), a mechanism
by which synaptic strength is mediated in
the brain by the relative timing of pre- and
post-synaptic spikes. One neuron strengthens
its connection to a second neuron if it fires
just before the second neuron, whereas the
connection strength is weakened if the firing
order is reversed. These changes in synaptic
strength are thought to play an important role
in learning and memory in human brains.
Davies, who led the development of Loihi,
says the intention is to capture the rapid, life-
long learning that our brains are so good at
and current artificial intelligence models are
not. Like TrueNorth, Loihi is being distrib-
 uted to various researchers. As more groups
use these chips to model the brain, Davies
says, “hopefully some of the broader prin-
ciples of what’s going on that explain some
of the amazing capability we see in the brain
might become clear.”

Neuromorphics for neuroscience

For all their potential scientific applications,
TrueNorth and Loihi aren’t built exclu-
sively for neuroscientists. They’re primarily BUILDING BLOCKS: Each SpiNNaker chip is pack-
aged together with memory (above), then tiled
research chips aimed at testing and optimiz-
together into larger devices, such as the 48-node
ing neuromorphic architecture to increase board on the right. Multiple boards can be connected
its capabilities and ease of use, as well as together to form larger SpiNNaker systems (below).
exploring various potential commercial
applications. Those range from voice and
PHOTOS COURTESY UNISEM EUROPE LTD; © UNIVERSITY OF MANCHESTER, BY STEVE FURBER AND COLLEAGUES; UNIVERSITY OF MANCHESTER

gesture recognition to power-efficient robot-

ics and on-device machine learning models
that could power smarter smartphones and
self-driving cars. The EU’s Human Brain
Project, on the other hand, has developed
two neuromorphic hardware systems with
the explicit goal of understanding the brain.
BrainScaleS, launched in 2016,5 com-
bines many chips on large wafers of sili-
con—more like ultra-thin frisbees than fin-
gernails. Each wafer contains 384 analog
chips, which operate rather like souped-up
versions of Indiveri’s analog chips, optimized
for speed rather than low power consump-
tion. Together they simulate about 200,000
neurons and 49 million synapses per wafer.
BrainScaleS, along with the EU’s other
neuromorphic system, called SpiNNaker, ben-
efits from being part of the Human Brain Proj-
ect’s large community of theoretical, experi-
mental, and computational neuroscientists.
Interactions with this community guide the
addition of new features that might help sci-
entists, and allow new discoveries from both
systems to quickly ripple back into the field.
Steve Furber, a computer engineer at
the University of Manchester in the UK,
conceived of SpiNNaker 20 years ago,
and he’s been designing it for more than a
decade. After toiling away at the small dig-
ital chips underlying SpiNNaker for about
six years, Furber says, he and his colleagues
achieved full functionality in 2011. Ever
since, the research team has been assem-
bling the chips into machines of ever-
increasing size, culminating in the million-
processor machine that was switched on in
late 2018.6 Furber expects that SpiNNaker
should be able to model the 100 million
neurons in a mouse brain in real time—
something conventional supercomputers
would do about a thousand times slower.
Access to the EU Human Brain Project
systems is currently free to academic labs.
Neuroscientists are starting to run their
own programs on the SpiNNaker hardware
to simulate high-level processing in specific
subsystems of the brain, such as the cere-
bellum, the cortex, or the basal ganglia. For
example, researchers are trying to simulate
a small repeating structural unit—the corti-
cal microcolumn—found in the outer layer
of the brain responsible for most higher-
level functions. “The microcolumn is small,
but it still has 80,000 neurons and a quar-
ter of a billion synapses, so it’s not a trivial
thing to model,” says Furber.
Next, he adds, “we’re beginning to sort
of think system-level as opposed to just indi-
vidual brain regions,” inching closer to a full-
scale model of the 85 billion–neuron organ
that powers human intelligence.
Mimicking the brain
Modeling the brain using neuromorphic
hardware could reveal the basic princi-
ples of neuronal computation, says Rich-
ard Granger, a computational neuroscien-
tist at Dartmouth College. Neuroscientists
can measure the biophysical and chemical
properties of neurons in great detail, but it’s
hard to know which of these properties are
actually important for the brain’s computa-
tional abilities. Although the materials used
in neuromorphic chips are nothing like the
cellular matter of human brains, models
using this new hardware could reveal com-
putational principles for how the brain shut-
tles and evaluates information.
Replicating simple neural circuits in sil-
icon has helped Indiveri discover hidden

05 . 2019 | T H E S C IE N T IST 3 5
 BUILDING A FUNCTIONAL MODEL
OF THE BRAIN
Neuromorphic technology is powering ever bigger and more-complex brain models, which had begun to reach their limits with modern super-
computing. Spaun is one example. The 2.5 million–neuron model recapitulates the structure and functions of several features of the human brain to
perform a variety of cognitive tasks. Much like humans, it can more easily remember a short sequence of numbers than a long sequence, and is bet-
ter at remembering the first few and last few numbers than the middle numbers. While researchers have run parts of the current Spaun model on
conventional hardware, neuromorphic chips will be crucial for efficiently executing larger, more-complicated versions now in development.

Motor
6 output Motor
5 processing

Motor
Parietal area
cortex

Information
4 decoding
Prefrontal
cortex

Visual
cortex Temporal cortex

1 Visual Input

Working
3 memory

Information
2 encoding

When shown a series of numbers  1 , Spaun’s visual system compresses When asked to recall the sequence sometime later, the model’s information
and encodes the image of each number as a pattern of signals akin to decoding area decompresses each number, in turn, from the stored list  4,
the firing pattern of biological neurons. Information about the content and the motor processing system maps the resulting concept to a motor com-
© MARINA MUUN

of each image—the concept of the number—is then encoded as another
spiking pattern 2 , before it is compressed and stored in working memory
along with information about the number’s position in the sequence  3.
This process mimics the analysis and encoding of visual input in the visual
and temporal cortices and storage in the parietal and prefrontal cortices.
5 . Finally, the motor system translates the motor com-
mand firing pattern 
mand for a computer simulation of a physical arm to draw each number  6.
This is akin to human recall, where the parietal cortex draws memories
from storage areas of the brain such as the prefrontal cortex and translates
them into behavior in the motor area.
benefits of the brain’s design. He once gave a
PhD student a neuromorphic chip that had
the ability to model spike frequency adap-
tation, a mechanism that allows humans to
habituate to a constant stimulus. Crunched
for space on the chip, the student decided
not to implement this feature. However, as
he worked to lower the chip’s bandwidth
and power requirements, he ended up with
something that looked identical to the spike
frequency adaptation that he had removed.
Indiveri and his colleagues have also discov-
ered that the best way to send analog signals
over long distances is to represent them not
as a continuously variable stream, for exam-
ple, but as a sequence, or train, of spikes,
just like neurons do. “What’s used by neu-
rons turns out to be the optimal technique
to transmit signals if you want to minimize
power and bandwidth,” Indiveri says.
Neuromorphic hardware can also allow
researchers to test their theories about brain
functioning. Cornell University computa-
tional neuroscientist Thomas Cleland builds
models of the olfactory bulb to elucidate the
principles underpinning our sense of smell.
Using the Loihi chip enabled him to build
hardware models that were fast enough to
mimic biology. When given data from che-
mosensors—serving as artificial versions
of our scent receptors—the system learned
to recognize odors after being exposed to
just one sample, outperforming traditional
machine learning approaches and getting
closer to humans’ superior sniffer.
“By successfully mapping something
like that and actually showing it working in
a neuromorphic chip, it’s a great confirma-
tion that you really do understand the sys-
tem,” says Davies.
Cleland’s olfaction models didn’t always
work out as expected, but those “failed”
experiments were just as revealing. The
odor inputs sometimes looked different to
the sensors than the model predicted, possi-
bly because the odors were more complex or
noisier than expected, or because tempera-
ture or humidity interfered with the sensors.
“The input gets kind of wonky, and we know
that that doesn’t fool our nose,” he says. The
researchers discovered that by paying atten-
tion to previously overlooked “noise” in the
odor inputs, the olfactory system model could
correctly detect a wider variety of inputs. The
results led Cleland to update his model of
olfaction, and researchers can now look at
biological systems to see if they use this previ-
ously unknown technique to recognize com-
plex or noisy odors.
Cleland is hoping to scale up his model,
which runs in biological real time, to analyze
odor data from hundreds or even thousands
of sensors, something that could take days
to run on non-neuromorphic hardware. “As
long as we can put the algorithms onto the
neuromorphic chip, then it scales beautifully,”
he says. “The most exciting thing for me is
being able to run these 16,000-sensor data-
sets to see how good the algorithm is going to
get when we do scale up.”
SpiNNaker, TrueNorth, and Loihi can all
run simulations of neurons and the brain at
the same speed at which they occur in bio-
logy, meaning researchers can use these
chips to recognize stimuli—such as images,
gestures, or voices—as they occur, and pro-
cess and respond to them immediately. Apart
from allowing Cleland’s artificial nose to pro-
cess odors, these capabilities could enable
robots to sense and react to their environ-
ment in real time while consuming very little
power. That’s a huge step up from most con-
ventional computers.
For some applications, such as model-
ing learning processes that can take weeks,
months, or even years, it helps to be a bit faster.
That’s where BrainScaleS comes in, with its
ability to operate at about 1,000–10,000
times faster than biological brains. And the
system is only getting more advanced. It’s
in the process of being upgraded to Brain-
ScaleS2, with new processors developed in
close collaboration with neuroscientists.
The new system will be able to better
emulate learning and model chemical pro-
cesses, such as the effects of dopamine on
learning, that are not replicated in other
neuromorphic systems. The researchers say
it will also be able to model various kinds
of neurons, dendrites, and ion channels,
and features of structural plasticity such as
the loss and growth of synapses. Maybe one
day the system will even be able to approx-
imate human learning and intelligence. “To
understand biological intelligence is, I think,
by far the biggest problem of the century,”
says Johannes Schemmel, a biophysicist at
Heidelberg University who helped develop
BrainScaleS. (See “The Intelligence Puzzle,”
The Scientist, November 2018.)
Current artificial intelligence systems still
trail the brain when it comes to flexibility and
learning ability. “Google’s networks became
very good at recognizing images of cats once
they were shown 10 million images of cats,
but if you show my two-year-old grandson
one cat he will recognize cats for the rest of
his life,” says Furber.
With advances to Loihi set to roll out
later this year, Eliasmith hopes to be able
to add more high-level cognitive and learn-
ing behaviors to his Spaun model. He says
he’s particularly excited to try to accurately
model how humans can quickly and easily
learn a cognitive task, such as a new board
game. Famed AI gamers such as AlphaGo
must model millions of games to learn how
to play well.
It’s still unclear whether replicating
human intelligence is just a matter of build-
ing larger and more detailed models of the
brain. “We just don’t know if the way that
we’re thinking about the brain is somehow
fundamentally flawed,” says Eliasmith. “We
won’t know how far we can get until we have
better hardware that can run these things in
real time with hundreds of millions of neu-
rons,” he says. “That’s what I think neuro-
morphics will help us achieve.” g
Sandeep Ravindran is a freelance science
journalist living in New York City.
References
1. C. Eliasmith et al., “A large-scale model of the
functioning brain,” Science, 338:1202–05, 2012.
2. D.S. Modha, R. Singh, “Network architecture of the
long-distance pathways in the macaque brain,” PNAS,
107:13485–90, 2010.
3. P.A. Merolla et al. “A million spiking-neuron
integrated circuit with a scalable communication
network and interface,” Science, 345:668–73, 2014.
4. M. Davies et al. “Loihi: A neuromorphic manycore
processor with on-chip learning,” IEEE Micro,
38:82–99, 2018.
5. J. Schemmel et al., “A wafer-scale neuromorphic
hardware system for large-scale neural modeling,”
Proc 2010 IEEE Int Symp Circ Sys, 2010.
6. S.B. Furber et al., “The SpiNNaker Project,” Proc
IEEE, 102:652–65, 2014.
05 . 2019 | T H E S C IE N T IST 37
A Deeper
Look
With the help of computer programs that learn
from experience, researchers look for meaning
in vast volumes of image data.

BY JEF AKST

S
ix years ago, Steve Finkbeiner of the Gladstone Insti-
tutes and the University of California, San Francisco,
got a call from Google. He and his colleagues had
invented a robotic microscopy system to track single cells over
time, amassing more data than they knew what to do with. It
was exactly the type of dataset that Google was looking for to
apply its deep learning approach, a state-of-the-art form of
artificial intelligence (AI).

FROM ©ISTOCK.COM
CREDIT LINE
MODIFIED
 “We generated enough data to be interesting, is basically
what they said,” Finkbeiner recalls of the phone conversation.
“They were interested in blue-sky ideas—problems that either
humans didn’t think would even be possible or things that a
computer could do ten times better or faster.”
One application that came to Finkbeiner’s mind was to have a
neural network—an algorithm that commonly underlies deep learn-
ing approaches (see “Primer” on page 12)—examine microscopy
images and draw from them information that researchers had been
unable to visually identify. For example, images of unlabeled cells
taken with basic light microscopes do not reveal many details beyond
the cell’s overall size and shape. If a researcher is interested in a par-
ticular biological process, she will typically use stains or fluorescent
tags to look for finer-scale cellular or molecular structures. Maybe,
Finkbeiner thought, a computer model could learn to see these fine
details that scientists couldn’t when looking at untreated samples.
In April of last year, he and his colleagues published their
results: after some training, during which the deep learning
networks were shown pairs of labeled and unlabeled images,
the models were able to differentiate cell types and identify
subcellular structures such as nuclei and neuronal dendrites.1
“The upshot of that article: the answer was, resoundingly, yes.
You could take images of unlabeled cells and . . . predict images
of labeled and unlabeled structures,” says Finkbeiner. “It’s a
way to, almost for free, get a lot more information [on] the
cells you’re studying.” The very next month, a second team of
researchers similarly showed that a deep learning model could
identify stem cell–derived endothelial cells without staining.2
(See “Into the Cell” on page 18.)
cal Research Centre of the Hungarian Academy of Sciences.
“It’s really changing the field of image analysis.”
Identity crisis
There are many ways scientists design intelligent computer sys-
tems to identify cells of interest. Last year, Aydogan Ozcan of the
University of California, Los Angeles, and colleagues built sen-
sors that focused a deep learning network on microscopy images
generated by a mobile phone microscope to screen the air for
pollen and fungal spores. The model accurately identified more
than 94 percent of samples of the five bioaerosols it was trained
on.3 “This is where AI is getting really powerful,” says Ozcan, who
notes that traditional screening methods require sending samples
out for analysis, while this method can be done on the spot. In
2017, a group of Polish researchers used deep learning algorithms
PAINT BY NUMBERS: Using unlabeled images, a neural network identi-
fies neurons derived from induced pluripotent stem cells (green) in a cul-
ture containing diverse cell types (nuclei in blue). The error map highlights
pixels where the model’s prediction was too bright (magenta) or too dim
(teal). Scale bars = 40 mm.
BRIGHTFIELD TRUE NEURON LABEL

Deep learning is really dominant at the

moment. It’s really changing the field of
image analysis.
—Peter Horvath, Hungarian Academy of Sciences

Such is the power of AI, and deep learning in particular.

The approach draws inspiration from the neuron-based archi- PREDICTED NEURON LABEL ERROR MAP
tecture of the animal nervous system, with many layers of inter-
connected nodes that communicate information and learn from
experience. From basic research needs, such as cell type identi-
fication, to public health and biomedical applications, sophis-
ticated machine learning models are changing how researchers
interact with visual data.
Because of their simplicity and power, these approaches
CELL, 173:P792–803.E19, 2018

are quickly becoming popular for biological image analyses.

“The use of deep learning in microscopy—there was really
almost nothing in 2017,” says Samuel Yang, a research scien-
tist at Google who collaborated on the cell microscopy proj-
ect with Finkbeiner. “In 2018, it [took] off.”
“Deep learning is really dominant at the moment,” agrees
Peter Horvath, a computational cell biologist at the Biologi-

05 . 2019 | T H E S C IE N T IST 3 9
 to classify bacterial genera and species in microscopy images, an
important part of basic and applied research in fields such as agri-
culture, food safety, and medicine.4
In the biomedical sciences, AI approaches that distinguish
among human cell types could improve diagnosis. Finkbeiner
has partnered with the Michael J. Fox Foundation for Parkinson’s
Research to create a neural network that can differentiate between
stem cell–derived neurons from patients with Parkinson’s disease
and those from healthy controls. Traditionally, researchers have
identified a variable that differs between diseased and healthy cells,
but the AI model can learn to use any number of qualities in the
image to make its assessment, Finkbeiner says. This multivariate
approach should reveal a more complete view of the cell’s health,
he says, and help researchers screen for treatments that move a cell
toward a healthy state using this more holistic perspective. “With
deep learning, you can really transform the way screening is done.”
The use of AI for analysis of microscopy images is nothing
new to the cancer field, which has been applying machine learn-
ing approaches to analyze micrographs of biopsy samples for more
than a decade. In the last couple of years, deep learning has become a
popular tool to try to improve diagnoses and treatment regimes. (See
“Run Program: Cancer” on page 22.) After training on images that
have been graded by a pathologist, these models can learn to classify
various tumors, including lung and ovarian, and to predict cancer
progression, often more accurately than clinicians can.
PREDICTING PROSTATE CANCER PROGRESSION: The normal prostate
contains microscopic glands that appear as ring-like or circle structures
(grey in leftmost image). Artificial intelligence (AI) based methods, includ-
ing deep learning approaches, may identify abnormalities in these and
other structures to predict clinical outcomes from histological images of
prostate cancer. In the far right image, red is used to denote complete ring
structures while green denotes abnormal ring fragments indicative of can-
cer. This information can be used to predict the cancer’s grade, a measure
of disease severity. AI algorithms can also evaluate levels of the androgen
receptor (blue in center image), which, when present and activated, drives
tumor growth and cell proliferation (bright yellow).
40 T H E SC I EN TIST | the-scientist.com
The key to these artificially intelligent models is the com-
puter’s agnostic approach, says Carlos Cordon-Cardo, a physi-
cian scientist at the Icahn School of Medicine at Mount Sinai
who last year coauthored a study on a machine learning algo-
rithm that graded prostate cancers with a high level of accu-
racy. “Rather than force the markers, [it’s] more of an open
box; the algorithm selects what may be the best players.”
Which elements of the cellular environment are impor-
tant for a deep learning model to make accurate evaluations
It’s still pretty early days in the field. But
already, we’ve been pretty blown away by
how powerful the approach is.
—Steve Finkbeiner, Gladstone Institutes and
University of California, San Francisco
is often unclear. “It learns by itself. . . . I cannot say what the
neural network does,” says Christophe Zimmer, a computa-
tional biophysicist at the Pasteur Institute in Paris. “And the
neural network cannot tell me, unfortunately.”
Nonetheless, that information is valuable, as it could provide
insights into disease processes that might in turn lead to better
diagnostics or treatments. So researchers will “try to break the
model,” Finkbeiner explains. By taking away or blurring parts
of the image, scientists can see what reduces the model’s perfor-
mance. This, then, points to those elements as being key for the
model’s assessments, yielding clues about what the models “see”
in the images that might have some biological underpinning.
“It’s one thing to produce data but another to produce
knowledge,” says Cordon-Cardo.
AI improves resolution
Using deep learning to analyze microscopy images is just one
example of how AI can influence visual data, says Ozcan. “This
is the first wave—let there be an image captured as before . . .
COURTESY OF CARLOS CORDON-CARDO
 MODEL INPUT MODEL OUTPUT GROUND TRUTH
and we’ll act on [it] better than before. The second wave is
even more exciting to me. That is how AI is helping us with
image generation.”
Finkbeiner’s work with Google is one example. The deep
neural network can, using brightfield images, generate images
that look like fluorescence micrographs. And Ozcan and his
colleagues have used deep learning approaches to improve the
quality of smartphone microscopes5 and create holographic
reconstructions to visualize objects in three dimensions based
on a single snapshot.6 Most recently, his group enhanced basic
images taken with benchtop microscopes into images of a qual-
ity similar to or better than those taken by Nobel Prize–win-
ning super-resolution technologies.7 (See “AI Networks Gen-
erate Super-Resolution from Basic Microscopy,” The Scientist,
December 17, 2018.) These improved images can then be fed into
deep learning models that analyze their content, Ozcan says.
“It’s still pretty early days in the field,” says Finkbeiner.
But already, “we’ve been pretty blown away by how powerful
the approach is.” His team is now adapting its robotic system
to integrate AI directly into microscopes, teaching them to
recognize individual cells not by their physical location but
by “facial” recognition. One day, he says, researchers may be
able to program the system to conduct experiments autono-
mously—delivering drugs to certain cells based on what it
OZCAN LAB AT UCLA
observes, for example—which “could accelerate discovery and
lead to less bias,” Finkbeiner says. (See “Profile” on page 54.)
In addition to being powerful, AI paired with microscopy is
user-friendly. “Deep learning is just so powerful and so easy to
use just kind of out of the box,” says Yang, if you have the right
UPSAMPLING: Deep learning approaches can transform images
taken with a smartphone microscope into an image akin to a
micrograph taken with a benchtop instrument (left), or turn such
standard micrographs into super-resolution captures (top).
problem and data. He says he was surprised the first time a col-
league showed him that a deep neural network model trained on
images of dogs, cats, and cars from the internet could be adapted
to cluster microscopy images of breast cancer cells “almost per-
fectly” based on what type of drug the cells had been treated with.
For cellular applications, most researchers will recommend their
models be trained on relevant sets of images to ensure accuracy,
but experts say the approach itself is essentially transferrable.
“If there is a state-of-the-art paper in computer vision,
it doesn’t really take any modification for it to be applicable
to biology,” says Allen Goodman, a computer scientist at the
Broad Institute of MIT and Harvard. “It’s Google, Apple, etc.—
we can benefit from these methods almost immediately.” g
References
1. E.M. Christiansen et al., “In silico labeling: Predicting fluorescent labels in
unlabeled images,” Cell, 173:P792–803.E19, 2018.
2. D. Kusumoto et al., “Automated deep learning–based system to identify
endothelial cells derived from induced pluripotent stem cells,” Stem Cell Rep,
10:P1687–95, 2018.
3. Y. Wu et al., “Label-free bioaerosol sensing using mobile microscopy and deep
learning,” ACS Photon, 5:4617–27, 2018.
4. B. Zieliński et al., “Deep learning approach to bacterial colony classification,”
PLOS ONE, 12:e0184554, 2017.
5. Y. Rivenson et al., “Deep learning enhanced mobile-phone microscopy,” ACS
Photon, 5:2354–64, 2018.
6. Y. Rivenson et al., “Phase recovery and holographic image reconstruction using
deep learning in neural networks,” Light-Sci Appl, 7:17141, 2018.
7. H. Wang et al., “Deep learning enables cross-modality super-resolution in
fluorescence microscopy,” Nat Methods, 16:103–10, 2019.
05 . 2019 | T H E S C IE N T IST 41
A Wider
View
AI makes inroads in analysis of images of the
world on a macroscopic scale.

BY CAROLYN WILKE

A
rtificial intelligence is changing how researchers
examine the microscopic biological world. At the
same time, machine learning approaches are being
applied to images on greater scales. From snapshots of the
brain and other organs to satellite images of Earth’s surface,
intelligent computer programs can spot trends or features
of complex systems that escape visual detection by experts.

FROM ©ISTOCK.COM
CREDIT LINE
MODIFIED
 Plant Spies
Instead of watering or applying pesticides to an entire field,
farmers may be able to be more selective, thanks to AI’s
ability to spot plants in need. Scientists and engineers from the
Spanish National Research Council (CSIC) in Córdoba, Spain,
deploy drones equipped with cameras to map areas of land and
snap pictures that can be mined for information using machine
learning and object-based image analysis, a method for group-
ing pixels into objects.
The team trained the model by providing information
about how much water plants had received along with a
set of pictures of the plants. In preliminary tests presented
at the SPIE Commercial + Scientific Sensing and Imaging
The researchers are hoping to create
AI-enabled “smart sprayers” that are
more efficient than ones currently on
the market.
JOSE M. PEÑA & JOE M. MAJA
meeting in April 2018, the algorithm could discriminate
between images of well-hydrated versus water-stressed
hydrangea and butterfly bush (doi:10.1117/12.2304739,
2018). These are “differences that are not obvious in the field,”
says Jose Peña, one of the CSIC researchers.
The team is also turning to AI to locate tiny weeds lurk-
ing in the field, allowing farmers to target small areas with
herbicides. For this application, the researchers employed
a machine learning approach called Random Forest that
learns from photos of the field. To differentiate between
crops and weeds, the model uses context clues such as size
or the placement of the plant in or outside of a row, achiev-
ing nearly 90 percent accuracy in one patch of sunflow-
ers (Remote Sens, 10:285, 2018). The researchers are work-
ing to optimize the algorithms, hoping to create AI-enabled
“smart sprayers” that are more efficient at detecting and
spraying weeds as they image rows of crops than ones cur-
rently on the market.
BIRD’S EYE VIEW: A drone flies over an experimental field of ornamen-
tal plants while snapping photos with a camera attached to its underside
(above). The images are fed into an object-based image analysis algorithm
that predicts the water stress condition of each plant in several processing
stages (left). The original image (top left) is segmented by the algorithm
(top right), grouped into plants (bottom left), and edges are removed (bot-
tom right), before extracting spectral data that are translated to a water
stress rating.
05 . 201 9 | T H E S C IE N T IST 4 3
 Camera Traps
Aided by cameras with shutters triggered by motion, animal
researchers can keep an eye on their field sites even from far
away. But such camera traps snap away at anything that passes
by, and it still takes a lot of human effort to slog through photos
to identify the animals and make note of what they’re doing. (See
“Wild Ones,” The Scientist, April 2017.)
A tool developed last year by researchers at the University
of Wyoming showed that AI—along with tens of thousands
of volunteers—can help with this task. In a project called
Snapshot Serengeti, citizen scientists labeled 3.2 million
pictures—tagging them with information such as the spe-
cies present, numbers of individuals captured in the shots,
and the animals’ behavior. The researchers then developed a
convolutional neural network, a deep learning program that
mimics the way the brain makes connections, and showed
it the images that had been annotated by online volunteers.
After training, the model correctly identified roughly 94 per-
cent of the images (PNAS, 115:E5716–25, 2018).
The model can process the majority of the images, matching
the accuracy of human assessments, and hand the tough ones off to

SNAPSHOT SERENGETI PROJECT

experts, says Mohammad (Arash) Norouzzadeh, the first author of
the group’s work and a PhD student at the University of Wyoming.
Of course, some images are tricky even for people, he adds. For
instance, when an animal is moving, is too close to or too far from
the camera, or is partly out of frame, the picture may not receive
enough labels to be used for training. The next step is to develop
more-advanced algorithms that can extract information from cam-
era trap projects with less training, Norouzzadeh says. NOW YOU SEE ME: The neural network labels this image “zebra, moving.”

Forecasting Wildfires
Satellite cameras capture wildfires blazing across the land. With
these pictures in hand, researchers at the University of Waterloo
in Ontario, Canada, are using an AI strategy called reinforcement
learning to create models that predict how the fires spread. In an
iterative process using images from previous fires, the model receives
images showing a fire’s location every 16 days, the length of time
between satellite pictures. The model then predicts the next 16 days’
spread and receives feedback about the accuracy of its prediction,
improving the model’s understanding of how fires move. As it pro-
gresses, the model learns “rules” that wildfires follow—for example,
that fire stops when it meets a lake (Front ICT, 5:6, 2018).
The researchers found that their model holds up well
against others developed by machine learning and against
USGS/NASA LANDSAT PROGRAM; SRIRAM GANAPATHI SUBRAMANIAN
AND MARK CROWLEY, TAKEN FROM FRONT ICT, 5:6, 2018
physics-based models. Tackling wildfire spread by apply-
ing AI to real world datasets presents a challenge because
the data are noisy. “The real world has a lot of complications
that we don’t anticipate,” says study coauthor Mark Crowley,
a computer scientist at Waterloo. “It will push us to find bet-
ter algorithms or software.”
Other problems that reinforcement learning could study in
the environmental realm involve the spread of infectious dis-
ease and climate, says Crowley. Researchers have already applied
other machine learning techniques to predict flooding (Neural
Comput Appl, 27:1129–41, 2016) and drought (Geomat Nat Haz
Risk, 8:1080–102, 2017), and computer scientists are continu-
ally working to make their tools more powerful.

44 T H E SC I EN TIST | the-scientist.com
 Brain Scans
Scientists use three-dimensional MRI scans to track how a
brain showing signs of Alzheimer’s disease changes over time.
With the goal of improving Alzheimer’s disease diagnoses,
researchers from Columbia University and Carnegie Mellon
University trained a convolutional neural network to mine
those scans. After being shown brain scans from patients with
the disease and from healthy controls, the model could dif-
ferentiate between diseased and healthy brains in scans it
hadn’t previously seen with 93 percent accuracy (bioRxiv,
doi:10.1101/456277, 2018). Another model that analyzed 2-D
slices extracted from the 3-D scans also had good success,
indicating that patients could someday receive a diagnosis
from a shorter scan that images less of the brain, according
to the researchers.
“A lot of neurologic and psychiatric con-
ditions are . . . localized to certain areas
of the brain,” says Frank Provenzano,
one of the study’s authors. In addition
XINYANG FENG/COLUMBIA UNIVERSITY
shed light on which regions of the brain drive the disease.
Alzheimer’s shrinks the brain, causing changes to several
parts of the organ, so the scientists were surprised that their
model identified one region—the hippocampal formation,
which includes the hippocampus and nearby entorhinal cor-
tex—as the major contributor to its prediction of Alzheimer’s.
Researchers are using similar AI approaches in other
health-care contexts, including assessing MRI images of
infants’ brains to assess the risk of autism (Nature, 542:348–
51, 2017), analyzing tumors in the liver (Radiology: Artificial
Intelligence, 1:e180019, 2019), and scanning the eye to probe
retinal disease (Brit J Ophthalmol, 103:167–75, 2019).

to improving Alzheimer’s diagnoses,

deep learning approaches could

AI SCANS: A 3-D rendering of the human brain

shows regions of the brain in different colors.
The most important areas to a deep learning
method’s classification of Alzheimer’s disease
are the hippocampus (yellow) and entorhinal
cortex (blue), shown in the inset.

FAST-MOVING FIRES:
Satellite imagery show-
ing the Fort McMurray
Fire in Alberta, Canada,
in 2016 (left) is ana-
lyzed by a reinforcement
model that predicts the
fire’s movement (right).
Correctly predicted fire
pixels are red, blue pix-
els were predicted incor-
rectly, and black pixels
were correctly labeled
not ablaze.

05 . 201 9 | T H E S C IE N T IST 4 5
 EDITOR’S CHOICE PAPERS
The Literature
GENETICS
CRISPR Predictions
THE PAPERS
M.W. Shen et al., “Predictable and precise
template-free CRISPR editing of pathogenic
variants,” Nature, 563:646–51, 2018.
F. Allen et al., “Predicting the mutations gener-
ated by repair of Cas9-induced double-strand
breaks,” Nat Biotechnol, 37:64–72, 2019.
During gene editing with CRISPR technol-
ogy, the Cas9 scissors that cut DNA home in
on the right spot to snip with the help of guide
RNA. The way the genetic material is stitched
back together afterward isn’t terribly precise,
though; in fact, scientists have long thought
that without a template, the process is ran-
dom. However, “there’s been anecdotal evi-
dence that cells don’t repair DNA randomly,”
geneticist Richard Sherwood of Brigham and
Women’s Hospital tells The Scientist. A 2016
paper also suggested patterns in the repairs.
Sherwood wondered if artificial intelligence
could predict these outcomes.
In a study published last year in Nature,
Sherwood and colleagues describe how they
trained a machine learning algorithm called
inDelphi to predict repairs made to DNA
snipped with Cas9, using experimental data
from 1,872 target sequences cut and then
restitched in mouse and human cell lines. The
algorithm showed that 5–11 percent of the guide
RNAs used induced a single, predictable repair
genotype in the human genome in more than 50
percent of editing products. In other words, the
edits aren’t random, the team reports.
Separately, Felicity Allen and Leopold
Parts of the Wellcome Sanger Institute in
the UK and colleagues created an algorithm
called FORECasT (favored outcomes of repair
events at Cas9 targets) to do the same thing.
Based on a library of 41,630 guide RNAs
and the sequences of the targeted loci before
and after repair—a dataset that totaled more
46 T H E SC I EN TIST | the-scientist.com
CRISPR-Cas9
double strand break
Types of repairs
Insertion of
single base pair
Small deletion
Microhomology
deletions
CRYSTAL BALL: CRISPR guide RNAs target specific spots in the genome for the Cas9 enzyme to cut, forming a
double-strand break. A machine learning algorithm predicts which types of repairs will be made at a site targeted
by a specific guide RNA. Possibilities include an insertion of a single base pair 
change known as a microhomology deletion  c. .
than 1 billion repairs in various cell types—the
model showed that the majority of repairs are
either single base insertions, small deletions,
or longer deletions called microhomology-
mediated deletions, and are based on specific
sequences that exist at the Cas9-cut site. The
algorithm was then able to use the sequences
that determine each repair to predict Cas9
editing outcomes, the researchers reported in
Nature Biotechnology. The predicted repairs
are similar to Sherwood’s, but based on much
more data, Allen and Parts say.
“It’s the right place and the right
time for these predictions to occur,” says
Rich Stoner, the chief science officer at
Synthego, a genome engineering company
interested in developing repair-prediction
programs, similar to inDelphi, FORECasT,
and a third one called SPROUT, for com-
mercialization. However, Stoner notes,
a still-unpublished analysis of the three
algorithms’ results reveals that at times
they all made vastly different predictions
for the same cuts in the same types of cells,
a. , a small deletion 
b. , or a larger
suggesting that the algorithms’ accuracy
needs improvement.
Accurate predictions of sequence repair
could allow researchers to computationally
predict the precise guide RNAs that will repro-
duce exact human patient mutations, leading
to the development of better research models
to study genetic disease. Sherwood and his col-
leagues also showed that their algorithm could
predict which guide RNAs would be needed
to—without a repair template—correct dis-
ease-causing mutations found in human
patients, a clinical application of CRISPR that
is still years, if not decades, from becoming a
reality. The predicted repairs worked on cell
lines from patients with a rare genetic disease
that causes a blood clotting deficiency and albi-
nism, and another that includes growth failure
and nervous system deterioration.
Next, Sherwood says, “we would want
© KELLY FINAN
to test whether we can fix disease-causing
mutations in animal models, with an even-
tual goal of doing so for human patients.”
—Ashley Yeager
 CLEANING UP RNA SEQ DATA: Computational methods help researchers reduce noise in the data generated during single-cell RNA sequencing.
TECHNIQUES
Intelligent Data Miners
THE PAPERS
G. Eraslan et al., “Single-cell RNA-seq denoising using a deep count
autoencoder,” Nat Commun, 10:390, 2019.
M. Büttner et al., “A test metric for assessing single-cell RNA-seq batch
correction,” Nat Methods, 16:43–49, 2019.
In the not-so-distant past, researchers had to pool thousands of cells
together for bulk RNA sequencing, yielding an averaged snapshot of
gene expression. But advances in technology and significant reductions
in cost now enable scientists to sequence RNA from single cells,
unleashing a flood of transcription data.
“It used to be that you had to wait for the biologists [to generate
data for analysis], but now we are the slow guys,” says Fabian Theis, a
computational biologist at Helmholtz Zentrum München in Germany.
“There’s just so much data to analyze that we can’t keep up.”
One difficulty with single-cell RNA sequencing data is separating
meaningful variation from noise. For instance, a gene may appear
“silent” because it is not expressed or because its expression was
missed for technical reasons. Theis and colleagues try to cut that
noise with an artificial intelligence algorithm called a deep count
autoencoder (DCA), an artificial neural network that can compress
gene expression data into fewer dimensions, distilling the information
down to its most important relationships.
To see how well these parameters capture the full picture, the
algorithm recreates a full-size dataset and compares it to the original,
noting the differences. It repeats this process, stopping once the
program doesn’t achieve an improvement after 25 cycles. In two trials
with simulated data, the scientists removed some sequences from
the database to introduce noise and found that the autoencoder could
recover cell-type information that had become masked. In their Nature
© ISTOCK.COM, JXFZSY
Communications paper published early this year, they also applied the
algorithm to several examples of real transcriptome data, using it in one
case to identify the cell types in a blood sample—a task required for
various medical and research applications.
Another group published an AI-based tool similar to DCA a few
months earlier, which could also recover data hidden by noise and cluster
cells into subgroups based on their mRNA (Nat Methods, 15:1053–
58, 2018). Compared with more-traditional statistical approaches,
the autoencoder techniques are “a more universal approach, quite
elegant, where you . . . can let the machine learning take care of fitting
all the parameters,” says Peter Kharchenko, a computational biologist
at Harvard University who was not involved in this research. Plus, he
adds, they’re very flexible and scalable. “The huge advantage of these
[models] is that it’s easier to build on these tools.”
In addition to extracting more information from individual data sets,
researchers can combine data from different days and augment their
pool of sequences with ones from other labs. Considering multiple data
sets together allows researchers to view a more complete landscape
of cellular biology, says Zhichao Miao, a computational postdoc at
the European Molecular Biology Laboratory–European Bioinformatics
Institute. However, sequence reads in different datasets are influenced
not only by biological variation in the cells being sequenced—differences
over time or between treatment groups—but by unintended variation in
experimental conditions and in the methodology, such as the sequencing
protocol used.
Scientists use statistical approaches to remove such technical
noise—so-called batch effects—while trying to capture and dissect
biological variation. But there wasn’t a quantitative metric for measuring
how “batchy” the data remain, says Miao. So he, Theis, and their
collaborators developed a method called k-nearest-neighbor batch-
effect test, or kBET, that determines variance in the datasets and scores
the different approaches on how well they eliminate batch effects to
leave data that are “well mixed.”
The kBET work, published online in Nature Methods last December,
is a “good step forward,” says Kharchenko, but new approaches may be
required to evaluate batchiness in highly variable datasets, such as RNA
sequences from cancer patients and healthy people. “If you consider
the difference between samples to be technical variability, then it’s clear
what you want to do with it. You want to remove it,” says Kharchenko,
whose lab is developing tools for analyzing datasets with more-
systematic differences. “If we move to that more challenging setting [of
comparing very different groups], then the question itself becomes a
little bit less obvious.” —Carolyn Wilke
05 . 2019 | T H E S C IE N T IST 47
 PROFILE
Autopilot Advocate
Carnegie Mellon computational biologist Bob Murphy is betting that high-throughput experiments
orchestrated by machine learning algorithms will help crack biology’s mysteries.
BY SHAWNA WILLIAMS
I
t was the code that hooked Bob Murphy on biology. At age
13, he visited New York City’s American Museum of Natu-
ral History with his parents and picked up a copy of Isaac
Asimov’s book The Genetic Code in the gift shop. After reading
it, “I came downstairs, and I told my parents, ‘I know what I
want to do with my life,’” he recalls. “I was just fascinated by
the idea that you could decode biological information, that
biological systems were built on this DNA material that could
be converted into RNAs and proteins.”
That was in the mid-1960s. Murphy, true to his word, went on
to study biochemistry at Columbia University. Then in 1974, when
he was in graduate school at Caltech, he encountered another type
of code that would shape his career. One day, after he’d extracted
proteins from chromatin and run polyacrylamide gels of the sam-
ples, he and others in his lab were “having discussions about how
to analyze [the gels], and somebody said, ‘You know, that’s the
kind of thing that you could use a computer for,’” Murphy recalls.
“And I literally said, ‘What do you mean, use a computer?’”
Murphy was introduced to the computing lab at Caltech, and
was soon learning to code—first in BASIC, and later in Fortran
and MACRO-11. His advisor, James Bonner, acquired the lab’s
first minicomputer during Murphy’s time there, and Murphy
remembers connecting the lab’s spectrophotometers and other
instruments to the computer and developing software to allow
automated data collection. With that computational assistance,
he studied chromatin structure, determining that the distances
between DNA-histone complexes called nucleosomes change dur-
ing DNA replication. It was the start of a career devising com-
puter programs to answer biological questions.
HARNESSING BIG DATA
Murphy earned his PhD in 1980 and returned to Columbia for a post-
doc with Charles Cantor. In Cantor’s lab, Murphy started working on
identifying interactions between histones, which led him to the
question of how cells take up certain chromatin components in the
first place. He tackled the problem using a fluorescence cytom-
eter that Cantor had just acquired for the lab. “You could put a
suspension of cells in the instrument, and it would flow them
through a laser and then measure the fluorescence given off by
each cell,” Murphy explains. “This was a very early high-through-
put, data-generating instrument, and it was perfect for the kinds
of things that I was interested in, especially because it produced a
lot of data quickly.” Using fluorescent probes, he and Cantor were
among the first to show that the compartments containing mol-
48 T H E SC I EN TIST | the-scientist.com
ecules that have been gobbled from the cell’s surface via endocy-
tosis undergo a rapid drop in pH. The researchers also detailed
the kinetics of what happens to those materials once a compart-
ment enters the cell.
When it came time to apply for faculty jobs, Murphy saw an
ad that seemed too perfect to pass up: Carnegie Mellon Univer-
sity (CMU) in Pittsburgh was starting a new center for applying
fluorescence to biology experiments and was looking for profes-
sors. Murphy joined CMU in 1983 and has never left.
Murphy’s early work at CMU continued in a similar vein to his
research in Cantor’s lab, using fluorescence spectrometry to trace
the kinetics of what happens to materials after they are endocy-
tosed into cells. For example, his team found strong evidence that
the transmembrane protein Na+,K+–ATPase regulates acidifica-
tion that he and Cantor had identified in endosomes.
SEEING IS BELIEVING
In the course of his fluorescence spectrometry research, Murphy
learned that if one of his studies found, for example, that a partic-
ular cargo would end up inside a specific type of cellular compart-
ment, reviewers would ask what the compartment looked like.
“My first reaction was, well . . . the point here is to study the kinet-
ics and show the biochemistry of this,” he says. But eventually he
relented. “We would then just go and do a very simple microscopy
experiment, and that would make the reviewers happy. It actu-
ally hadn’t changed the story, but there was a picture to go along
with the story.”
Similarly, at scientific talks he went to in the early 1990s, Mur-
phy would see microscopy images given center stage—and, in his
view, used to support models of what happened inside cells via
tenuous reasoning. “I kept saying to myself, somebody has to try
to find a way to make these images into something closer to data,
something that you actually can operate on. . . . At a certain point
I decided that we would try to do it.”
Murphy “is one of these people who’s brilliant, and he’s almost
© CARNEGIE MELLON UNIVERSITY
always right,” says Mario Roederer, who was Murphy’s first gradu-
ate student in the mid-1980s and is now an immunologist at the
National Institute of Allergy and Infectious Diseases. “His work
ethic was amazing.”
That work ethic served Murphy well as he and then MD/PhD
student Michael Boland went about trying to computationally
analyze images of cells. They would label proteins that home to
specific organelles, then feed images of the labeled cells into a
computer to train it to use pattern recognition to group together

ROBERT MURPHY
Ray and Stephanie Lane Professor of Computational Biology,
Carnegie Mellon University
Computational biology department head, Carnegie Mellon University
Senior Fellow, Allen Institute for Cell Science
Fellow, American Institute of Medical and Biological Engineering, 2007
Senior Member, Institute of Electrical and Electronics Engineers, 2007
Honorary Professor of Biology, Albert Ludwig University of Freiburg, 2011
Senior Member, International Society for Computational Biology, 2018
Greatest Hits
• Found that ligands are acidified rapidly after being endocytosed
in mouse cells.
• Identified the transmembrane enzyme Na+,K+–ATPase as a key
regulator of endosome acidification.
• Trained an artificial neural network to recognize from microscopy
images where a protein had localized in HeLa cells, and later built a
model to predict a given protein’s distribution in cells.
• Showed that a machine learning algorithm with control of a
robotic liquid handler and microscope can be used to predict
the effects of a set of chemical compounds on the subcellular
localization of a set of proteins while directly testing just less than
a third of the possible combinations.
• Founded CMU’s computational biology department in 2009, as
well as the world’s first master’s program in automated science,
which will begin this fall.
cells that contained the same labeled protein. It was a machine
learning approach to visually identifying organelles within cells that
removed the need for a human to first make the determination of
which organelles were shown in training images. As a result, it could
potentially achieve greater accuracy than people could.
“When we started, I would go to cell biology meetings and talk
about the idea of recognizing what organelle a protein is in by this
automated approach, and the reaction most people had was . . . ‘No,
you have to go to grad school in cell biology to be able to tell the dif-
ference between [organelles].’ It took a lot of subsequent work to
convince people that this was a viable approach,” Murphy recalls.
His group’s first paper on the image recognition program came
out in 1997, and eventually his team was able to achieve “basically
perfect” accuracy with it, he says. Yet all was not smooth sailing.
He had originally hoped the program would learn to recognize pro-
teins with similar distribution patterns as members of a common
class—for example, to group different lysosome-localizing proteins
in the same bucket. Instead, the program picked up on subtle dif-
ferences in the distribution patterns, such that “almost every new
protein we looked at wasn’t readily recognized as being one of those
classes,” he explains. Rather than continue trying to develop a clas-
sification model, the group switched gears, embracing the messy
complexity of protein dynamics. They trained a model to break
down each protein’s distribution into a mixture of fundamental
patterns, so that it could determine which organelles a protein was
likely to be found in.
A NEW PARADIGM
Murphy’s fascination with biology has remained, even as the idea
that first attracted him to the field—that life is a code that can
be cracked, with each gene corresponding to a single function—
has failed the test of time. “The way we used to think about these
kinds of problems was to try to simplify things as much as pos-
sible—reductionism—but it’s been at least 20 years since we’ve
realized that that’s not going to work, that biological systems are
complex systems,” he told The Scientist late last year, in an inter-
view in his corner office in one of Carnegie Mellon’s newer build-
ings, overlooking the Steel City. Murphy talked about slides pro-
jected on a large wall-mounted flat-screen TV with the polish of
someone practiced at conveying the reasons for his excitement
to nonspecialists. Given biology’s complexity, he says, it’s just not
possible to do all of the experiments needed to figure out all the
interactions that govern each of a cell’s functions in a traditional,
hypothesis-driven way. “We need to have a way in which we can
05 . 201 9 | T H E S C IE N T IST 49
 PROFILE
only do only the experiments that we need to do and not do all
possible experiments.”
There are no hard and fast rules in biology, as there are in phys-
ics, Murphy elaborated in a later phone interview, because there are
always exceptions. Computational models provide a way forward
for biologists. For example, let’s say a researcher had 96 drug can-
didates and wanted to know how they’d act on 96 different proteins
within a cell line. Doing 9,216 experiments is out of the question,
so instead, the researcher aims to do some fraction of those experi-
ments, and use the results to train a machine learning program to
model what the outcome of the others would have been.
That training process will be most effective if it’s active rather
than passive, Murphy says. That means that, rather than feed-
ing a computer program a large data set for training, he wants
researchers to hand the reins over to the program from the get-
go, enabling it to determine which experiments’ results would be
most useful for improving its model—and then, to go get those
results. By hooking up computers that run machine learning pro-
grams to instruments such as robotic liquid handlers and micro-
scopes, and putting needed starting materials, such as drug can-
didates and cell lines, within robot arm’s reach, his group has
created automated setups that can construct and continuously
refine experiments in response to research questions. In the drug
candidate and cell-line example, the program starts out running
experiments with nearly random combinations of the two. As it
runs microarray analyses to “see” the results, it builds a model
predicting what the results would be of all possible combina-
tions. The program then interrogates that model to see which
of its predictions are the most uncertain, and runs those experi-
ments, using the results to further refine the model. As the cycle
repeats, the accuracy of the model increases.
“It doesn’t need us. It’s the same way that a self-driving car works,
in that you get in the self-driving car, and you tell it, ‘I want to go to
Cleveland,’ and it figures it out. You don’t tell it how to get to Cleveland,
you just tell it what your goal is,” Murphy says. Accordingly, he’s dubbed
these active machine learning setups “self-driving instruments.”
DRIVING INTO THE FUTURE
Not surprisingly, realizing this ambitious vision of experiments
with minimal human oversight requires plenty of people-powered
planning, setup, and tinkering. Even as his own lab works out the
kinks, Murphy has been working on launching a new CMU mas-
ter’s program in automated science to teach trainees how to set up
and maintain self-driving instruments. The first master’s program
students will arrive this fall to an array of new equipment to prac-
tice on, including robotic liquid handling, microscopy, and nucleic
acid extraction instruments. Murphy expects the program’s grad-
uates to fan out to industry employers and national labs, or to go
on to earn PhDs and eventually set up their own research labs that
harness artificial intelligence and automation.
The students in the new program should be prepared for a chal-
lenge. Both Roederer and Greg Johnson, who completed his PhD in
Murphy’s lab in 2016 and is now a scientist at the Allen Institute for
5 0 T H E SC I EN TIST | the-scientist.com
Cell Science in Seattle, say Murphy is a tough and rigorous mentor.
In one-on-one meetings, Johnson recalls Murphy repeatedly say-
ing, “What is the question that you’re asking, and why is it impor-
tant?”—and how does that jibe with the computational methods
used to address it. “[He] definitely was the first person who I met
who had a clear articulation of the tight coupling between compu-
tational modeling and biological research,” Johnson says.
The machine doesn’t need us. It’s the same
way that a self-driving car works, in that
you get in the self-driving car, and you
tell it, “I want to go to Cleveland,” and it
figures it out. You don’t tell it how to get to
Cleveland, you just tell it what your goal is.
- Bob Murphy, CMU
Murphy expects that the utility of such modeling, coupled with
self-driving instruments, will extend to experiments on many differ-
ent species. He and colleague Joshua Kangas have already collabo-
rated with plant biologists to automate an experiment on how vari-
ous chemicals affect the growth of Arabidopsis protoplast cultures,
and while that particular study didn’t work out as hoped, Murphy
and Kangas think machines could one day steer such plant experi-
ments, and perhaps similar experiments in animals. Alternatively,
a machine learning program could produce a readout of instruc-
tions for, say, mouse experiments it needs to refine its model, and
a researcher could perform the experiments and feed the data back
into the program. “The methods that we’re working on are general-
izable, and we try to find collaborations that will test the parts of the
methods. . . that we think might need the most work,” Murphy says.
Using images along with machine learning, Murphy thinks
biology will advance toward a much more detailed structural
understanding of spatial relationships within cells—where a given
protein will be found in relationship to the cell membrane, for
example, or to microtubules. Having that foundation will make
it feasible to investigate how perturbations such as mutations
or drug candidates change those orientations. Building com-
prehensive computational models of cells’ spatial relationships
will require the work of many labs, Murphy says, and he thinks
the role of his lab is to “develop tools that will enable that, and
describe a way in which we think that this task could be done.”
In one recent step in that direction, he and a graduate student
worked with Seema Lakdawala of the University of Pittsburgh
School of Medicine to use images to train a model of the spatial
relationships of different segments of influenza RNA to predict
how they likely come together within an infected cell to produce
new infectious particles.
Ultimately, Murphy says, “We really believe that these self-driving
instruments are going to change the way science is done.” g
 SCIENTIST TO WATCH
Nick Turk-Browne: Pattern Seeker
Professor, Department of Psychology, Yale University, Age: 37
IBY CATHERINE OFFORD
W
hen Nick Turk-Browne was
a teenager, he read V.S.
Ramachandran’s Phantoms
in the Brain, a gift from his father, and was
fascinated by the book’s message that “our
experience of the world is constructed by
our brain,” he says. “It blew my mind. It got
me very much interested in this question of
how do we experience the world, and how
does our brain construct that experience?”
This fascination has been a guiding
force for Turk-Browne, now a professor of
cognitive neuroscience at Yale University.
After graduating from the University
of Toronto in 2004, he moved to Yale’s
psychology department for a PhD, and “was
a superstar from the beginning,” recalls
psychologist Marvin Chun, one of Turk-
Browne’s advisors. “He absolutely is one
of the most productive and creative PhD
students I’ve mentored in my entire career.”
In grad school, Turk-Browne focused
on how the brain orchestrates statistical
learning. Unlike episodic memory, which
allows conscious recall of snippets of
information such as phone numbers or
birthdays, statistical learning involves the
more subconscious association of events or
objects, such as remembering a book with
the table it’s placed on, or a person with
the building they’re usually seen in. “It’s
about extracting patterns, or regularities,
across experiences,” explains Turk-Browne.
This type of learning is critical “to generate
expectations about the world, to interact,
to behave in an efficient manner when
confronted with new situations.”
Using behavioral experiments and
© JANE SHAUCK PHOTOGRAPHY
functional MRI (fMRI) brain scans, Turk-
Browne found that the hippocampus, an
area usually linked to episodic memory,
unexpectedly also showed activity
associated with statistical learning, even
when people weren’t paying attention
to learning tasks.1 “The fact that the
hippocampus was engaged in this relatively
unconscious process of statistical learning REFERENCES
was very surprising.” 1. N.B. Turk-Browne et al., “Neural evidence
In 2009, straight out of grad school, of statistical learning: Efficient detection of
Turk-Browne started his own lab at Princeton visual regularities without awareness,” J Cogn
University. “We’d never seen someone Neurosci, 21:1934–45, 2009. (Cited 315 times)
who had been that productive, not just in 2. A.C. Schapiro et al., “Shaping of object
terms of number of papers but in terms of representations in the human medial
the substance, as a graduate student,” says temporal lobe based on temporal
Princeton psychologist Ken Norman. “It was regularities,” Curr Biol, 22:P1622–27, 2012.
clear that he was ready to leap right into a (Cited 220 times)
faculty position.” 3. A.C. Schapiro et al., “Complementary learning
A few years in, Turk-Browne and systems within the hippocampus: A neural
grad student Anna Schapiro found network modelling approach to reconciling
that, during statistical learning, the episodic memory with statistical learning,”
hippocampus and nearby brain regions Philos Trans R Soc B, 372:20160049, 2017.
represent objects based on how they’re (Cited 62 times)
temporally associated with one another—
whether they’re seen together, for
example—rather than on function
or appearance— say, associating
books with other books.2 “We think
that this is a basic mechanism for
how the hippocampus is able to
store these sorts of regularities,”
Turk-Browne says.
Using neural network models
of the hippocampus, Turk-Browne,
Schapiro, and Norman uncovered
evidence for distinct anatomical
pathways for episodic and statistical
learning.3 Turk-Browne has also
worked on novel tools such as
BrainIAK, machine learning
software developed with Intel
that runs advanced analyses
of fMRI data. In 2017, he
returned to Yale as a professor
and, as expected, he’s an
“outstanding teacher and
departmental citizen,” says
Chun, who adds that he
frequently finds himself
learning from his former
student. “It’s a real
thrill to have him
back.” g
05 . 2019 | T H E S C IE N T IST 51
 LAB TOOLS
Automating the Fight Against Superbugs
Antibiotic resistance is on the rise. Can AI help?
BY AMBER DANCE
S
hould you be so unlucky as to wind
up in the hospital with a drug-
resistant bacterial infection, doc-
tors will need to figure out which antimi-
crobial drug has the best chance of killing
your particular pathogen. With antibiotic
resistance on the rise—and predicted to
kill 10 million people per year by 2050—
it’s not always an easy choice.
It would help clinicians to be able to
mine your superbug’s genome for DNA
sequences that indicate susceptibility or
resistance to antibiotics. As a step toward
that goal, bioinformaticians are tap-
ping artificial intelligence to identify the
52 T H E SC I EN TIST | the-scientist.com
most relevant sequences. They’re mak-
ing progress, thanks to databases stuffed
with thousands of genomes from differ-
ent strains of pathogenic bacteria, along
with corresponding data on whether
those strains were susceptible or resis-
tant to dozens of antibiotics.
Some researchers are training
machine learning algorithms to iden-
tify known drug resistance genes in
new strains of a pathogen. Others are
using AI to hunt for entirely new resis-
tance genes, seeking a better under-
standing of how bacteria fight off drug
treatments. And some are moving into
metagenomes, aiming to understand the
resistance profile of environments such
as wastewater effluent.
Challenges remain before AI is able
to prescribe your antibiotics, though, says
James Davis, a computational biologist at
Argonne National Laboratory. For one, fast,
point-of-care sequencing remains expen-
© ISTOCK.COM, LUISMMOLINA
sive—and less accurate than slower con-
ventional methods. For another, databases
are often skewed toward resistant strains,
because hospitals sequence the most dif-
ficult cases, but including genomes from
antibiotic-susceptible strains would help
the algorithms perform better, he says.
 Here, The Scientist profiles three
recent studies applying machine learning
to the antibiotic resistance problem.
DRUGS & DOSES
What drugs have the best shot at cur-
ing an infection? Some scientists have
relied on known antimicrobial resis-
tance genes and proteins to match bac-
terial strains with drugs that are most
likely to kill them. Davis says AI can do
better, by analyzing entire genomes for
both known and potentially unknown
genes related to drug resistance or sus-
ceptibility. He and his team developed
a machine learning approach to identify
key differences between resistant and
susceptible strains, and thus predict the
drug-response profile of novel strains.
The algorithm may also help scientists
identify novel resistance genes.
The researchers recently tested their
approach on Salmonella, a top cause
of food poisoning (J Clin Microbiol,
57:e01260–18, 2019). Though the infec-
tion usually isn’t severe, strains resistant
to antibiotics can make people sicker.
Bacteria from a patient
DNA
sequences
THE SCIENTIST STAFF
Wastewater or other
environmental sample
INPUT
The researchers used 5,278 Salmonella
genomes from the US Food and Drug Admin-
istration’s National Antimicrobial Resis-
tance Monitoring System, along with so-
called minimum inhibitory concentrations,
or MICs, for 15 antibiotics—that is, the lowest
amount needed to block growth of each strain
in the lab. All the bacteria had been isolated
from raw meat and poultry for sale or from
livestock being slaughtered for food.
The researchers used a program
called the K-mer Counter (KMC) to
divide each of those genomes into over-
lapping 10-mers. For example, if a
hypothetical sequence started with
AAAAAGGGGGTTTTTCCCCC, the
first 10-mers would be AAAAAGGGGG,
AAAAGGGGGT, AAAGGGGGT T,
and so on, starting one base farther
along each time. Then the computer
counted how many times a given 10-mer
appeared in each genome: the number
of AAAAAGGGGGs, AAAAGGGGGTs,
AAAGGGGGTTs, and so forth. These
were the features fed into the machine
learning algorithm, along with MIC data,
to train it to predict MICs on its own.
Ideal prescription
AI to identify
resistance genes
Sequences of novel
resistance genes
MACHINE LEARNING
The team applied a machine learn-
ing algorithm called extreme gradient
boosting (XGBoost). Using those 10-mer
counts, the computer designs decision
trees to predict the right MICs. Each
decision point uses one of the 10-mers to
help it classify a given genome as resis-
tant or susceptible to various drugs. The
algorithm then assigns different lev-
els of importance to each 10-mer, and
designs trees repeatedly, in rounds called
“boosts,” until it gets the lowest error it
can for its MIC predictions compared to
the true MICs. The researchers ran the
algorithm 10 times, each time leaving out
a different tenth of their dataset. They’d
train the computer on the other 90 per-
cent of the data, then use the remaining
ten percent to test its accuracy.
OUTPUT
When given an entirely new genome, the
program predicts which drugs the strain
will be resistant or susceptible to, along
with the relevant dose. In the team’s test,
with strains in the reserved 10 percent,
the algorithm was 95 percent accurate.
By rerunning their tests with 15-mers
for each pathogen genome, and consid-
ering each antibiotic individually, the
researchers identified DNA snippets asso-
ciated with resistance or susceptibility to
each drug. Comparing those 15-mers to
the Salmonella sequence, the research-
ers started to figure out which genes
were most important in making these
predictions. In fact, many of the genes the
algorithm chose corresponded to known
drug resistance genes, indicating that the
algorithm was on the right track. But not
all pointed to well-understood resistance
genes, suggesting the AI might be pick-
ing up on genetic features as yet unknown
to scientists that are also associated with
resistance. “There’s biology there that’s
worth studying,” says Davis.
INTELLIGENT DESIGN: Researchers are
using artificial intelligence to explore the DNA
sequences of bacteria that infect people and
contaminate the environment in order to iden-
tify known and new drug resistance genes.
05 . 2019 | T H E S C IE N T IST 53
 LAB TOOLS
PRO
• The machine learning algorithm is not
biased by a list of known resistance
genes, or even protein-coding genes,
allowing it to identify new genetic fac-
tors potentially involved in resistance
across the entire genome.
CON
• The machine identifies 10- and 15-mers
associated with drug responses, but
it’s not immediately clear which genes
are relevant, or whether an individual
sequence promotes resistance or sus-
ceptibility. Davis adds that it is usually
possible to infer this information once
he compares the 10- or 15-mers to the
bacterial sequence.
TRY IT
Download the code at https://github.com/
PATRIC3/mic_prediction.
GENE PROSPECTING
Researchers studying microbial resis-
tance have generally focused on gene
5 4 T H E SC I EN TIST | the-scientist.com
products that directly interact with the
drug in question. But other kinds of
genes—for example, genes that affect
the permeability of the bacterial cell
wall, or how the cell pumps out toxins
and waste—might also influence sus-
ceptibility to antimicrobials.
Erol Kavvas, a bioengineering grad-
uate student at the University of Cal-
ifornia, San Diego, hunted for novel
resistance genes in the genome of Myco-
bacterium tuberculosis (Nat Commun,
9:4306, 2018). This bacterium infects
some 10 million people worldwide each
year, and more than 500,000 of these
infections are resistant to commonly-
prescribed antibiotics. “There’s a lot of
complexity to drug resistance in TB,”
Kavvas says.
INPUT
The researchers used 1,595 M. tuberculosis
genomes from the Pathosystems Resource
Integration Center (PATRIC) database,
plus whether each genome was from a
strain resistant or susceptible to 13 differ-
ent antibiotics.
First, the researchers determined
the pangenome—the full list of every
possible protein-coding gene—from all
the M. tuberculosis strains in their data-
set. Based on this list, they identified
all possible alleles that could potentially
be present in a given TB genome. Then,
they noted whether the genome of each
individual strain possessed each allele
or not. Together with the resistance
data, these yes-or-no allele lists created
a multidimensional matrix.
MACHINE LEARNING
Kavvas applied an approach called sup-
port vector machine, or SVM. The algo-
rithm is designed to group similar data
and draw boundaries between the groups.
For example, for a simple, two-dimen-
sional input matrix with just two kinds
of variables, it might draw lines between
groups. For the multidimensional matrix
Kavvas created, it draws a multidimen-
sional divider, called a hyperplane,
between resistant and susceptible strains.
To identify the most important genes
for resistance, Kavvas also applied a tech-
nique called the L1-norm. Simply put, he
told the computer to use a small number
of genes to draw the boundary.
OUTPUT
The algorithm provides a list of genetic
mutations involved in resistance to each
drug, ranked by order of importance.
Overall, Kavvas identified 33 known drug
resistance genes; this information could
help doctors choose the right drug for a
patient with TB.
He also found 24 novel resistance
genes, many of which are involved in
metabolism and cell wall processes.
He hopes experimental biologists will
pick up on his results and work out how
those genes help neutralize antibiotics.
© ISTOCK.COM, GAETAN STOFFEL
PRO
• Many models are biased by using a
standard reference genome, which
may or may not represent the most
common strains in circulation. By
using the pangenome instead, the
team avoided this bias.

CON
• So far, Kavvas has only included vari-
ants in protein-coding genes, so he
could miss relevant non-protein-coding
elements, such as genes for regulatory
RNAs, in other parts of the genome.
TRY IT
Download the code at https://github.com/
erolkavvas/microbial_AMR_ML/.
GET META, GO DEEP
Microbes pick up new drug resistance
genes from other bacteria, swapping
them like trading cards. The swap meet
goes down in places where microbes mix,
© SHUTTERSTOCK.COM, URFIN
such as wastewater from hospitals or
farms where antibiotic use is high. Even
after water treatment, traces of resistance-
related DNA remain.
To assess the risk in water samples,
researchers often compile metagenomes—
that is, all the DNA within a microbial
community—then look for known, indi-
vidual antibiotic resistance genes that are
homologous to sequences in their sample.
But making those comparisons requires
defining a threshold of similarity—say,
50-90 percent—that counts as close
enough to call a bit of DNA a resistance
gene. Researchers often set high, stringent
thresholds, resulting in a high rate of false
negatives, says Liqing Zhang, a bioinfor-
matician at Virginia Tech. That is, many
true resistance genes are overlooked.
Zhang and colleagues developed a new
tool to assess the resistance genes in envi-
ronmental samples. Called DeepARG (for
antibiotic resistance genes), it compares the
environmental DNA to all known resistance
genes, one at a time, instead of to a single,
most-homologous gene (Microbiome, 6:23,
2018). That helps because it focuses the
comparison on broad categories of resis-
tance genes and what they have in common,
so the algorithm can identify novel genes
that share those common features.
INPUT
First, the researchers built a database
of known resistance genes and which of
30 different drugs they affect, collected
from three sources: the Comprehensive
Antibiotic Resistance Database (CARD),
the Antibiotic Resistance Genes Data-
base (ARDB), and the Universal Protein
Resource (UNIPROT). They call the data-
base DeepARG-DB.
They then used 70 percent of the
10,602 genes from UNIPROT to train the
machine learning algorithm. To develop
the input data, they had the computer
compare the sequence of each gene from
UNIPROT, individually, to the known
resistance genes from the other two data-
bases. The result was a list of thousands of
similarity scores for each UNIPROT gene.
MACHINE LEARNING
Zhang’s group used a deep learning
model. These types of algorithms are
inspired by how the human brain is
thought to work, and they assign differ-
ent weights to inputs to come up with
the most accurate output. During the
training, the computer figured out how
to weight those similarity scores to make
the best predictions of antibiotic resis-
tance category for each UNIPROT gene.
The researchers built two differ-
ent models for different kinds of DNA
sequences. DeepARG-SS works for short
reads, of 100 base pairs or so, like the
reads one typically gets from metage-
nomic sequences. DeepARG-LS works
with longer, gene-length reads.
OUTPUT
When tested with the remaining 30 per-
cent of UNIPROT sequences it hasn’t
been trained on, Zhang’s algorithm gen-
erates the probability that each sequence
reflects a gene for resistance to each of
the 30 categories of antibiotics. It was
able to identify antibiotic resistance
genes with low rates of both false nega-
tives and false positives.
DeepARG’s predictions match well
with other reports. The researchers
compared their results to a recently
published list of 76 new antibiotic resis-
tance genes (Microbiome, 5:134, 2017).
“Sure enough, yes, we predicted 65 of
them,” says Zhang.
Her collaborators can now apply Deep-
ARG to assess wastewater and other envi-
ronmental samples. For example, they can
test how wastewater treatment alters the
profile of resistance genes.
PRO
• DeepARG does not require strict cut-
offs to identify genes as related to drug
resistance, so it provides fewer false neg-
atives than standard comparisons.
CON
• The database only considers entire
genes as resistance-related or not; it
lacks the resolution to identify single
nucleotide polymorphisms associated
with resistance, or mutations that indi-
rectly influence resistance pathways.
TRY IT
Download DeepARG-DB, or compare
your sequences to it, at https://bench.
cs.vt.edu/deeparg; or download the code
at https://bitbucket.org/gusphdproj/
deeparg-ss.
05 . 2019 | T H E S C IE N T IST 5 5
 BIO BUSINESS

Computing a Cure
The pharmaceutical industry is looking to machine learning to
overcome complex challenges in drug development.

BY BIANCA NOGRADY

A
ustralia’s government drug
safety watchdog sounded the
alarm about the oral antifungal
agent terbinafine in 1996. The drug,
sold under the brand name Lamisil by
pharma giant Novartis, had come onto
the market in 1993 for the treatment of
fungal skin infections and thrush. But
three years later, the agency had received
a number of reports of suspected adverse
reactions, including 11 reports of liver
toxicity. By 2008, three deaths from liver
failure and 70 liver reactions had been
pinned on oral terbinafine.
Researchers in Canada identified
the biochemical culprit behind terbin-
afine’s liver toxicity—a compound called
TBF-A that appeared to be a metabolite
of terbinafine—in 2001. Clinicians
quickly learned to monitor and manage
this potential toxicity during treatment,
but no one could work out how the
compound actually formed in the liver,
or could experimentally reproduce its
synthesis from terbinafine in the lab.
Then, in 2018, graduate student Na
Le Dang at Washington University in
St. Louis hit upon a way to use artificial
intelligence (AI)—specifically, a machine
learning algorithm—to work out the pos-

MODIFIED FROM ©ISTOCK.COM, THE SCIENTIST STAFF

sible biochemical pathways terbinafine
takes when it is metabolized by the liver.
Trained on large numbers of known
metabolic pathways, the algorithm had
learned the most likely outcomes when
different types of molecules were broken
down in the organ. With that information,
it was able to identify what no human
could: that the metabolism of terbinafine
to TBF-A was a two-step process.
Two-step metabolites are much more
difficult than direct metabolites to detect
experimentally, which is likely why
this potentially lethal outcome wasn’t
flagged until after the product was on
the market, says S. Joshua Swamidass,
a physician scientist and computational
biologist at Washington University and
Dang’s supervisor. The discovery not
only shed light on a long-standing bio-
chemical mystery, but showed how the
use of AI could more broadly aid drug
discovery and development.
Given enough data, machine learn-
ing algorithms can identify patterns,
and then use those patterns to make
predictions or classify new data much
faster than any human. “A lot of the
questions that are really facing drug
development teams are no longer the
sorts of questions that people think
that they can handle from just sorting
through data in their heads,” Swamidass
says. “There’s got to be some sort of sys-
tematic way of looking at large amounts
of data . . . to answer questions and to
get insight into how to do things.”
AI is well positioned to handle
the complexity of the rules that must
be applied to understand these data,

5 6 T H E SC I EN TIST | the-scientist.com
too, says Regina Barzilay, a computer
scientist at MIT and a scientific advisor
for drugmaker Janssen, a subsidiary of
Johnson & Johnson. “When we study
chemistry, we definitely study a lot of
rules and we understand the mechanism,
but sometimes they’re really, really com-
plex,” she says. “If [a] machine is pro-
vided with a lot of data, and the problem
is formulated correctly, it has a chance to
capture patterns which humans may not
be able to capture.”
Machine learning couldn’t have
matured at a better time for the pharma-
ceutical industry, says Shahar Keinan,
cofounder and chief scientific officer
at Cloud Pharmaceuticals, a company
focusing on AI–based drug discovery.
She argues that the steady decline in
the number of new drug targets, new
mechanisms, and novel first-in-class
drugs coming to market each year is an
indication that the current system of
drug discovery is insufficient to meet
modern challenges. “Right now, you
need to do more work to get to those
kind of first-in-class [drugs],” she says.
“The way to overcome that . . . is to find
something new in what [data] we have,
and this is where artificial intelligence
will come in.”
The pharmaceutical industry and its
investors seem to agree. Last year Cloud
Pharmaceuticals entered into a drug
discovery collaboration with pharma
giant GlaxoSmithKline. And UK-based
BenevolentAI, which describes its mis-
sion as “accelerating the journey from
data to medicine” in areas ranging from
drug discovery to clinical development,
recently earned an eyebrow-raising
multibillion-dollar valuation.
Designing better drugs
More than 450 medicines worldwide
have been withdrawn from the market
post-approval in the last half century as
a result of adverse reactions, with liver
toxicity the most common side effect.
But the metabolism of compounds by
organs such as the liver is extremely
complex and, as in the case of terbin-
afine, difficult to anticipate.
This is exactly the sort of problem
that machine learning can help solve—
and the data are already available to
help that process. For example, the US
federal government’s Tox21 program, a
collaboration among the Environmen-
tal Protection Agency, the National
Institutes of Health, and the Food and
It’s really our ambition to automate as much as possible, so
the chemists are just focusing on the much higher level, the
difficult problems, the strategy.
Drug Administration, maintains a large
data set of molecules and their toxicity
against key human proteins—perfect
fodder for AI to digest in search of pat-
terns of association between structure,
properties, function, and possible toxic
effects, Keinan says.
Cloud Pharmaceuticals is one
company that makes use of these data
as part of their workflow. “Now you
can train a machine learning algorithm
on this data set, then a new molecule
comes along, and you can just use your
prediction to say, ‘Is this molecule toxic
or not?’” says Keinan.
As well as identifying potential
toxicities, machine learning algorithms
could predict how a candidate molecule
will respond to different physical and
chemical environments, and so help
drug developers understand how that
molecule might behave in various tis-
sues in the human body.
A group led by physical chemist Scott
Hopkins of the University of Waterloo,
in collaboration with researchers at
Pfizer, has been training an algorithm
to do just that, using data on 89 small-
molecule drug candidates obtained by
performing a type of spectrometry that
measures how quickly molecules absorb
or lose water.
“If our drug molecule absorbs a lot
of water very quickly and doesn’t give it
up, it tells us that that drug is going to be
very soluble in water,” Hopkins says. “It’s
going to dissolve easily in your stomach
and enter your bloodstream fairly
quickly.” After the algorithm learned the
associations between certain molecular
structures and solubility from those
89 molecules, it was able to accurately
predict the key properties of a similar
molecule, the team reported at the end
of last year (Nat Commun, 9:5096).
—Adrian Schreyer, Exscientia
Although screening for potential
toxicities and for biochemical properties
are essential sub-tasks in the process of
drug development, a more tantalizing
question is whether AI could suggest
the structure of a new therapeutic
molecule from scratch.
At Maryland-based biotech Insilico
Medicine, CEO Alexander Zhavoronkov
and colleagues are using a type of algo-
rithm called a generative adversarial
network to help develop entirely novel
small-molecule organic compounds
aimed at treating everything from
cancer to metabolic disease to neuro-
degenerative disorders. This algorithm
consists of two deep neural networks
pitted against one another, Zhavor-
onkov explains.
The first deep neural network has the
challenge of coming up with outputs—
molecular structures—in response to
a series of inputs, namely the desired
functional and biochemical charac-
teristics of those structures, such as
solubility, targets, or bioavailability. The
other deep neural network has the job of
critiquing those outputs. “They engage
in this adversarial game,” Zhavoronkov
says. “They basically kind of compete
with each other . . . and in this process,
after many, many iterations, they learn
to generate something new.”
Insilico Medicine recently synthesized
its first molecule from this process in part-
nership with China-based pharmaceutical
company Wuxi AppTec. The two compa-
05 . 2019 | T H E S C IE N T IST 57
BIO BUSINESS

“It’s really our ambition to automate as investing up to $67 million in a drug-

nies have now entered into a collaboration
agreement to test other candidate mol-
ecules designed by the adversarial network
approach for multiple orphan drug targets.
Other companies, such as Exscientia,
a UK-based, AI-driven drug discovery
company, are bringing machine learning
much as possible, [so] the chemists are
just focusing on the much higher level,
the difficult problems, the strategy,” says
Adrian Schreyer, chief technology officer
at the company. This means the biologi-
cal targets and the molecules designed to
bind them will all be influenced by the
outputs of their AI platform.
But there is still a human hand on the
tiller. The algorithm “is fairly autonomous
in terms of generating compounds, [but]
the final selection is done by chemists,”
Schreyer says. This is partly because the
patent system demands a human name
to an invention, but also because the
selections that the chemists make from
the AI-generated options can be fed back
into the algorithm to fine-tune its deci-
sion-making protocol for the next time.
“If you see twenty to thirty compounds
and [humans] pick ten . . . by virtue of
the decision making you can see what’s
discovery collaboration with the com-
pany. Another recent financing round
has earned the company $26 million in
investment from companies including
Celgene Corporation and GT Health-
care Capital Partners.
The machine’s limit
Despite progress in using AI for drug
development, it’s not time for humans to
step out of the process just yet, says Bar-
zilay. For now, she explains, AI in pharma
is analogous to a smart kitchen: “You
have a microwave, and you have a coffee
machine, and you have this and that, but
none of it actually cooks you the dinner. .
. . You need to put it all together to make
a dinner, even though all these things can
help you to do it faster and better.”
While Barzilay and colleagues are
part of a pharma-funded research col-
laboration with MIT to bring AI into drug

MODIFIED FROM ©ISTOCK.COM, THE SCIENTIST STAFF

discovery, she points out that machine
learning has not yet brought a molecule
If a machine is provided with a lot of data, and the problem is to market. “It just assists humans to do a
formulated correctly, it has a chance to capture patterns which faster and better job in various sub-tasks,
humans may not be able to capture. but we still don’t have AI coming up with
—Regina Barzilay, MIT new drugs at this point,” she says.
Artificial intelligence is also limited
by the quality of the inputs—in par-
to bear on the entire process of drug dis- preferable, you can use that process itself ticular, the quality of the data that it
covery. Exscientia is building a platform to build a machine learning model.” learns from, says Schreyer. “If the data
that aims to mimic the decision making Exscientia’s approach has gener- is flawed, then the results are likely to be
of a medicinal chemist while also learning ated considerable interest from the flawed as well, which can be a problem
from human medicinal chemists’ input. pharmaceutical industry, with Roche if you get data from third parties,” he

HUMAN VERSUS MACHINE

Machine learning has the capacity to analyze vast quantities of biochemical data, and do so much faster and more accurately than a
human brain. But just how much better than a person can an algorithm really perform?
MIT computer scientist Regina Barzilay and her colleagues have attempted to answer this question by pitting a machine learning
algorithm against ten human chemists challenged with predicting the outcomes of organic reactions.
Participants and the algorithm were given 80 sample organic reactions and asked what the most likely product of each of those
reactions would be. The algorithm had been trained on a database of organic reactions from the US Patent and Trademark Office, while
eight of the ten humans were graduate, postdoctoral, and professor-level chemists with considerable experience in organic chemistry,
and the other two were graduate students in chemical engineering (ACS Cent Sci, 3:434–43, 2017).
The algorithm was, on average, more accurate than the chemists: the scientists’ average accuracy was 48.2 percent, while the algo-
rithm scored 69.1 percent. In fact, there was just one person who prevented artificial intelligence from claiming a total victory, Barzilay
says. “The only human who actually outperformed the machine, and not by a big margin, was the head of our chemistry department.”
 says. Experimental data are not perfect
measurements of the real world, and
there are always assumptions and fitting
involved. If the algorithm and its users
One tantalizing question is whether AI could suggest the
structure of a new therapeutic molecule from scratch.
do not sufficiently take those biases
and weightings into account, then the
outputs will also be biased.
Sometimes data can be lacking
altogether, particularly for recently
discovered molecules. At Cloud Phar-
maceuticals, researchers are getting
around this problem by using compu-
tational chemistry to design a range
of molecules that could fit a particular
biological target, then using that range
as the data set that the machine learn-
ing algorithm can learn from to help
design new candidate molecules.
Advancing Diagnostics Together
AUGUST 20–22, 2019
WASHINGTON, DC
Grand Hyatt Washington
Register by April 26, 2019 and SAVE UP TO $400!
Another issue for some researchers
is the so-called “black box” nature of
the algorithms themselves: data goes in,
answers come out, but the internal work-
ings remain a mystery. Swamidass says
that this presents a fundamental conflict
for scientists. “What does it mean to be
applying black-box methods in fields
that are all about understanding what’s
inside of a black box?” he says. “We don’t
want to just have something that works
in science, we want to understand the
underlying processes there.”
But Barzilay says that the relation-
ships between inputs and outputs
might often go be yond human
comprehension. “When we’re talk-
ing about complex organic reactions
11th Annual
with different solvents and different
temperatures and other things, then
it becomes really difficult even for the
experienced human to predict what is
the right outcome, so what does it mean
[to] give an explanation?”
There is unquestionably a lot of hype
around the potential of AI in drug dis-
covery—Swamidass likens this period
to the internet boom of the late 1990s.
But many, such as Schreyer, are excited
about the possibilities that this new
technology offers, particularly when it
comes to finding novel therapeutics for
difficult-to-treat diseases.
“If you can improve . . . how efficiently
you do drug discovery fivefold, tenfold,
or even more,” he says, “then from an
economic perspective . . . you’re able to
take on more risky projects because the
cost of failure is much lower.” g
Bianca Nogrady is a freelance science
writer based in Sydney, Australia.
800+
Diagnostics
Professionals
58%
of Attendees from
IVD & Pharma
60+
Industry Leading
Sponsors &
Exhibitors
42%
C-Suite &
Directors
NextGenerationDx.com
 READING FRAMES
Robots and Eureka Moments
Artificial intelligence can be trained to recognize patterns and predict results.
But will machine learning ever be able to make novel scientific discoveries?
BY KARTIK HOSANAGAR
M
ost of the practical AI success sto-
ries in recent years have involved
what computer scientists call super-
vised machine learning: the use of labeled
datasets to train algorithms to automate
what had been a human activity. For exam-
ple, take a dataset of symptoms and test
results of thousands of patients, along with
their eventual diagnosis by doctors, and train
an algorithm to learn the patterns in the
dataset—that is, which symptoms and clini-
cal markers predict which diseases. Similarly,
take a dataset of labeled images and train an
algorithm to recognize people’s faces. These
successes show that machine learning can,
with the right training data, approximate
tacit human knowledge. But is it possible for
AI to extract knowledge unknown even to
experts? Can we automate something like
scientific discovery?
One potential approach, which I discuss
in my recently published book A Human’s
Guide to Machine Intelligence, came from
the late Don Swanson, an information sci-
entist at the University of Chicago. Swanson
was reading about the Inuit diet when one
detail of it—high fish consumption—caught
his attention. The research suggested that
elevated fish oil intake increases blood flow,
reduces blood vessels’ reactions to cold, and
dampens platelet-triggered clotting. The
opposite of these changes in the blood sys-
tem, Swanson happened to know, were all
associated with Raynaud’s disease, a syn-
drome that causes blood vessels to constrict
in response to low temperatures or stress.
Fish oil, Swanson hypothesized, might help
treat Raynaud’s disease.
Swanson found many studies that con-
firmed the observations that 1) fish oil
improved blood circulation and 2) Raynaud’s
disease was associated with poor blood circu-
lation. But none of the existing research sug-
gested that fish oil could be an effective treat-
ment for Raynaud’s. In 1986, Swanson wrote
6 0 T H E SC I EN TIST | the-scientist.com
a research paper proposing the hypothesis.
In 1989, a clinical study conducted in the
rheumatology clinic at Albany Medical Col-
lege confirmed Swanson’s hypothesis.
Swanson’s main insight was that new
knowledge could be uncovered by con-
necting disparate fields of knowledge: If A
(fish oil) was related to B (blood flow) and
B was related to C (Raynaud’s symptoms),
then there might be a potential relation-
ship between A and C.
Swanson and University of Illinois at
Chicago psychiatry professor Neil Smal-
heiser developed a computer program
called Arrowsmith that plucked out such
hypotheses from medical research data-
bases, with a focus on theories generated
out of links between disparate specialties.
Swanson later hypothesized a relation-
ship between magnesium deficiency and
migraine headaches that was also supported
by subsequent clinical research.
Over the years, Arrowsmith has had lim-
ited effect, but Swanson’s early foray sug-
gests that finding relationships in data from
disparate fields can help tap into undiscov-
ered knowledge hidden in data. Although
Swanson’s efforts were fully manual, such
a process can indeed be automated to help
uncover knowledge that scientists might not
have discovered yet.
An alternative approach is illustrated
by Google Deep Mind’s Go-playing soft-
ware AlphaGo Zero. While the original
version of the software was trained heavily
on past games played by human Go play-
ers, AlphaGo Zero didn’t bother studying
human moves; instead, its entire train-
ing dataset was self-generated. The soft-
ware, armed with basic rules of Go, played
millions of games against itself. Next, it
analyzed those games to figure out which
moves helped and which ones hurt.
While supervised learning relies on
cleanly labeled training data, AlphaGo Zero
Viking, March 2019
learned from data generated by an algo-
rithm itself via exploration, an approach
known as reinforcement learning. Such
algorithms explore different actions and
learn which actions lead to a better per-
formance. Instead of being restricted to
analyzing the data already obtained, the
approach can explore the space of poten-
tial actions and prioritize what to test
next. This ability to take stock of multiple
hypotheses and explore them (that is, con-
duct experiments and acquire data to vali-
date hypotheses), all while recognizing the
cost of exploration, can be a big boost to
scientific discovery. For example, drug dis-
covery relies on coming up with millions of
candidate molecules and running a series
of experiments to identify if some molecule
seems to work.
While AI is automating routine tasks in
a variety of industries, there is great promise
for its application in science as well. g
Kartik Hosanagar is the John C. Hower
Professor at the Wharton School of the
University of Pennsylvania, where he
studies technology and the digital economy.
Read an excerpt of A Human’s Guide to
Machine Intelligence at the-scientist.com.
FOLLOW US ON INSTAGRAM
the_scientist_magazine
The Guide
Continuous Measurements
of Cell Monolayer Barrier
Function (TEER) in
Multiple Wells.
The TEER 24 System electrically monitors
the barrier function of cells grown in culture
upon permeable membrane substrates. The new TEER 24
system accommodates standard 24 well membrane inserts in
a disposable 24 well microplate. The instrument is placed in a
standard CO2, high humidity incubator and connected to a PC.
Dedicated software presents real-time continuous measurement
of TEER in ohm-cm2.
Technical. Guide. Free.
Update.
Worthington educational materials have
guided generations of life science research
for decades.
And thanks to you, we are cited in respected
scientific journals more than any other
primary enzyme producer across the globe.
Order your NEW Tissue Dissociation
Guide today. Call: 1.800.445.9603, Fax:
1.800.368.3108,
or go to: worthington-biochem.com
Expertise for a new generation of researchers.

For more information visit WORTHINGTON BIOCHEMICAL CORPORATION

APPLIED BIOPHYSICS 1.800.445.9603
www.biophysics.com/products-ecisTeer24.php worthington-biochem.com

Insect Medium Hepatocytes

IS Sf DonorPlexTM
• Serum-free, animal component-free, and low • High-quality cryopreserved pooled
hydrolysate donor suspension hepatocytes are
• Formulated for scalability for the consistent now available
growth and yields of proteins, viral vectors, and • Designed to offer a reliable, robust,
viral-like particles using baculovirus expression and customizable tool for studying the
systems (BEVS), in Sf9 and Sf21 cells hepatic metabolism of drugs
• Supports high, viable cell density growth and protein yields in • Proprietary algorithm that predicts yield, viability, and functional
baculovirus expression systems at small- and large-scale phenotype prior to manufacture ensures individual customer
specifications are met

FUJIFILM Irvine Scientific Lonza

www.irvinesci.com www.lonza.com

Microbiological Incuba- Flow Cytometer

tors BIT/BIF Series ZE5
• Added to the company’s ambr® 15 • Offers a unique option for histology
fermentation and ambr® 250 high laboratories
throughput systems for microbial • Can be placed on any benchtop as a plug-
applications and-play product
• Addresses the bottleneck associated • Combines a quiet fan and a Peltier design
with offline sampling optimized for robust performance
• Features state-of-the-art non-invasive reflectance measurement, saving
time, effort, and sample volume

Sartorius Stedim Biotech TECA

www.sartorius-stedim.com www.thermoelectric.com

6 2 T H E SC I EN TIST | the-scientist.com
BioProcess
International
September 9-12, 2019
Boston Convention & Exhibition Center
Boston MA

Part of

THE LARGEST BIOPROCESSING EVENT BRINGING

YOU THE SCIENCE, TECHNOLOGIES AND PARTNERS
NEEDED TO ACCELERATE PROMISING BIOLOGICS
TOWARDS COMMERCIAL SUCCESS

View the Full Speaker List Today at

www.BPIevent.com
 FOUNDATIONS

Learning Machine, 1951

BY JEF AKST

A
s an undergraduate at Harvard
in the late 1940s and in his first
year of grad school at Princeton
in 1950, Marvin Minsky pondered how
to build a machine that could learn.
At both universities, Minsky studied
mathematics, but he was curious about
the human mind—what he saw as the
most profound mystery in science. He
wanted to better understand intelligence
by recreating it.
In the summer of 1951, he got his
chance. George Miller, an up-and-coming
psychologist at Harvard, secured funding
for Minsky to return to Boston for the
summer and build his device. Minsky
enlisted the help of fellow Princeton
graduate student Dean Edmonds, and the
duo crafted what would become known as ARTIFICIAL INTELLIGENCE TAKES BABY STEPS: The SNARC machine included 40 artificial neurons (one
pictured), which were interconnected via a plugboard and held in racks in a contraption about the size of a grand
the first artificial neural network. Called
piano. At one end of the neuron was a potentiometer (bar on far right), a sort of volume knob that could adjust
SNARC, for stochastic neural-analog the probability that an incoming signal would result in an outgoing signal. If the neuron did fire, a capacitor
reinforcement calculator, the network (red) on the other end of the neuron retained a memory of the firing for a few seconds. If the system was
included 40 interconnected artificial “rewarded”—either by the researchers pushing a button or an electrical signal from an outside circuit—a chain
neurons, each of which had short-term connected to the volume knobs for all 40 neurons would crank. This would cause the volume knob to increase
the future probability of the neuron firing, but only if a magnetic clutch had been engaged by a recent firing.
and long-term memory of sorts. The
short-term memory came in the form of a
capacitor, a piece of hardware that stores led toward the finish line, the system 28.) But nearly 70 years ago, Minsky felt
electrical energy, that could remember for adjusted to increase the likelihood of neural networks were too limited because
a few seconds if the neuron had recently that firing pattern happening again. Sure they didn’t have a sense of expectation,
relayed a signal. Long-term memory was enough, the rats began making fewer which is critical for filling in gaps in human
handled by a potentiometer, or volume wrong turns. Multiple rats could run at perception. He shifted his focus to symbolic
knob, that would increase a neuron’s once to increase the speed at which the artificial intelligence, which, instead of
probability of relaying a signal if it had system learned. trying to mimic the brain’s approach to
just fired when the system was “rewarded,” “We sort of quit science for a while to information processing, draws inspiration
either manually or through an automated watch the machine,” Minksy told the New from the nature of human thought,
electrical signal. Yorker in a 1981 profile. “We were amazed doing computations based on high-level
Minsky and Edmonds tested the that it could have several activities going concepts that are interpretable by people.
model’s ability to learn a maze. The on at once in its little nervous system.” Minsky, who died in 2016, did
details of how the young researchers Artificial neural networks continued think that neural networks would be
COURTESY OF MARGARET MINSKY

tracked the output are unclear, but one
theory is that they observed, through
an arrangement of lights, how a signal
moved through the network from a
random starting place in the neural
network to a predetermined finish line.
The duo referred to the signal as “rats”
running through a maze of tunnels.
When the rats followed a path that
to advance over the next seven decades,
but it’s only been since the turn of the 21st
century that the approach has taken center
stage in the field of artificial intelligence.
Deep neural networks, which involve
many layers of computation, are now
powering myriad applications, including
ever-more-realistic models of the human
brain. (See “The Silicon Brain” on page
one component of a truly intelligent
machine, notes his daughter, interactive
computing expert Margaret Minsky, a
visiting professor at New York University
Shanghai. She wonders what he would
think about that today. “Given the things
[artificial neural networks] do and don’t
do, and how they fit into systems now, I
really want to know what he would say.” J

6 4 T H E SC I EN T I ST | the-scientist.com
DeLIVERing
to Your Needs!
Quality-Assured and Fully-Characterized
Primary Human Hepatocytes (PHH)

Our PHH batches are characterized by:

• CYP Induction
• Plateability & Morphology
• Phase I and II Enzyme Activities
• HLA characterization

1. Select Access current inventory

2. Test Evaluate batch performance

3. Reserve Batch sizes from 50 to 500 vials

For more information, contact

admetox@sial.com

© 2019 Merck KGaA, Darmstadt, Germany and/or its

affiliates. All Rights Reserved. MilliporeSigma and the
vibrant M are trademarks of Merck KGaA, Darmstadt,
Germany or its affiliates. All other trademarks are the
property of their respective owners. Detailed information
on trademarks is available via publicly accessible
resources.

The life science

business of Merck
KGaA, Darmstadt,
Germany operates as
MilliporeSigma in the
U.S. and Canada.
 10:37 AM OCT 11, 2021

THE MOMENT A DIFFICULT

BIOLOGICAL QUESTION HAS
AN ANSWER_

WITH AN APPROACHABLE, AFFORDABLE AUTOMATED CELL SORTING SOLUTION

IN YOUR LAB. Cell sorting may be complex, but it doesn’t need to feel complicated or out
of reach. With intuitive software that requires minimal training, the BD FACSMelody™ cell
sorter enables deep scientific insights with reproducible results, cost savings and workflow
efficiencies. Discover how better instrumentation can free up your time, so you can focus
your expertise where it matters most. Discover the new BD.

Advance your research at bd.com/ResearchSolutions

Class I Laser Products. For Research Use Only. Not for use in diagnostic or therapeutic procedures.
BD, San Jose, CA, 95131, U.S.

BD, the BD Logo and FACSMelody are trademarks of Becton, Dickinson and Company or its affiliates.
© 2019 BD. All rights reserved. 0219-1699