8/17/2014 Information Processing: Neural Networks and Deep Learning

http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 1/12
More

Next Blog Create Blog

Sign In
Information Processing
Pessimism of the Intellect, Optimism of the Will Archive Favorite posts
Twitter: @steve_hsu
Stephen Hsu
Google+ Profile
Saturday, August 16, 2014
Neural Networks and Deep Learning
One of the SCI FOO sessions I enjoyed the most this year was a
discussion of deep learning by AI researcher Juergen
Schmidhuber. For an overview of recent progress, see this recent
paper. Also of interest: Michael Nielsen's pedagogical book
project.
An application which especially caught my attention is described
by Schmidhuber here:
Many traditional methods of Evolutionary Computation
[15-19] can evolve problem solvers with hundreds of
parameters, but not millions. Ours can [1,2], by greatly
reducing the search space through evolving compact,
compressed descriptions [3-8] of huge solvers. For
example, a Recurrent Neural Network [34-36] with
over a million synapses or weights learned (without a
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 2/12
teacher) to drive a simulated car based on a high-
dimensional video-like visual input stream.
More details here. They trained a deep neural net to drive a car
using visual input (pixels from the driver's perspective, generated
by a video game); output consists of steering orientation and
accelerator/brake activation. There was no hard coded structure
corresponding to physics -- the neural net optimized a utility
function primarily defined by time between crashes. It learned
how to drive the car around the track after less than 10k training
sessions.
For some earlier discussion of deep neural nets and their
application to language translation, see here. Schmidhuber has
also worked on Solomonoff universal induction.
These TED videos give you some flavor of Schmidhuber's sense
of humor :-) Apparently his younger brother (mentioned in the
first video) has transitioned from theoretical physics to algorithmic
finance. Schmidhuber on China.
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 3/12
Older Post
Posted by Stephen Hsu at 7:28 PM
Labels: ai, brainpower, computing, machine learning, scifoo,
singularity
0 Comments information processing Login
Sort by Best Share
Start the discussion
Be the first to comment.
Subscribe Add Disqus to your site
Favorite
Links to this post
Create a Link
Home
Subscribe
Posts
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 4/12
Blog Archive
2014 (116)
08 (7)
Neural Networks and Deep Learning
Y Combinator: "fund for the pivot"
Designer babies: selection vs editing
SCI FOO 2014: photos
Next Super Collider in China?
@ SCI FOO 2014
It's all in the gene: cows
07 (15)
06 (12)
05 (19)
04 (12)
03 (18)
02 (16)
01 (17)
2013 (211)
2012 (222)
2011 (267)
2010 (234)
2009 (204)
2008 (255)
2007 (196)
2006 (154)
2005 (253)
2004 (100)
Labels
physics (285)
finance (245)
globalization (220)
genetics (205)
brainpower (193)
photos (130)
technology (120)
China (119)
science (118)
economics (116)
american society (115)
credit crisis (115)
psychometrics (110)
Comments
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 5/12
travel (99)
innovation (86)
psychology (81)
universities (81)
human capital (80)
credit crunch (78)
iq (78)
higher education (77)
startups (75)
gilded age (66)
biology (62)
careers (62)
income inequality (60)
elitism (58)
autobiographical (55)
cognitive science (55)
ai (52)
books (51)
evolution (49)
quantum mechanics (49)
genius (46)
bgi (45)
cdo (45)
caltech (43)
kids (43)
social science (43)
derivatives (41)
genetic engineering (41)
statistics (38)
talks (38)
education (36)
mma (36)
mortgages (36)
bubbles (35)
history of science (35)
behavioral economics (34)
genomics (32)
hedge funds (31)
mathematics (31)
bounded rationality (30)
intellectual history (29)
silicon valley (29)
efficient markets (27)
podcasts (27)
quants (27)
literature (26)
many worlds (26)
black holes (25)
geopolitics (25)
sports (25)
subprime (25)
MSU (24)
entrepreneurs (24)
expert prediction (24)
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 6/12
housing (24)
political correctness (24)
taiwan (24)
academia (23)
foo camp (23)
obama (22)
physical training (22)
ufc (22)
machine learning (21)
sci fi (21)
ultimate fighting (21)
athletics (20)
film (20)
google (20)
jiujitsu (20)
wall street (20)
berkeley (19)
cds (19)
economic history (19)
feynman (19)
history (19)
computing (18)
goldman sachs (18)
movies (18)
affirmative action (17)
bjj (17)
harvard (17)
security (17)
treasury bailout (17)
freeman dyson (16)
race relations (16)
von Neumann (16)
internet (15)
japan (15)
singularity (15)
nuclear weapons (14)
university of oregon (14)
blogging (13)
neuroscience (13)
oppenheimer (13)
personality (13)
quantum field theory (13)
realpolitik (13)
biotech (12)
gender (12)
happiness (12)
india (12)
politics (12)
probability (12)
venture capital (12)
algorithms (11)
conferences (11)
hedonic treadmill (11)
malcolm gladwell (11)
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 7/12
meritocracy (11)
music (11)
scifoo (11)
autism (10)
cosmology (10)
fitness (10)
mutants (10)
net worth (10)
olympics (10)
social networks (10)
string theory (10)
aspergers (9)
chess (9)
entropy (9)
geeks (9)
neanderthals (9)
new yorker (9)
wwii (9)
ability (8)
cryptography (8)
flynn effect (8)
italy (8)
keynes (8)
nobel prize (8)
robot genius (8)
usain bolt (8)
aig (7)
alpha (7)
ashkenazim (7)
basketball (7)
christmas (7)
complexity (7)
crossfit (7)
dating (7)
eugene (7)
football (7)
harvard society of fellows (7)
hugh everett (7)
manhattan (7)
nerds (7)
real estate (7)
television (7)
Einstein (6)
alan turing (6)
anthropic principle (6)
art (6)
blade runner (6)
data mining (6)
determinism (6)
dna (6)
fx (6)
games (6)
les grandes ecoles (6)
nassim taleb (6)
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 8/12
qcd (6)
success (6)
teaching (6)
turing test (6)
volatility (6)
Fermi problems (5)
academia sinica (5)
bayes (5)
bobby fischer (5)
climate change (5)
energy (5)
environmentalism (5)
france (5)
free will (5)
game theory (5)
height (5)
james salter (5)
luck (5)
noam chomsky (5)
nsa (5)
philip k. dick (5)
philosophy of mind (5)
poker (5)
privacy (5)
prostitution (5)
renaissance technologies (5)
software development (5)
tail risk (5)
war (5)
warren buffet (5)
100m (4)
Go (4)
Iran (4)
Poincare (4)
borges (4)
cambridge uk (4)
charles darwin (4)
class (4)
creativity (4)
econtalk (4)
fake alpha (4)
global warming (4)
godel (4)
government (4)
hormones (4)
humor (4)
inequality (4)
iraq war (4)
kerviel (4)
markets (4)
microsoft (4)
monsters (4)
paris (4)
perimeter institute (4)
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 9/12
pseudoscience (4)
research (4)
russia (4)
soros (4)
trento (4)
vietnam war (4)
200m (3)
babies (3)
bill gates (3)
brain drain (3)
censorship (3)
charlie munger (3)
chet baker (3)
cold war (3)
correlation (3)
demographics (3)
ecosystems (3)
encryption (3)
equity risk premium (3)
facebook (3)
fannie (3)
feminism (3)
fst (3)
information theory (3)
intellectual property (3)
intellectual ventures (3)
judo (3)
kasparov (3)
lewontin fallacy (3)
lhc (3)
michael lewis (3)
mixed martial arts (3)
moore's law (3)
nathan myhrvold (3)
neal stephenson (3)
new york times (3)
nonlinearity (3)
patents (3)
path integrals (3)
quantum computers (3)
risk preference (3)
sad but true (3)
search (3)
sec (3)
sivs (3)
society generale (3)
solar energy (3)
alibaba (2)
assortative mating (2)
bear stearns (2)
bruce springsteen (2)
charles babbage (2)
cloning (2)
david mamet (2)
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 10/12
democracy (2)
digital books (2)
donald mackenzie (2)
eliot spitzer (2)
empire (2)
exchange rates (2)
freddie (2)
gaussian copula (2)
industrial revolution (2)
james watson (2)
jim simons (2)
language (2)
ltcm (2)
magic (2)
mba (2)
mccain (2)
national character (2)
nicholas metropolis (2)
no holds barred (2)
offices (2)
oligarchs (2)
olympiads (2)
palin (2)
pca (2)
pop culture (2)
population structure (2)
prisoner's dilemma (2)
rationality (2)
skidelsky (2)
socgen (2)
sprints (2)
thailand (2)
variance (2)
abx (1)
anathem (1)
andrew lo (1)
antikythera mechanism (1)
athens (1)
atlas shrugged (1)
ayn rand (1)
bay area (1)
beats (1)
book search (1)
bunnie huang (1)
car dealers (1)
carlos slim (1)
catastrophe bonds (1)
cdos (1)
ces 2008 (1)
chance (1)
cheng ting hsu (1)
children (1)
cochran-harpending (1)
cpi (1)
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 11/12
david x. li (1)
dick cavett (1)
dolomites (1)
drugs (1)
eharmony (1)
epidemics (1)
escorts (1)
faces (1)
fads (1)
favorite posts (1)
fiber optic cable (1)
francis crick (1)
gary brecher (1)
gizmos (1)
greece (1)
greenspan (1)
heinlein (1)
hypocrisy (1)
igon value (1)
iit (1)
inflation (1)
information asymmetry (1)
iphone (1)
jack kerouac (1)
jaynes (1)
jfk (1)
john dolan (1)
john kerry (1)
john paulson (1)
john searle (1)
john tierney (1)
jonathan littell (1)
las vegas (1)
lawyers (1)
lee kwan yew (1)
lehman auction (1)
les bienveillantes (1)
lowell wood (1)
lse (1)
mating (1)
mcgeorge bundy (1)
mexico (1)
michael jackson (1)
mickey rourke (1)
migration (1)
mit (1)
money:tech (1)
monkeys (1)
myron scholes (1)
netwon institute (1)
networks (1)
newton institute (1)
nfl (1)
oliver stone (1)
8/17/2014 Information Processing: Neural Networks and Deep Learning
http://infoproc.blogspot.nl/2014/08/neural-networks-and-deep-learning.html 12/12
phil gramm (1)
philanthropy (1)
philip greenspun (1)
portfolio theory (1)
power laws (1)
randomness (1)
recession (1)
sales (1)
simulation (1)
singapore (1)
skype (1)
smpy (1)
standard deviation (1)
star wars (1)
starship troopers (1)
students today (1)
supercomputers (1)
systemic risk (1)
teleportation (1)
tierney lab blog (1)
tomonaga (1)
twitter (1)
tyler cowen (1)
ussr (1)
venice (1)
violence (1)
virtual meetings (1)
virtual reality (1)
war nerd (1)
wealth effect (1)