USO
2) United States Patent
ay
PHOTOGRAPHING DEVICE, MOVING
BODY FOR PHOTOGRAPHING, AND.
PHOTOGRAPHING CONTROL APPARATUS,
FOR MOVING BODY
Applicant: OLYMPUS CORPORATION, Tokyo
«P)
Inventors
Yoll Osanal, Hochioji QP); Tsutomu
Igarashi, Tokyo (IP); Taichiro Kouch,
Hino (JP); Osamu Nonaka, Sagamihara
«e)
(73) Assignee: OLYMPUS CORPORATION, Tokyo
«P)
(4) Notice: Subjoct to any disclaimer, the term of this
patent is extended or adjusted under 35
USC. 154(b) by 134 days
Appl. No. 15/457,268
Filed: Mar. 13, 2017
Prior Publication Data
'S 201810227469 AL Aug. 9, 2018
G0) Foreign Application Priority Data
Mat. 23, 2016 (UP) 2016-088814
(1) Ine ch.
HOIN 5232 (2006.01)
HOIN 5225 (2006.01)
Ga2R 27764 (2005.01)
B6AC 39702 (2005.01)
BosD 47/08 (2005.01)
a
coe
HON $2254 (2013.01); B6IC 39/024
(013.01); Go2B 27/644 (3013.01), HOIN
$252 (201301); HOSN 5/232 (201301)
STABILITY CONTROL
$1
1035629482
US 10,356,294 B2
Jul. 16, 2019
(10) Patent No.:
(4s) Date of Patent:
B64C 2201/127 (201301); BEC 2201/11
(2013.01); B64D 47/08 (2013.01),
Field of Classification Search
crc BG4C 2201/127; BO#C 39/024; BOC
2201/027; BaD 47/08; GBB 15,06:
GOSB 17/561; GO2B 27/644; GOB
27/646; GOSD 1/0004; HON 5/2254;
HOSN 7/18; HOAN 5/2328; GOIC 21/18;
GOIC 25/00; GOIM. 1/12; GOIM 1/36
‘See application file for complete search history
658)
(56) References Cited
us
PATENT DOCUMENTS.
core 2018
24811
8.184.965 B2* $2012 Hayashi
798450 B2* 82014 Aicher
(Continved)
Chia Wei A Chen
Scully, Seot, Murphy &
(74) Attorney, Agent, oF Frm
Prosser, PC.
on ABSTRACT
A photographing device attachable 1 a moving body
‘includes: First control section configured to predict and
joblain a change in physieal quantity bused on an alteration
in a photographing state: and a communication contol
section configired to transmit information related to the
change in the physical quantity. The photographing device
improves stability in the moving body and stability of
photographing. Information related wo influence on the m0v-
sng body’ based on a change in an outer shape and a ebange
‘in # postion ofthe center of gravity of the photographing
‘device mounted on the moving body is shared by the moving
body to improve the stability in the moving body and the
stability of photographing,
7 Claims, 18 Drawing Sheets
PREDICT CHANGE IN PHYSICAL QUANTITY
BEFORE ALTERATION IN DEVICE STATE
$3
[ALTER DEVICE STATE IN COOPERATION
WITH MOVING BODY TO KEEP BALANCE
DO NOT ALTER DEVICE STATEUS 10,356,294 B2
Page 2
66) References Cited
USS. PATENT DOCUMENTS.
8801302 B2° 82014 OF GoxB 17561
396401
gastos B22 s2018 Hoaw 470,
DSIDSAI 8 * 62018 Chen ar
2o1g001eo22 AL* 1/2018 Grecia ‘ieMt 1100
306419
2o1aons7278 14 Wang FIC I10|
390155
20190370250 AL* 122015 Bachrach c0sp 10016
7012
201610014309 AL_ 1/2016 ison ta.
aovooosos4t Al* 22016 Beary {6050 10088
7012
20160171330 AL* 62016 Mente cook 95333
30170
20160286143 ALS 92016 Ohtsuka Hoax 523209
doreowisas Als 102016 Bell HOA 52328
2016Os?79S0 AL* 112016. Bachrach Gos 10016
Ale 112016 Jak Ho4N 17002
ALS “12017 Takayanagi 0" Hoan $2003
Ale 32017 Winter HON 188
dorpoogs2i4 AL* 32017. Dopey ‘cases s 0021
dorpoooatss Ale 32017 Wang G03 17861
20170106976 AI* 42017 Kubars BBs 39.004
dorpoles AL* 62017 Kang ‘wos 523287
aorpoarrtea Als $2017 Saika Flom L302
2010269604 AL* 92017 Shironiat”” AOID 44008
2oWwoo1770L AL* 12018 Beckman ‘G28 2701
2owons6H6s AL* 10.2018. Desweler OAD 102
* cited by examinerUS 10,356,294 B2
Sheet 1 of 18
Sul. 16, 2019
U.S. Patent
gL
mmm || o ve
eg] 3
+o ne nouoas | } nowoas | jy tosh3s
yoo | |Nouwuado
ouoas 2wuc¥o03 NOLLWASTA
I I 1
Hous | 92, [ NOLOR SWUOIS
7oHLNOo NOLLWINYOANI ALTIBWIS NOLLOas
oHDOL 0H a2t dNild 30¥Wl
eZ SS NOT NOL WALSAS
gp | tous [| [ROS t-——[_ ako raw stony avolldo
jouvonnniool")|7o4LNon aa
sosoW | —~-[ NOILO3S “TOWINOO PLL
ral NIWNRSL eer
zi ~| hous |_| (NOIOaSTOuINOD] «= PAE eo AS
ooo a
NOL YONONNOS | Nou.93s 1OXLNOD iL
a 3OIN3 ONIHeVY9OLOHd
7
OLUS 10,356,294 B2
Sheet 2 of 18
Sul. 16, 2019
U.S. Patent
9%
9G2.
qg@
er
ANSWOW fuzz
Loy
82 NOLS “TONINOD INSNZ9ONF ] |__| ATddNs
NOlO3S owvonnnog) | LNOLLVOINNWWOD AIddNS Y3IMOd YIMOd,
a NOLLOIS si" NOLLO3S TOULNOD NOILOSS
lunanapanr} 3UnsOd INoISindOud
Noloas ONGHOOSY f=) | BuNlSOd | 922-F eras oumnoo] | “pom
| [_NOWoas inanzoanr—L.| [Nouoas | 22 |__nowoawa | [| woaraw
3UNUSOd ONY 3GNLL TV TEs) Warenneeriray ua
NoILoas INaWa0GNF |_| |ANANSAONT\p2z | Noistndoud
NOLLOSYIC CNY NOLUSOd ezz
NOLLO3S NOLLISINOOV NOLLOaS NOLLO3S TOMLNOD
NOLLVWHOANI30vds [7] | TO¥LNOS ee
?
02
ab ‘SlsU.S. Patent Jul. 16,2019 Sheet 3 of 18 US 10,356,294 B2
FIG. 2A
30
t
PHOTOGRAPHING REMOTE CONTROL APPARATUS
32_ 32a 34
COMMUNICATION n
mea. To} | pyspray
contro | | SECTION
SECTION
STORAGE| | 36
OPERATION SECTION | |”
SECTION
31
FIG. 2B
40
?
MOVEMENT REMOTE CONTROL APPARATUS.
a 42 42a 44
EATON
eomunicarow_ | eormascial| [spray
SECTON 7
contro. | | SECTION
“745._| SECTION
Co ==
‘comupucaron f STORAGE
seoToN | [OPERATION] *| section |] 46
_ SECTION
t
alU.S. Patent Jul. 16,2019 Sheet 4 of 18 US 10,356,294 B2
FIG. 3
STABILITY CONTROL,
St
PREDICT CHANGE IN PHYSICAL QUANTITY
BEFORE ALTERATION IN DEVICE STATE
$3
DO NOT ALTER DEVICE STATE
ALTER DEVICE STATE IN COOPERATION
WITH MOVING BODY TO KEEP BALANCE
IU.S. Patent Tul. 16, 2019 Sheet 5 of 18 US 10,356,294 B2
FIG. 4
Gt G2 § G3 G4G5 FC
A aie elit
HEH
ecie G2 $ G3 G4G5 FC
™ ar Am on
fl
GI G2sas G4 GSFC
4 mrara on am
——— fhe eaU.S. Patent Sul. 16, 2019
Sheet 6 of 18
CAMERA CONTROL
JUDGE COMMUNICATION STATE|~-S11
START AE AND IMAGE PICKUP }~-S12
DRONE MODE?
YES 4
<{§ THERE DRONE COMMUNICATIONS
YES
TRansutT Mage [S'S
$17
>
27
NO_ FOLLOW OTHER OPERATION
HOTOGRAPHING OPERATION RECENVED
YES _;S!
PHOTOGRAPH AND RECORD
$21
Gove scion ore Sea. >-NS
I $28 [yes _-s22
US 10,356,294 B2
S15
TRANSMIT TO DRONE
ISSUE WARNING]
NOCE CHANGE PHYSICAL QUANTITY
q $23
CCONMUNIGATE VATH DRONE
S24
{UDGED AVAILABILITY?
S26 S25
ALTERDEWGE STATE AOCORONG
TOSTHBLE AERA TON CONDON
PRONPT TO DISPLAY WARNING
CN DISPLAY SECTION
ee eee eee
in $29
0 << STOP DRONE?
YES
STOPU.S. Patent Tul. 16, 2019 Sheet 7 of 18 US 10,356,294 B2
MOVING BODY CONTROL
WOGE STATE earl
(ALTITUDE, RADAR BUILT-IN CAMERA, GPS)
$32
1S THERE PROGRAM OR IS MOVEMENT S,, NO
HANDLING SCNAL RECENED! $33
i FLIGHT BE STOPPED
{S THERE MOVENENT OBSTACLE? YES
NO__-$38
‘MOVE ACCORDING TO PROGRAM $34
OR HANDLING RESULT {§ RETURN BETTER?
YES $35 $36
PERFORM RETURN CONTROL | [ PERFORM STILL CONTROL
TRANSMIT STATE INFORMATION|
‘TO HANDLING SIDE
{8 THERE CAMERA COMMUNICATION?
ACQUIRE DATA FROM PHOTOGRAPHING DEVICE |-~S42.
;~S39
I
JUDGE WHETHER TT CAN BE HANDLED [~-S43
S44
YES S47 $45
TRANSMIT CONDITION GENERATE UNAVAILABLE RESPONSE
I $48 I S46
EAT AO CONTROL UNeRSPECCcoRDTON] [DISPLAY ON HANDLING SECTION siDE|
|U.S. Patent Jul. 16,2019 Sheet 8 of 18 US 10,356,294 B2
FIG. 7A
44a
{
62
—< 34aU.S. Patent Jul. 16,2019 Sheet 9 of 18 US 10,356,294 B2U.S. Patent Sul. 16, 2019 Sheet 10 of 18 US 10,356,294 B2
FIG. 9A
CENTER
OF
GRAVITY
TIME PERIOD.
BACK
FIG. 9B
FORWARD
LIFTING /
POWER
TIME PERIOD,U.S. Patent Jul. 16,2019 Sheet 11 of 18 US 10,356,294 B2
FIG. 10
STABILITY CONTROL
JUDGE WHETHER OPERATION Is PHOTOGRAPHING
DEVICE STATE ALTERATION OPERATION
I
PREDICT CHANGE IN PHYSICAL QUANTITY |~S52
f~S51
NO
$3
DO NOT ALTER DEVICE STATE
[ALTER DEVICE STATE IN COOPERATION
|WITH MOVING BODY TO KEEP BALANCE |
IU.S. Patent Jul. 16,2019 Sheet 12 of 18 US 10,356,294 B2
FIG. 11
CAMERA CONTROL
JUDGE COMMUNICATION STATE}~S11
I
START AE AND IMAGE PICKUP }~S12
$13
DRONE MODE?
YES
No_ FOLLOW OTHER OPERATION
‘TRANSMIT IMAGE
$21
PHOTOGRAPH AND RECORD (MOVABLE SECTION OPERATION SIGNAL?
1 $28 [YES 822
TRANSMIT TO DRONE JUDGE CHANGE IN PHYSICAL QUANTITY]
1 S61
[COMMUNICATE WITH MOVEMENT
REMOTE CONTROL APPARATUS:
NO
[ATERDEVICE STATE ACCORDING | [PRONPT TO DISPLAY WARNING|
[TOSTABLE ALTERATION CONDITION} | ON DISPLAY SECTION
$29
‘STOP DRONE?
YESU.S. Patent Jul. 16,2019 Sheet 13 of 18 US 10,356,294 B2
FIG. 12
MOVING BODY CONTROL
JUDGE STATE
(ALTITUDE, RADAR, BUILT-IN GAMERA, GPS)
$32
{S THERE PROGRAN ORS HOVEHENT, NO.
HANDLING SINAL REOENED?
{s THERE MOVEMENT OBSTACLE?
NO__-s38
MOVE ACCORDING TO PROGRAM
OR HANDLING RESULT
S31
$36
PERFORM RETURN CONTROL | | PERFORM STILL CONTROL
——
871
{INFORMATION REQUEST FROM MOVEMENT
REMOTE CONTROL APPARATUS?
[TRANSMIT STATE INFORMATION
‘TO HANDLING SIDEU.S. Patent Jul. 16,2019 Sheet 14 of 18 US 10,356,294 B2
FIG. 13
Gometeprconet) BODY CONTROL
FEOUEST TRANSMISSION OF STATE nrORUATION}/“S8"
|
RECEIVE STATE INFORMATION
|S THERE COMMUNICATION FROM
PHOTOGRAPHING REMOTE CONTROL APPARATUS?
YES
[ACQUIRE DATA FROM PHOTOGRAPHING] S84
REMOTE CONTROL APPARATUS
Fo
JUDGE WHETHER IT CAN BE HANDLED |~S43
S44
CAN IT BE HANDLED?
YES S47
$82
$83
$45
TRANSMIT CONDITION GENERATE UNAVAILABLE RESPONSE
1 S48, 1 S46
[reAT ao cowTROL DER SPEOFAG CoNDTION] [DISPLAY ON HANDLING SECTION SIDE
IU.S. Patent Jul. 16,2019 Sheet 15 of 18 US 10,356,294 B2
FIG. 14A
40
Q
76
76a | 1» t-44a
77_|_|__ POSSBLENOVEUENT OF ~~ 75
a CENTER OF GRAVITY
78_| Leenrer oF} wn 11 | 79
GRAVITY | |VELOcITYU.S. Patent Jul. 16,2019 Sheet 16 of 18 US 10,356,294 B2
FIG. 15
ZOOM PHOTOGRAPHING CONTROL
ALTER TO ELECTRONIC ZOOM
I
‘SET ANGLE OF VIEW ACCORDING TO INCLINATION ANGLE |~S93
ENDU.S. Patent Jul. 16,2019 Sheet 17 of 18 US 10,356,294 B2
FIG. 16U.S. Patent Sul. 16, 2019 Sheet 18 of 18 US 10,356,294 B2
FIG. 17A
a7 18 96 97 91b 96
RS zs]
=} LS \
: : )
95 95 oe
FIG. 17B
a7 8% 96 97 82 96 a7 92°
R
FIG. 17C
96 97 93 96 98¢ 96US 10,356,294 B2
1
PHOTOGRAPHING DEVICE, MOVING
BODY FOR PHOTOGRAPHING, AND.
PHOTOGRAPHING CONTROL APPARATUS
FOR MOVING BODY
CROSS REFERENCE TO RELATED
APPLICATION
‘This application claim is honfit of Japanese Application
No. 2016-058814 filed in Japan on Mar. 23, 2016, the
‘contents of which are incorporated by this reference.
BACKGROUND OF THE INVENTION
1. Field of the Invention
‘The present invention relates to a photographing device,
‘4 moving boy for photographing, and a photopraphing
‘control apparatus for moving body suitable for a remote
handling or autonomous fying objeet such as a drone.
2. Description of the Related Art
In revent years, an unmanned aera, such as a drone,
that flies in a remotely handled or stonomous manner has
been commercialized. For example, @ relatively stall
‘unmanned aizceat is utilized due to an advantage that
information can be safely and efficiently acquired in a
‘dangerous area tha shard fora manned aircraft to reach, an
accident site of a disaster area in which observation at
relatively low altitude is required, and the like. A camera is
bul in the drone othe ike to stain the objec
However, a relatively small lens or image pickup device
js often used in the built-in eamera, and a picked-up image
with sufficiently satisfactory image quality may not be
‘obtained. Therefore, photographing device with high fune~
tionality and high performance may be adopted and attached
‘wo the unmanned areral. Particularly, a camera that can be
‘equipped with a high-performance leas having a zoom
Junetion, an autofocus Function, and the lke ean be adopted
‘o easily obtain an image with image quality desired by a
‘Note that a gimbal system that can aecurately support a
photographing device on a moving body, such as an airera
's disclosed in the specification of U.S. Patent Application
Publication No, 20150014309,
SUMMARY OF THE INVENTION
An aspect ofthe present invention provides photograph=
ing device atachable w a moving body, the photographing
‘device including a fist control section configured to predict
tnd obtain a change in physical quantity. based on an
alteration in a photographing sate; and a communication
‘contol section configured to transmit information related to
the change in the physical quantity.
"An aspect of the present invention provides a moving
body for photographing to which a photographing device is
tached, the moving body for photographing including: &
‘communication contol section coatighzed to receive 8
‘change in physical quantity related to the photographing
‘device generated based on an alteration in a photographing
state of the photographing device; «second control section
‘configured 0 generate information based on the change in
the physical quantity or based om ® moment generated
acconling to the change in the physical quantity; and a
0
o
2
propulsive force control seotion configured to contol pro-
pulsive force based on the information generated by’ the
second control section,
‘An aspect of the preset invention provides a photograph
ing control apparatus for moving body including: a lst
control section configured to obisin a change in physical
quantity related to 3 photographing device attached to
moving body based on a photographing state alteration
‘operation ofthe photographing device; anda second control
section configured to predict and obtain the change in the
pysieal quantity oe a moment to be generated inthe moving
body acconting to the change in the physical quantity and
szenemite information based on the obtained change in the
physical quantity or dhe obtained momeat.
‘The above and other objects, features and advantages of
‘he invention will become more clearly understood from the
following description refering tothe accompanying draw-
ings
BRIEF DESCRIPTION OF THE DRAWINGS
PIG. 1A is a block diagram showing photographing
device according to a first embodiment of the present
invention, the photographing device configuring part of @
photographing coatrol apparatus for moving body:
FIG. 1B is a block diagram showing a moving body for
photographing according to the first embodiment, the mov~
jing body configuring part of the photographing contol
Apparat for moving body;
FIG. 2A is a block diagram showing a photogrpbing
remote control apparatus of the photographing device of
FIG. 1A;
FIG. 2B isa block diagram showing a moves
ccantrol apparatus of the moving body of FIG, 1B
FIG, 3s a flowehar showing stability comtol by the
photographing device of the present embodiment;
FIG. 4 isa cross-sectional view of Tens showing an
‘example ofa configuration of respective lenses in an optical
system Ma;
FIG. § is a flowchart showing camera conta;
FIG. 6 is a owchart showing moving body control;
PIG. TAs an explanatory diagram showing a situation of
controling photographing:
FIG. TB is an explanatory diagram showing a situation of
controling photographing;
FIG. 8A isan explanatory diagram for deseribing contol
of the moving body when the photographing device 10 is
poomings
FIG. 8B isan explanatory diagram for deseribing contol
of the moving body when the photographing device 10 is
‘zooming:
FIG. 9A is an explanatory diagram showing a change ia
the contr of gravity ofthe photographing device 10;
PIG. 9B is an explanatory diagram showing a change in
forward lifting power comesponding to the change in the
center of gravity of FIG. 8
FIG. 10 isa flowchart adopted in second embodiment of
the present inveation
FIG. 11 is an explanatory diagram showing camera con-
two by the photographing remote control apparatus;
PIG. 12 isan explanatory diagram shoving moving body
control by the moving body:
FIG. 13 isan explanatory diagram shoving moving body
‘control by the movement emote contol spparats;
FIG. 144 isan explanatory diagram showing a situation
of controlling the moving boey:
at remoteUS 10,356,294 B2
3
FIG. 148 is an explanatory diagram
‘of controlling the moving body’
IG, 18 is flowchart for deserihing contol adopted ia &
fourth embodiment
FIG. 16 is an explanatory diagram for describing the
fourth embodiment
TIG. 17A is an explanatory dingram for describing the
fourth embodiment
TIG. 178 is an explanatory dingram for describing the
fourth embodiment; and
FIG. 17C isan explanatory dingram for describing the
fourth embodiment
owing a situation
DETAILED DESCRIPTION OF THE
PREFERRED EMBODIMENTS
Hereinafter, embodiments of the present invention will be
described in detail with reference tothe drawings,
First Embodiment
FIG. 14 is a block diagram showing @ photographing
device according to a first embodiment of the prescat
invention, the photographing device configuring part of a
photographing control apparatus for moving bod FIG. 1B
184 block diagram showing a moving body for photogzaph
ing according to the first embodiment, the moving body
‘configuring part ofthe photographing contol apparatus for
moving body. FIG. 2A is a block diagram showing @
photographing remote contol apparatus of the photograph-
tng device of FIG. 1, PIG, 28 isa block diagram showing
‘a movement remote contol upparats ofthe moving body of
FIG. 1B.
The moving body may be, for example, an unmanned
aircraft, such as a drone, or may be a radio control vehicle
‘or & radio coateol ship. The moving body is @ remote
handling or autonomous unmanned moving object. In the
present embodiment, the photographing device configuring
the photographing control apparatus for moving. body
‘obtains, aa change inthe physica quantity, movement of
the center of gnivity or a change in the shape of the
photographing device which affects the movement and the
Posture ofthe moving body, and traits the change in the
physical quantity toward the moving body. Inthe moving
body, the change in the physical quamity ean be used to
‘control the movement and the posture ofthe moving body to
improve the stability of the moving body. The movement of
the center of gravity or the change in the shape of the
photographing device is permitted nd controlled while the
Stability of the moving body is ensured, and this allows
securing photographing while stabilizing the moving body.
FIG. 3 isa flowchart showing the stability control by the
photographing device of the present embodiment.
In step SI, the photographing device predicts the change
Jn the physical quantity based on an alteration in the device
state (photographing state) before the alteration of the device
state. For example, a movement of a 700m lens ora focus
Jens ofthe photographing device changes the position ofthe
‘center of gravity of the photographing device. When, for
‘example, @ stroboseope part built in the photographing
‘device pops up, and the shape of the photographing device
js changed, the change in the shape ofthe stroboseope part
‘changes the wind resistance, When a photographing state
alteration operation is done, such a the lens movement and
the pop-up of the stroboscope part, the photographing device
predicts, as an amount of change inthe physical quantity, the
‘change inthe position of he center of privity othe change
4
in the wind resistance base on the alteration in the device
state (photographing state), prior to the alteration in the
device sate
In step S2, the photographing device communicates with
the moving body’ to transmit the change in the physical
quantity to the moving body and obisins an answer for
‘Whether the change inthe physical quantity ean be allowed
in the moving body. [fan answer of not allowing the change
is obtained from the moving body, the photographing device
does ot permit the alteration in the device sate instep 3.
For example even if he user performs a zoom operation, the
‘operation is not performed
‘On the ther hand, if the photographing device obtains an
answer of allowing the change from the moving body instep
S2, the photographing device alters the device state in
cooperation with the moving body in step $4. That is, the
‘moving body allows the change inthe physical quantity of
the photognphing device while setting a condition neves-
say’ for maintaining the stability ofthe moving body. ‘The
photogruphing device alles the device slate forchanging the
Physical quantity while following the condition designated
by the moving body.
‘In this way, the photographing device predicts in advance
‘the change in the physical quantity that alfets the service of
the moving body and transmits the information to the
:moving body inthe present embodiment. The photographing
device andthe moving body provided with the photograph
ing device peeform the canto] in cooperation to ensie the
stability of the moving body even when the state of the
photographing device is altered. The stateof the photograph
dng device can he altered while the stability of the moving
body is maintained, and the stability of photographing is
censured even when the state of the photographing device is
altered.
‘Note that although most of the configunition of the
photographing control apparatus for moving body is pro-
¥ided inthe photographing device inthe example illustrat
inthe present embodiment, the photographing control appa
rus for moving body may be provided ina photograpsing
remote control apparatus configured to remotely control the
photographing device or may be provided in the moving
body such as. drone, The photographing contol apparatus
{or moving body may be provided in a movement emote
control apparatus configured to remotely conieol the moving
body or may be dispersed and provided in the apparatuses
‘The photographing contol apparatus for moving body may
be provided in an apparatus other than the apparatuses.
In FIG. 1, an attachment member not shown ean attach
a photographing device 10 to 9 moving body 20 of FIG, 1B.
The photographing device 10 includes an image pickup
section I. The image pickup section fis provided with an
image pickup device not shown configured by a CCD or
(CMOS sensor or the lke, and an optical system a leads aa
‘object optical image to an image pickup surface ofthe image
pickup device. A photographing control section 12a of a
feontral section 12 is configured to control a mechanical
section not shown of the optical system to drive and control
focus lens, 700m lens, anda diaphragm configured in the
‘optical system Ta, The image pickup device photoclett-
cally converts the object optical image to obtain a picked-up
mage. Noe tha the contol section 12 may be configured by
8 processor, such as a CPU, net shown and may be config-
‘ured to realize each funetion aeconding to programs stored in
fa memory not shown,
Tn the present embodiment, a stability information storage
section II configured to hold information related to the
‘optical system Hr is provided in the image pickup sectionUS 10,356,294 B2
5
1, Note that although the optical system Mais provided i
the image pickup section 11 in the example illustrated ia
FIG. A, an interchangeable lens attachable to and detachable
from a housing of the photographing device 10 may be
adopted as an optical system, and an object optical image
fom the interchangeable lens may be formed onthe image
pickup surface of the image pickup device
FIG. 4 is a cross-sectional view of lens showing an
‘example ofa configuration of respective enses in the optical
system Ha. An upper row of FIG. shows a wide-angle end
slate. in which an infinite objet point is focused, A middle
row shows an intermediate state B, anda lower row shows
telephoto end state C.
In the example of FIG. 4, the optical system includes, in
‘onder om the objec side a first lens group GH with positive
reffactive power, a second lens group G2 with negative
refiaetive power, an aperture diaphragm S, third lens group
(G3 with positive refractive power, a fourth lens group Ga
with negative effective power, and a filth lens group GS
With positive refractive power In FIG. 4, P represents @
parallel plate configuring a low-pass filter provided with @
Wavelengih band restriction coat for restricting infrared
light, C represents a parallel plate of a cover glass of an
‘electronic image pickup device, and I represents an image
plane. Note that # multilayer fl for restricting the wave:
Jength band may be applied on the surface ofthe cover glass
C. The cover glass C may have a low-pass filter function,
‘With the lens configuration the first lens yroup G1 moves
toward the object side during zooming from the wide-angle
‘end to the telephoto end, The second Tens group G2 moves
toward the image side. ‘The third lens group G3 moves
toward the object side. The fourth lens group G4 moves
toward the object side. The fifth lens grup GS moves
toward the object side and then moves toward the image
side. Therefore, as fr intervals among the lens groups the
Jnerval between the fist lens group GI and the second lens
‘group G2 increases, the interval between the second fens
group G2 and the thid lens group G3 decreases, the interval
between the thing lens group C3 and the fourth lens group
‘G4 changes, and the interval between the fourth lens group
(G4 and the fifth Jens group GS increases. The eperture
emote control apparatus alters the device state in eoupent-
tion with the moving body. The photographing. remote
‘control apparatus follows the condition designated hy the
moving body and controls the photographing. device by
enerating an operation signal for altering the device state
that changes the physical quantity:
[Note that respective processes of respective sleps SS
$552, and S2 to $4 of FIG. 10 are enabled by, for example,
ring information for the processes in a storage section
provided in the photographing remote control apparatus.
In this way, the photographing remote eontrol apparatus
‘configured to remotely conte the photographing device and
the moving body provided with the photographing device
perform the cooperative contol in the present embodiment
to ensure the sibility of the moving body even whea the
state ofthe photographing device is altered. The state of the
Photographing devive can be altered while the stability of
the moving body is maintained, and the stability of photo-
traphing is ensured even when the state of the photogzaph-
fing device is altered
‘Note that although the stability control by the photograph
ing control apparains for moving body is executed by the
‘cooperative control through the communication between the
photographing remote contol apparatus and the moving
body inthe example illustrated in the present emboctiment,
itis apparent thatthe same stability contro can be exceed
by cooperative contro through communication between the
Photographing device and the movement remote control
apparatus.
‘Third Embodiment
FIGS. 11 10 13 are flowehants for describing contol
‘adopted in third embodiment ofthe present invention, FIG,
AL shows camera control by the photographing remote
‘control apparatus. FIG. 12 shows moving body conteol by
the moving body: PIG, 13 shows moving body contol by the
‘movement remote contro apparatus, In FIGS, 11 to 13, the
same procedures as in FIGS, 5 and 6 are provided with the
same reference signs, and the description will not be
repented. The hardware configuration ofthe present embodi-
ment isthe same asin the first embodiment
o
20
‘The present embodiment illustrates another example of |
the photographing control spparatus for moving body,
wherein the stability control is realized by cooperative
‘control through communication between the photographing
remote contro apparatus configured to remotely operate the
Photographing device and the movement remote control
apparatus contigured to remotely operate the moving body.
To increase the quality of an image obtained by photo-
araphing, control of the zoom, the exposure, the focus, and
the Tike tends to be complicated, and concentrating. on
handling the moving body becomes difficult. Therefore, the
present embodiment shows an example in wick different
persons separately operate the moving body, suchas a drone,
and the photographing device
FIGS. 14A and 14B are explanatory diagrams showing 3
simation of controlling the moving body. As shew ia FIG.
T4A, an operator 71 uses, for example, a right hand 72 0
rasp s housing 40 ofthe movement remote control sppa-
‘atus 40 and uses lft hand 73 to operate an operation lever
4405 Yor operating the moving hody 20, Note that as showa
in FIG. 7A, itis assumed thatthe photographer 61 operates
the photographing remote control appamitis 30 t0 control
photographing by the photogmphing device 10.
FIG. 14B shows an operation screen 7S displayed on a
splay screen 4 of thedisplay'section 44 ofthe movement
emote contro! apparatus 40. In the example of FIG. 14B, a
‘map display 76 indicating the position of the drone 50 is
splayed in the operation sereen 78, A map is displayed in
the map display 76, and a point display 76a indicates the
position of the drone $0, A message display region 77 is
raved inthe operation sercen 75, and messages according
{o operation of message display buttons 78 and 79 are
splayed in the message display region 7. The message
splay button 78 is a bution for displaying a message of
‘whether to transmit an availability response or to transmit an
“unavailability response Tor the movement of the center of
gravity ofthe photographing device 10. Inthe example of
FIG. 143, the display of “possible movement of center of
aravity” indicates that the availabilty response is to be
{ransmitted, The message display button 79 isa button for
splaying « display indicating the wind velocity in the
message display region 77.
‘The camera control of FIG. 11 is different from the camera
control of FIG. § in that the execution subject of each
process is the photogrephing remote control apparatus 30
(panier, the contr section 32) instead of the photo-
graphing device 10 (particularly, the contol section 12). In
the present embodiment, the photographing remote control
‘ppsints 0 and the movement remote contrl apparats 0
‘communicate with each other for cooperative conto, and
the judgement process of the drone communication in step
SM of FIG. 5 is not included in FIG. 11. The flow of FIG.
11 is provided with step S61 for communication wi the
‘movement remote contol appara 40, in place of step S23
of IG. §,
‘The flow of FIG. 12 shows control by the moving body
20, and steps S31 to S39 ofthe Now of FIG, 6 are adopted
Note that although state information is transmitted in
response 10 an information request from the movement
remote control apparatus 40 instep S71 of FIG. 12, the state
information may be trasmitted to the movement remote
ceontrol apparates 40 regardless ofthe information request.
FIG, 14 shows control by the movement emote contol
pparats. Steps S43 to S48 in the flo of FIG. Gare adopted
in the flow of FIG. 13. The moving body contol of FIG. 13
is different fom the moving body contol af FIG. 6 in that
the execution subject of exch process is the movemesUS 10,356,294 B2
Pi
emote contol apparatus 40 (particularly. the control sti
42) instead of the moving body 20 (particulary, the contol
section 22),
The contol section 42 of the movement remote contol
apparatus 40 requests the moving body 20 to transmit the
sate information in step SBI and receives the state infor
mation instep $82, In step S83, the contro section 42 judges
‘whether communication from the photographing remote
‘control apparatus 30 is done. When the result of the judge
ment ofthe change in the physical quantity is transmitted
from the photographing remote contro] apparatus 30 in step
S61 of FIG. 11, the contol section 42 feccives the trans
mission dat,
The state information from the moving body 20 and the
information related to the change in the physical quantity
from the photographing remote control apparatus 30 are
tiven to the contol section 42, and the control section 42
‘obtains the stable alteration condition in the moving body
20, Note thatthe contol section 42 may take the information
‘of the wind power into aecount in calculating the stable
alteration condition. Ifthe eontrl section 42 judges that the
‘change in the physical quantity due to the contol of the
movable section of the photographing device 10 by the
photographing remote coatrol apparatus 30 cannot maintain
the stability ofthe moving body 20, the contol section 42
shils the process to step S48, The control section 42
generates an unavailability response and displays the
“unavailability response on the display section 44
‘When the contol section 42 generates an availabilty
response, the control section 42 transmits, to the moving
body 20, te operation signal for controling the propulsive
{orce of the moving body 20 according tothe stable altens-
tion condition and transmits, 10 the photographing remote
control apparatus 30, the device state alteration contol
information including the stable alteration condition, The
‘control section 32 of the photographing remote contol
apparatus 30 controls the movable section of the photo-
tniphing device 10 while satisfying the stable alteration
‘condition,
In this way, the stability of the moving body and the
stability of photographing ean also be maintained in the
present embodiment when an operation of moving the
movable section of the photographing device is performed.
Note that the communication distance between the photo-
graphing device and the photographing remote control spp
ral is shorter than the comaunication distanee between the
‘moving body and the movement remote control apparats ia
some eases, Even in seh a eas, the photographing remote
‘contol apparatus and the movement remote control apps
ratus communicate With each other to perfoan cooperative
‘contol for maintaining the stability in the present embod
‘ment, and the contro canbe surely performed even when the
distance between the photographing remote control appara
tus and the moving body is relatively long. Allo par ofthe
remote control apparatus and the movement remote control
apparatus can be configured by smartphone in some cases,
‘The degree of fisedom of display is relatively high in the
smartphone, Therefore, the present embodiment has an
‘advaotage that a user-friendly GUI ean be provided when &
user operation is necessary for controlling the stability
Fourth Embodiment
FIGS, 18, 16, and 178 to 17C relate ta fourth embod
‘ment of the present invention. FIG. 18 is a flowchart for
describing contol adopted in the fourth embodiment, and
FIGS. 16 and 174 10 17C are explanatory diagrams for
0
o
2
describing the fourth embodiment, The hardware coatign-
‘ition ofthe present cibodiment is the same as in the first
embodiment
‘Even when the availabilty response is generated forthe
change in the physical quantity, sueh as the movement ofthe
center of gravity, of the photographing device 10, the
‘moving body 20 may be tilted to some extent in some cases
due to the movement of the center of gravity or the like of
the photographing device 10. The present embodiment
improves the stability of photographing even inthis case
‘The low of FIG. 18 shows a 200m photographing contol
‘ow executed in step 826 of FIG. § or 11, For example,
assuming thatthe process of step S26 is executed by the
control section 12 of the photographing deviee 10, the
‘control section 12 judges in step S91 of FIG. 18 whether the
‘moving body 20 is inclined equal to or more than a prede-
termined angle due to the movement ofthe center of gravity
or the like of the photogmphing device 10, The control
sections 12 and 32 acquire the information related to the
‘moving body 20 in step $23 of FIG. $ oF step S61 of FIG.
1 and the information also inclades information related
the inclination angle of the moving hody 20, The control
section 12 uses the information related to the inclination
‘angle of the moving body 20 acquired in step $28 to julge
‘whether the inctinstion angle of the moving body 20 is
changed equal to or more than the predetermined angle
FIG. 16 deseribes the inclination ofthe drone 80 provided.
with the photographing deviee 10, A wide-angle state 81
indicates a state in which the lens barrel 10a is extremely
contracted, and an angle of view 8a is in a widest state. An
intermediate state 2 isa state in which the lens barrel 1a
is extended a Title, and an angle of view 856 is in an
intrmdite state. telephoto state 83 isa state in which the
Jens barrel 10a is extremely extended, and an angle of view
cis in w narrowest slate. In the respective states 81 to 83,
the inclination angle ofthe drone $0 varies as indicated by
altemate fong and shor dash line. The inclination angle of
the state BI is the smallest, The inclination angle ofthe state
82 is Langer, and the inclination angle of the state 83 is the
largest
FIGS, 17A to ITC show picked-yp images in substantially
the same field of view ditcction picked up by the photo-
araphing device 10. FIG. 17A shows a picked-up image
‘when the photographing device 10 is fixed on the ground.
FIGS. 178 and 17C show picked-up images when the
photographing device 10 is attehed 0 the drone $0,
Now. itis assumed, for example, that picked-up images
91a to De shown in FIG. 17A are picked up in the
wide-angle state, the intermediate state, nnd the telephoto
state when the photographing, device 10 is fixed on the
sarund, The picked-up images 91a to 9c include sn image
ff ground 98, an image of cloud 96 floating inthe sky and
‘an image of an airplane 97 flying in the shy.
FIG. 17B shows picked-up images 92a to 92c when the
same photographing direction as in FIG. 17A is photo-
‘graphed by the photographing device 10 attached 0 the
Grone $0. The picked-up images 92a t0 92c are photo-
raphe in the whleeangle state 81, the intermediate state 82,
fand the telephoto state 83 of FIG. 16, respectively. The
mage 926 is an image of part of the image 92a, and the
image 92¢ isan image of part ofthe image 926. When the
‘mages are picked up on the telephoto sie, the fens barrel
109 ofthe photographing devi 10 extends forwaed, and the
position ofthe center of gravity moves forward. As a result
ff the movement, the drone $0 is in a state extremely
fnclined forward and downward as indicated by the tele-
photo state 83. As a result of the inclination, the image 2cUS 10,356,294 B2
23
js an image expanding the lower part of the image 926, and
most of the airplane 97 sticks out from the photographing
range
“Therefore, when the control section 12 detects that the
inclination ofthe drone 80 is changed equal to oF more than
the prodctermined angle in step SL in the present embodi-
‘ment, the conto section 12 switehes the optical zoom to the
‘electronic zoom instep $92. The control section 12 then sts
the angle of view according to the inclination angle of the
‘drone $0 in step $93. For example, when the front of the
“drone 50 js inclined downward, the image ofthe upper pat
js eu y the elecitoic 200m Toran angle of view equivalent
to the inclination angle
FIG. 17C shows picked-up images 930 to 9%c in the
swidesangle state 81, the intermediate state 82, and the
telephoto state 83 in this ease, The images 930 and 936 are
the same images as the images 82a and 92b, respectively.
“The image 93e picked up on the tlephoto side is obtained
by cutting a range of the upper part by the electronic 200m,
‘equivalent to the angle of view corresponding t the incl
nation angle of the drane 50. As a rest the airplane 97 is
surely photographed inthe image 93c as in the casein which
the photographing device 10 is fixed on the ground to pick
up an image,
In this way, even when the moving body is inlined, the
zoom is switched to the electronic 200m on the telephoto
side to cut the range corresponding tote inclination angle
if the inclination angle is equal to oF more than the prede-
termined range in the present embodiment. Stable photo-
araphing is possible regardless of the inclination of the
‘moving body.
‘Although a digital camera is used to describe the device
{or photographing in each ofthe embodiments, the camera
may be a digital singleslens reflex camera oF a compact