The 2017 ISEE

SMART SUBTITLING SOFTWARE FOR THE BETTER

READABILITY
Dang Thanh Tin(1), Le Tuan Vu(1), Van Dinh Trieu(1)
(1)
Bach Khoa University , Viet Nam
Email: dttin@hcmut.edu.vn

methods is in section 2, result analysis in section 3. The

ABSTRACT
conclusion will be in the section 4.
Nowadays, the worldwide expansion of the Internet
brings about the higher demand for subtitle video. 2. METHOD
Subtitle is a textual version of a videos dialogue or 2.1. Tool
explanatory that appears onscreen. Due to that fact,
OpenCV is a library of programming functions
developing a software that facilitates video makers in
mainly aimed at real-time computer vision. Its vast and
adding and editing their subtitles is imperative. A study
progressively expanded library aids us effectively in
has been conducted to build up the SmartSubtitling
image processing. It includes various functions such as
software. It aims at adding and processing the subtitle
drawing, video input/output, shape detection ect. As an
smartly by the ability to evaluate the color average value
open source tool, OpenCV is suitable for studying. In
of the subtitle area. Then it defines the subtitle color so
this study, OpenCV version 3.2.0 is utilized.
that they are the most readable.
This software is programmed in C++language which
KEYWORDS: subtitling, smart subtitling, opencv,
is the Object Oriented programming. The test run and
subtitle process.
algorithm testing are carried out on Visual Studio 2010.
1. INTRODUCTION
To build Graphic User Interface (GUI), Qt 5.6.2 is
For the reason that viewers sometimes get annoyed chosen. It is a C++ based framework of libraries and
at the classic subtitles which are virtually unreadable as tools that enables the development of powerful,
its color coincides with the backgrounds. This degrades interactive and cross-platform applications and devices.
video quality and causes the interruption in viewer
comprehension. 2.2. Theoretical basis and idea
There are availably some subtitling softwares such As regard to subtitling, every single video frames
as MKVToolnix or Captions and Subtitles function by must be observed. Each of them is encoded into a matrix
Youtube website. They all have their own methods for which the coordinates stand for an image element- pixel
the better readability. Commonly, they use the default (Fig.1).
font as white word with black edge. Youtube, personally,
sets their white text in black cells. Sharing the same aim,
the SmartSubtitling is introduced as subtitling software
with the advanced function.
Smart subtitling is a complex function that involves
creating, editing or adjusting subtitles so that they are
always readable. On the other words, its task is to keep
the subtitle and background colors distinguishable even
when the background changes consecutively in term of
color and brightness. The initial requirement is to
accurately define subtitle area in the background then
segment it to darkness and brightness so as to properly
Figure 1. Illustrating the pixel of an image [1]
process its colors. Discussing briefly on the applied

1
 The 2017 ISEE

Thence, it is able to insert text into the frame at a
specified pixel.
A character is composed by the line segments and
the curves. In deed, drawing the word is conducted as
drawing then connecting those lines and curves [3][6].
each character. To do so, changing the line into character
array is compulsory, with each cell in the array is a
character, including space.
After detaching the word, the following step is to
calculate the length of each character and divide the
rectangle into cells.

Figure 4. Detaching a word into seperated characters [1].

Figure 2. Each character is composed by the line

segments and curves
There are many available algorithms to draw the line
and curve such as DDA Algorithm, Bresenhams Line
Generation, Mid-Point Algorithm, Hence, a reference
was made to select the right one instead of going further
into designing this algorithm. Likely, the available
library offers drawing fonts abolishing line and curve
connecting step.
Figure 5. Seperating the subtitle area into cells [1].
Accordingly, A calculation on the average color
value of the cells containing the characters is carried on
which helps to define the essential color in each cell.
Normally the color is chosen in contrast with the
average color.

n 1
Color average value formula: C(i ) n
i
To C(i) is the color value of pixel place i, n is the
total pixels of one cell. Beside the content, time that the
subtitle is rendered is another important factor needed to
be considered. This is executed by defining frame
position of start point and end point, the subtitle must be
displayed in this interval. Frame position is defined by
the formula:
Position = fps * [Time point]
To fps is the frame per second or frame rate, [time
point] is the point of time observed.
The following diagram depicts the process overview.
[1]
Figure 3. Available word fonts in OpenCV library
Input Grab video Input text
As to preparation process; premarily, background video frame
area has to be defined on frame containing the subtitle,
then subdivided into cells. Each cell has its own certain Next Divide texts
brightness level that helps define its subtitle color. The frame background into
details on how the cells are divided are going to be cells
discussed. By entering input including text and the End of file
subtitle position on the frame, the background area
Output Draw each Calculate the color
containing subtitle can be defined easily. It is a rectangle video character average value of
surrounding the text. Next, the rectangle is divided into subtitled of the text each cell
cells in such a way that its length is equal to the length of
Figure 6. The process overview

2
 The 2017 ISEE

2.3. Algorithm
[2]
2.3.1. Displaying video in GUI
The player is totally designed basing on Frame
reading function in OpenCV instead of the available
media player function in Qt library. Signal and Slot
mechanism is a central feature of Qt. Signal is emitted
when a particular event occurs. A slot is a function of a
class that is called in response to a particular signal.
Signal v Slot are used for communication between
objects. When there is a connection, an object is an
emitter and another plays a role as a receiver. By default,
for every emission, the corresponding signal receiver
implements its slot. To connect signal and slot the
Connect command is used:
connect(object1, signal, object2, slot)
To signal emitter, the loop will be executed to read
every video frame. For each time, it splits the frame into
an image and save it to a variable called img; then emits
a signal attached the img:
void sendProcessedImaged(QImage img)
To signal receiver, a corresponding slot is set up to
receive image img:
void updateUIframe(QImage img)
[8]
Then the img is displayed on a label in GUI . For
every receipt, label displays the receiving image.
2.3.3. Subtitling
Thank to the sufficient OpenCV and C++ libraries,
the process becomes much more convinient.
To open the given video, using VideoCapture
constructor the following syntax [1]:
VideoCapture::Videocapture(const string& filename)
To read each frame in the video, using:
VideoCapture::read(Mat& image)
Subtitle is entered as the string.
Background area on frame containing the subtitle
has to be defined. In this case, the alignment position is
chosen-center and under the frame about nine tenths of
frame length. To any subtitle section with its certain font,
It is possible to define the subtitle strings format. It is a
rectangle surrounding the text which is the same length
with the string and the same size with text font.
Supposing x,y are the coordinates of the rectangles
right-below angle, frame_width and frame_height is the
width and the length of a video frame, x and y are
computable. From then, the frame points in the rectangle
are defined:
Entering the Subtitle

Time interval between the two signal emission is

1
calculated by the formula: T fps (sec) . To reach
high appropriation, the following formula is used instead:
T fps1 106 ( sec) .
subSize = Subtitle length
2.3.2. Subtitle storage
A class is used to show subtitle characteristics.
Currently, the three characteristics under consideration
are: Start point, end point and content. Vectors are used
as the container. The following table is an example: frame _ width subSize
x
Table 1. Subtitle characteristics 2
9
y frame _ height
10
Start End
Point Point Content
(sec) (sec)

0 10 13 Hello!
Set of pixels in subtitle area

1 15 18 Hi !
Figure 7. The algorithm used to define subtitle area
2 20 23 How are you today?
As to character splitting, in principle, the string is
3 25 27 Good. And you? the contiguous sequence of characters. Every elements of
the array must be referred then the character is assigned

3
 The 2017 ISEE

to a variable in type of char [5]. However, OpenCV library
doesnt support drawing char function, it is impossible to
draw characters to the frame. Therefore, an intermediate
step must be executed to exchange the characters from
char type to string. Here is the algorithm:
the interval is set from 0 to 176 which returns the white
color, and the rest interval which is between 176 and 255
returns the black color.
Defining the object area

Entering any S string

Trimming the area to a new matrix:

Mat
i=0

False
i, j, S, avg = 0
i < S.length

True
False
Char a = S[i] j Mat.rows

j = j+1
i=0 True

Covert a to string type

i Mat.columns

False

Return a True

S = S +

i = i+1

i = i +1

End

Figure 8. The algorithm used to split characters

Each split word is considered a string. A algorithm
to define background color of this string is performed
then its color average value is calculated. Smoothly, the
defined area must be trimmed into a particular image to
calculate its color average value. To avg is the average
value under determination, Mat is the matrix of trimmed
image, the rule is to calculate each pixels color value
then the average color. Because the calculation is
executed on greyscale [4][7], the average color drops in the
interval from 0 to 255. The algorithm is shown in Figure
9.
After calculating the average value, the condition for
character color in each cell is restricted. Subsequently,
S
avg
Mat.rows Mat.columns
Figure 9. Calculating the color average value of a
defined area.
A more friendly approach is to put the text and its
background colors in contrast (255 [the average value
of color]). However, this leads to the unreadability,
coloring coincidence in some cases. For example: when
the color average value is 125, the calculated color is
130, they are nearly the same.

4

3. RESULT ANALYSIS
A test run is made including opening a video then
inserting the subtitle Hello World !!!!
Figure 10. A subtitled video.
According to certain practical tests and user
feedbacks, in most case, the algorithm performs properly.
GUI is regarded to be nice, friendly and to fulfill the
initial idea. It allows input video in various formats such
as .mpg, .mp4 and .avi. With this version, video output is
only saved in .avi format.
Figure 11. An unreadable moment
However, there are particular cases that the
algorithm has some drawbacks such as the unreadability
in Figure 11. For such background, the color average
value is not appropriate which leads to the unreadability.
The two subtitle features start point and end point are
calculated by second. This causes the inaccuracy in
displaying time which affects the subtitle continuity.
4. CONCLUSIONS
The assignment on Smart Subtitle is completed in
all edges. The result is as good as expectation. However
there are some drawbacks in the process such as subtitle
quality, time-consuming process, output video format is
limited- AVI only.
In the coming version, the software will be designed
and developed to get more advanced function:
The 2017 ISEE
Higher accuracy algorithm.
Enable user edit their subtitles after storing.
Previewing.
Adding three more subtitle features: Font, size
and customized coordinates availably in the
GUI.
The subtitle will be automatically split into rows
if it width excesses the frame width.
Time accuracy will reach millisecond.
Video output has wider kinds of format
Sound adding for rendered video and output
video.
The GUI will be designed nicer and more
friendly.
ACKNOWLEDGEMENT
The author is grateful to ASS Prof., Dr. Dang Thanh
Tin for his valuable criticism of the draft of this paper.
REFERENCES
[1] G. Bradski and A. Kaehler, Learning OpenCV, 1st
Edition, OReilly Media, Inc, 2008.
[2] J. Blanchette and M. Summerfield, C++ GUI
Programming with QT 4, 2nd Edition, Prentice Hall,
2006.
[3] P. Shirley, Fundamentals of Computer Graphics,
2nd Edition, A.K.Peters, 2005.
[4] S. Johnson, Stephen Johnson on Digital
Photography, 1st Edition, OReilly Media, Inc,
2006.
[5] Splitting a string by a character. Retrieved from:
https://stackoverflow.com/questions/10058606/splitt
ing-a-string-by-a-character
[6] Line Generation Algorithm. Retrieved from:
https://www.tutorialspoint.com/computer_graphics/l
ine_generation_algorithm.htm
[7] John, Three algorithms for converting color to
grayscale. Retrieved from:
https://www.johndcook.com/blog/2009/08/24/algorit
hms-convert-color-grayscale/
[8] Efficient way to display OpenCV image into Qt.
Retrieved from:
http://www.qtcentre.org/threads/56482-efficient-
way-to-display-opencv-image-into-Qt