UNITED STATES PATENT AND TRADEMARK OFFICE

__________________________________

BEFORE THE PATENT TRIAL AND APPEAL BOARD

__________________________________

APPLE INC.,
Petitioner,

v.

RED.COM, LLC,
Patent Owner.
__________________________________

Case No. IPR2019-01064

Patent No. 9,230,299
__________________________________

DECLARATION OF THOMAS GRAEME NATTRESS IN SUPPORT OF

PATENT OWNER RED.COM, LLC PRELIMINARY RESPONSE

RED.COM Ex. 2001

Apple v. RED.COM
IPR2019-01064
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

I, Thomas Graeme Nattress, declare and state as follows:

I. INTRODUCTION

1. I am a lead camera systems architect at RED.COM (“RED”), the

assignee of U.S. Patent Nos. 9,230,299 (“the ’299 patent”) and 9,245,314 (the ’314

patent). I am also a listed inventor on the ’299 and ’314 patents, and an inventor

on over twenty-five additional issued patents assigned to RED. I am submitting

this declaration in connection with Patent Owner Preliminary Responses to

IPR2019-010164 and IPR2019-01065, filed by Petitioner Apple Inc., relating to

the ’299 and ’314 patents, respectively.

2. I executed a previous declaration, which was submitted on January 31,

2014, in connection with a reexamination proceeding on U.S. Patent No. 8,174,560

(the “’560 patent”), assigned to RED and for which I am a named inventor (the

“2014 declaration”). A copy of my 2014 declaration can be found at pages 312-

345 in Petitioner’s Ex. 1002 in IPR2019-01064. I understand that Petitioner’s Ex.

1002 in IPR2019-01064 is a copy of the prosecution history of the ’299 patent. A

true and correct copy of my 2014 declaration is also attached hereto as Exhibit

2024 for use and reference in IPR2019-01065, as my 2014 declaration does not

appear in Ex. 1002 (’314 patent prosecution history) in IPR2019-01065.

3. A detailed summary of my biography and credentials can be found in

my 2014 declaration at paragraphs 1-6.

-1-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

4. In my 2014 declaration, I testified regarding the inventions claimed in

the ’560 patent. In particular, paragraphs 10-28 in my 2014 declaration describe

the state of the art at the time of the filing of the ’560 patent, and the deficiencies in

technology and conventional thinking that were overcome by the inventions

claimed in the ’560 patent. Those paragraphs relate to the April 11, 2007 time

frame, which is the earliest filing date to which the ’560 patent claims priority.

The ’299 and ’314 patents claim priority to the ’560 patent, and to the same April

11, 2007 provisional application as the ’560 patent. Thus, paragraphs 10-28 in my

2014 declaration refer to the deficiencies in technology and conventional thinking

that were overcome by the inventions in claimed in the ’299 and ’314 patents.

II. REDUCTION TO PRACTICE OF THE

RED ONE MOTION PICTURE CAMERAS

5. I first met Mr. Jim Jannard in December of 2005. At that meeting, we

discussed his desire to create the first ever digital motion picture camera that could

record compressed digital motion video at cinema quality levels, including 4K. I

was intrigued by the possibilities, because combining the ease and flexibility post-

production of digital video while maintaining cinema-quality frame rate and

resolution would be a game-changer in the world of movie making. But I also

knew that digital compression was highly disfavored in the movie camera industry

-2-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

due to its resulting artifacts and lower resolution, which were unacceptable for big-

screen cinema viewing.

6. To solve this problem, one key area I researched was the use of an

image sensor with a Bayer-pattern filter. This was one of several unconventional

avenues we explored at RED. At the time, such sensors were associated with

lower-quality consumer-grade video cameras and derided as incapable of providing

cinema-quality video due to artifact and resolution issues. In contrast, the industry

consensus held that cinema quality cameras would need to utilize three sensors,

with a prism to split red, green and blue light to each sensor. However, we

believed that the benefits of a Bayer-pattern image sensor could be optimized if the

image data remained in raw, mosaiced format for compression. We believed that

such a data workflow could allow the raw digital files to operate as a digital

negative, and would provide all of the post-production flexibility of being able to

manipulate the original raw data.

Developing REDCODE for the RED ONE Cameras

7. Immediately following my December 2005 meeting with Mr. Jannard,

I began working on the design of RED’s first commercial digital motion picture

camera that would become known as the RED ONE. This work would last all

throughout 2006 and into 2007 when we commercially launched the RED ONE

-3-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

video camera. My title on the RED ONE project was Problem Solver, and it

remains my title to this day.

8. As I explained in paragraphs 13-21 of my 2014 declaration, the

typical flowchart for prior art video image data capture and processing involved

demosaicing video data captured by a Bayer-pattern sensor prior to compressing

that data for storage. Paragraphs 22-28 of my 2014 declaration explain how we

rejected that conventional thinking by implementing known compression

techniques, such as JPEG 2000, in such a way as to operate on non-demosaiced

Bayer-pattern image data, a type of image data these techniques were not designed

to work on.

9. Accordingly, my work on the RED ONE camera was based on a video

image processing pipeline that did not include a conventional demosaicing step

prior to compression. Rather, the data flow upon which I based my research and

design work for the RED ONE camera generally operated along the following

four-step sequence for processing and compressing raw Bayer-pattern video image

data: First, raw image data was collected by the image sensor, which was a

Mysterium CMOS image sensor chip with a Bayer-pattern pixel filter, i.e., a sensor

chip that detected only one data value for each of the green, red and blue pixel

locations. The raw mosaiced data was outputted by the image sensor at a

resolution of at least 2K and at least 23 frames per second. Second, the raw Bayer-

-4-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

pattern image data was then sent to a Xilinx processing FPGA chip for pixel

correction and processing of the raw Bayer-pattern image data. Third, the pixel-

corrected and processed raw Bayer-pattern video image data was then sent to

Analog Devices compression chips, which utilized a mathematically lossy wavelet

compression codec known as JPEG 2000, to compress the processed raw Bayer-

pattern video image data. Fourth, the compressed raw Bayer-pattern image video

data was sent to a memory device, by way of a SATA port, for storage as a raw

data file that at one point in development was given a .JIM file extension in

homage to Mr. Jannard and the new file type that RED had created. We referred to

the above programing for the RED ONE cameras, and the resulting raw

compressed data files that it generated, as REDCODE.

10. Much of my research and design work that went into the RED ONE

focused on the second step above, i.e., how to process the raw Bayer-pattern video

image data before compression in way would optimize the decompressed and

demosaiced output. This was a key step in REDCODE because, without pre-

processing, JPEG 2000 compression was otherwise incapable of processing raw

Bayer-pattern image data in a useful way. To solve this problem, I devised and

tested a number of methods for processing the raw Bayer-pattern video image data

prior to compression. I ultimately devised two methods in particular that were

-5-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

employed on the RED ONE cameras, referred to as GAS (green average

subtraction) and pre-emphasis.

11. The GAS processing technique uses an averaging of green pixel

values to produce low noise, co-sited estimated green values. By way of

background, the highest quality demosaic algorithms work on a large kernel to

provide high quality estimates of edge direction. The edge direction is perturbable

from changes to the data in the input image data to the demosaic algorithm to a

much greater extent than the small local average used in the GAS method. In the

GAS method, green pixels located adjacent or in close proximity to a target pixel

are averaged to calculate an average green value, which is then subtracted from the

value of the target pixel, whether the target pixel is red or blue. For example, a

distinct average green value may be calculated for each target pixel location, then

those distinct average green values are subtracted from the values of the

corresponding distinct red and blue pixels. This subtraction can result in a large

number of zeros, for example, in areas of an image that are black, white or shades

of gray; where the values generated by the image sensor for all three colors are

equal.

12. The pre-emphasis processing technique can be used to help preserve

image detail. This technique helps tune the compression of pixel values through a

pre-emphasis curve that changes the gradient in different sensor code value regions

-6-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

to either emphasize or de-emphasize those pixel values relative to others in the

image. For example, this technique can be used to spread-apart (increase gradient)

pixel values in some ranges of values and squeeze-together (reduce gradient)

values in one or more other ranges. The spreading-apart of values preferentially

preserves the variations in values, and thus image details, in the ranges where the

values are spread apart so that the corresponding details better survive the

compression process. In one implementation, the pre-emphasis curve is set to

spread apart pixel values corresponding to the darkest image regions, whereas the

relatively brightest image regions were squeezed together. This allowed for

compression that better preserved the details in the darkest areas of the image,

which can otherwise often be washed out. After sufficient testing to find the most

ideal curve, keeping in mind the unfixed rendering intent of the raw image data, I

settled on a simple power law curve with adjustable black offset. This curve

resulted in low complexity and a single tuning variable, among other benefits.

13. The REDCODE programming described above formed the backbone

of the RED ONE cameras, including the two RED ONE cameras known as Boris

and Natasha.

The Boris RED ONE Motion Picture Camera

14. RED’s work refining and debugging the RED ONE continued

throughout 2006 and into 2007. With the goal of debuting the RED ONE at the

-7-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

upcoming NAB 2007 show taking place in Las Vegas on April 14-19, 2007,

several RED team members tested a RED ONE motion picture camera, nicknamed

Boris, on March 8, 2007 at RED’s Lake Forest, California, headquarters. I was

present for that testing, as was Jim Jannard, Jarred Land, Ted Schilowitz, David

Macintosh and Stuart English, among others. I documented the event by taking

photos with my personal camera. True and correct copies of seven of those photos

are attached hereto as Exhibits 2002-2008.

15. To confirm the dates of the photos that I took of the testing of Boris, I

ran ExifTool on the photo files to collect the metadata for the photos. Attached

hereto as Exhibit 2009 is a true and correct copy of the output of the metadata of

files on my camera, including those relating to the photos those I took of the

testing of Boris at RED headquarters. Exhibits 2002-2008, respectively,

correspond to IMG_4291, IMG_4327, IMG_4317, IMG_4338, IMG_4354,

IMG_4364 and IMG_4365, which all indicate a “Date/Time Original” of March 8,

2007. As an aside, the timestamp associated with the “Date/Time Original”

indication reports time for the Eastern time zone, which corresponds to my home

time zone at the time in Ottawa, Canada.

16. Exhibit 2002, reproduced below, shows Jim Jannard inspecting Boris.

-8-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

17. Exhibit 2003, reproduced below, shows, from left to right, David

Macintosh, Jim Jannard, Ted Schilowitz, Jarred Land and Stuart English standing

next to Boris after it had been set up on a tripod to conduct test footage.

-9-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

18. Exhibit 2004, reproduced below, is a close-up shot of Boris on the

tripod. David Macintosh can be seen in the background. This photo also shows a

wire running from Boris to a live monitor (out of frame) on which we viewed the

live feed from Boris. Also visible in this photo is the internal framework for

Boris’s hardware circuitry, as Boris’s cover is removed. This photo also shows

that a section of Boris between the body and lens had a black powder coating,

while the remainder of the exterior was milled aluminum.

-10-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

19. Exhibit 2005, reproduced below, is a photo looking head-on into the

front of Boris on the tripod. In the background of the photo, from left to right, is

David Macintosh, Jarred Land and Ted Schilowitz. The lens of my camera is

visible in the reflection of the lens on Boris. The reflection appears to show me

adjusting the focus of my camera as Jarred Land simultaneously adjusts the focus

on Boris.

-11-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

20. Exhibit 2006, reproduced below, shows Ted Schilowitz posing next

to a ColorChecker chart, also referred to as a Macbeth chart, while Boris is

recording video of the shot. In the upper-right-hand foreground, an image of Ted

Schilowitz can be seen in the monitor connected to Boris showing the live feed of

the video.

21. After recording test footage from Boris, we took the compressed raw

Bayer-pattern video image data that Boris had saved to file, and played it back on a

computer in decompressed and demosaiced form to examine the results. Exhibit

-12-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

2007, reproduced below, shows Ted Schilowitz reviewing the output of the Boris

test footage taken of himself.

22. The results of the test footage were a resounding success. Visual

inspection of the playback of the video files recorded by Boris confirmed that it

had recorded compressed raw mosaiced video image data that had been outputted

in visually lossless form with no visible compression artifacts after decompression

and demosaicing.

23. While the visually lossless nature of the Boris test footage was evident

upon visual inspection alone, confirmation of its high resolution and frame rate

was also confirmed by the computer program used to display the Boris video

-13-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

footage. Although the output parameters of the test footage of Ted Schilowitz

cannot be seen in the photo above, a photo of the output of test footage taken of

David Macintosh provides a clearer image. That photo, Exhibit 2008, is

reproduced below with a yellow circle annotation to indicate the output

parameters.

-14-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

24. Below is an enlargement of the output parameters contained in the

annotated circle in Exhibit 2008.

The output parameters read as follows:

Format Settings:

Format Custom: W: 4096 H: 2048

Aspect Custom: Scale: 1.0000

Framerate: 24.000

Show: Full res. High.

-15-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

At the top of the enlargement, the “.jim” file extension we used to denote

compressed raw Bayer-pattern video image data created by REDCODE can be

seen.

25. These parameters report an output resolution of 4096 x 2048, which

denotes 4K resolution. The framerate is reported as 24 frames per second. Based

on these parameters, and my inspection of the visually lossless video shot by Boris,

I knew by at least March 8, 2007 that our invention would work for its intended

purpose. My understanding was then further confirmed later in March when Sir

Peter Jackson successfully used Boris and its companion camera, Natasha, to shoot

his “Crossing the Line” film.

The April 2, 2007 Release Note for Boris and Natasha

26. As part of my research and development of REDCODE, I instructed

Wind River on a regular basis and conveyed how REDCODE needed to operate on

the RED ONE cameras.

27. Wind River would routinely provide documentation detailing the

progress of the tasks we had assigned them. Attached as Exhibit 2010 is a true

and correct copy of a Release Note, dated April 2, 2007, that Wind River provided

to RED in connection with the Boris and Natasha cameras. This document was

typical of the documentation that Wind River would generate on the RED ONE

-16-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

project, and provide to RED on or near the date indicated on the cover. RED has

maintained a copy of Exhibit 2010 since the RED ONE project.

28. The cover of the April 2, 2007 Release Note states that it was

“Prepared for Diamond.” Diamond was Wind River’s code name for RED. The

cover also indicates that the document is a “Release Note” for “Sundance.”

Sundance was Wind River’s code name for the RED ONE. Page four of the April

2, 2007 Release Note indicates that the “released cameras are designated as

‘Natasha’ and ‘Boris.’”

29. Page five of the April 2, 2007 Release Note states, “Captured .jim

files were verified by Diamond’s Graeme and David Macintosh to produce

expected quality and the expected results from the new RED/BLUE pre-emphasis

feature.” I am the Graeme referred to in this sentence. The “new RED/BLUE pre-

emphasis feature” refers to an adjustment to the pre-emphasis functionality already

running in REDCODE on Boris and Natasha. The “.jim” files are the compressed

raw Bayer-pattern video image data produced by REDCODE running on Boris and

Natasha.

Huffman Compression

30. As mentioned previously, the RED ONE cameras, including Boris and

Natasha, relied upon JPEG2000 compression, which is a mathematically lossy

compression technique. Huffman compression, as recited in the claims of the ’299

-17-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

patent, refers to a lossless compression technique that has been well known for

many decades.

31. A lossless compression technique provides a lower compression ratio

than lossy compression. While this difference impacts the write speed and

memory on the hard drive, it does not lower the visual quality of the output upon

decompression. Because the REDCODE on Boris and Natasha produced visually

lossless output with a lossy compression technique, a person of ordinary skill in the

art would know that RECODE would successfully produce visually lossless output

using a lossless compression routine like Huffman compression. This result

follows necessarily because a lossless compression technique removes any

potentiality for compression artifacts caused by lossy compression.

I declare that all statements made herein of my own knowledge are true and

that all statements made on information and belief are believed to be true, and

further that these statements were made with the knowledge that willful, false

statements and the like so made are punishable by fine or imprisonment, or both,

under Section 1001 of Title 18 of the United States Code.

-18-
Apple v. RED.COM
Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065

Executed on August 15, 2019 at Acton, Ontario, Canada.

30703838

-19-