GOES Satellite Hunt

(Part 1 – Antenna System)

So few people know that I started a crusade against GOES 13 Satellite.
My idea was to capture the GOES 13 signal (that’s reachable in São
Paulo) with a good SNR (enough to decode) and them make all the toolkit
to demodulate, decode and output the images and other data they send. I
wanted a high-res image, and the L-Band transmissions usually provide
that (GOES for example is 1km/px with whole earth sphere in frame. A
10000 x 10000 pixels image)

So I choose GOES over other Weather Satellites mainly because GOES

is a GeostationarySatellite. That means its position never change. That
was needed for me, because L Band usually needs a relatively big dish to
capture the signal, and if the satellite is moving, the antenna needs to
track it. That means: Alt-Az tracker (or something else) that will be most
likely more expensive than the whole capture system (at least in Brazil).
Since GOES does not move, I could just point my dish and forget about it.
It would always capture the signal.

So first things first. What dish

should I get?
I found some people in twitter ( More
specific @USASatcom and @uhf_satcom and just for historic
reasons: https://twitter.com/lucasteske/status/766484223431770112 )
that did Satellite Receiving and more specifically USASatcom also did
GOES receiving. And from time to time posted pictures like this:
“It’s been a while so I thought i would post a nice full disk false color

image from GOES-15 – just before mid-day.”

And then I found the IRC Channel called #hearsat in server StarChat (
irc.starchat.net ). So a lot of people actually helped me by giving me the
information I need to take a decision over which dish antenna should I
buy. They were suggesting a 1.2m offset dish (same time of the Sky,
Claro, and some other TV dishes, but bigger) for receiving GOES. Sadly I
couldn’t find a offset dish bigger than 90 cm here. So someone suggested
that I need at least 1.5m prime focus (the “old” type of TV dishes) that
should have equivalent surface area than a 1.2m offset dish. So I went to
Santa Efigenia Street (people from São Paulo will recognize) and went
to Sat Imagem Store. Since my father and I was already friends with a
seller, I asked what dishes they had to sell and said that I want something
around 1.5m. He said that he has a 1.9m dish to sell.

So I bought the dish, costed about R$200 (that’s about US$60) and also
some cables and adapters. So since I was very excited to get things
working, I asked my father’s help to assemble the antenna. I arrived
home about 5 PM so it was very late to assemble the antenna. So I let it
for the day after.
Assemble Process
So the assemble process was sort of straightforward. It was not easy to
do, but even without a manual (that did not came) you should be able to
figure out what piece goes where. We were in 4 persons, we started just
after lunch (about 1PM) and took us about 6 – 7 hours to get it
assembled. LOL

I also decided to inject Polyurethane Foam into the dish tube, so when it
rains it doesn’t get filled with water. The only tip I have is: Be careful, your
dish can grow balls.

So the dish assemble is done. Now we need to worry about the feed.

Dish Feed (The actual receiver)

So I did several tries for like one month and half. The guys in #hearsat
and me was struggling to find what was the reason that I couldn’t get the
signal. I have all pictures of my failed feeds
here: https://www.flickr.com/photos/energylabs/albums/72157675387388
535. I will not detail each one of the assembles of these feeds because it
didn’t work. So for historical reasons as well the reason why they didn’t
work is because of the illumination angle of the Helical Coils. Usually the
Helix Feeds have a very narrow beam width (and high gain), this causes
to only some portion of the dish to be actually used by the feed. I only
discovered that when I drawed my dish in FreeCad and did some
auxiliary lines to show the angles. Then I noticed that something was
REALLY wrong: Only like 60cm of my dish was actually illuminated, and
from this 60cm only 40cm was actually visible by the dish (we need to
discount the dead center). This is the drawing I did. So mybit from hearsat
suggested me a Wave Guide Feed. For those that used to hack wireless
systems, this is known as CanAntenna or Cantenna. Its very simple to do
and I found this calculator to get things better:

http://www.changpuak.ch/electronics/cantenna.php

So the image (from the site) is this:

So if you check, the only thing that actually uses your target wavelength is
the linear feed size (that uses lambda / 4). The other ones
use lambdag that is the guide wavelength. The guide wavelength is
calculated from the can diameter. So I just found a can here (Neston Can
for those who want to buy in São Paulo) that has basically “square” size.
The can has 12cm diameter and 12cm height. You can use basically any
size of cans. Just a few things are important using that calculator:

1. Keep your target frequency over TE11 and below TM01.

2. The length of the can should be at least 0.75 * lambdag. Don’t
worry if your can length is higher than that. Just try to keep close.
3. The probe length (linear feed) counts from the base of the
connector (where you put the screws.

More details about TE11 and TM01 are

here: http://www.daenotes.com/electronics/microwave-radar/waveguide-
modes-of-signal-propagation

In my case the parameters for the calculator was:

Freq. of operation [MHz] 1692

Can Diameter [mm] 120
Cut-Off Freq. for TE11 Mode [MHz] 1464.15
Cut-Off Freq. for TM01 Mode [MHz] 1912.38
Guide Wavelength [mm] (λg) 153.89
λg/4 [mm] 38.4725
0.75 * λg [mm] 115.4175
Wavelength λ/4 [mm] 44.325

These waveguides usually have really wide bandwidth ( mine can

probably get anything from TE11 to TM01 modes ). So the sizes are not
that critical. Even so, I would keep the probe (the linear feed) length close
to few mm of the lambda / 4 of the target frequency.

The LNA and Filter

So in my failure setups I was having a REALLY huge noise from the GSM
Band at 1800MHz. That was: even without any LNA I could get like 10dB
of GSM signal. So mybit suggested me to use some filtering to wipe out
this signal. So I bought some Lorch Filters (recommeded and sold by him)
that has a center frequency of 1675MHz and 150MHz bandwidth (so from
1600 to 1750 MHz).
I also bought 3 LNA4ALL (one with connectors, two without) to use in this
and other projects.

So if you want to build this as well, I strongly suggest to buy some

LNA4ALL from Adam (its the manufacturer). They’re used by a lot of
people around the world and they’re high end ones and relatively cheap
(not for us in Brazil, but he sends in registered letter, so it never gets
taxes and arrives in two or three weeks). More
info: http://lna4all.blogspot.com/

I also bought some aluminum boxes (in Santa Efigenia as well, but now
in Multcomercial ) for putting these LNA and filters. I made some tests
with the topology Feed -> LNA -> Filter, but it looked like the GSM
Signals was overloading the LNA. So I decided for a non-optimal setup
that is Feed -> Filter -> LNA that gave me what I want. So them I just
tried to fit everything inside a box and here is the result:

Then I finished by gluing in the CanAntenna:

And that it!

Positioning the Antenna

So the last thing was to position the antenna. I actually did this before the
step of the feed (since I tested several other feed types). But if you’re
planning to build everything this should be the order that you do the stuff.

So in the past it would been tough to position an antenna, but today we

have cellphones to help everything. One application I use in Android
is Satellite AR. Another good one is Pointer Antena.

Satellite AR has a preset list of Satellites (including GOES), and Pointer

Antena allows you to fill with the coordinates of the satellite. Both are
good and enough for positioning, just be carefull that your compass
needs to be calibrated.

Another solution would be use GPredict or Orbitron to get the Elevation

(The inclination in relation to ground) and Azimuth (the bearing in relation
to North) angles and position manually. I noticed that for my dish there is
a working band of about 4 degrees of Azimuth and Elevation that the
signal doesn’t change. I would recommend get your laptop / cellphone
and hookup to the feed and check where you get the best signal.
Other two things that you need to care is related to the feed itself. Since
we have a linear polarized feed, the rotation of the feed matters in relation
to the satellite. This is less critical and I found that even 15 degrees
rotation doesn’t change a lot the ssignal.
ignal. Other thing is the focal point of
the dish. You need to adjust the feed distance from the base of the dish to
also get the best signal. Mine has the probe (the linear feed inside the
can) in the focal point (so the can opening is far ahead). After a all, you
should get something like this (got with airspy, 10MHz bandwidth):
Continuing
So the next step is the whole decoding process. At the time of this writing
I did not finish the whole demodulate ->
> decode process. For now I got
the file output from GOES and need to decompress and convert the
images, so I’m close. Once I finish, everything should be published in a
Open Source License and I will create a new post here describing the
whole process.

1. side (Simple BPSK Demodulator)

(Part 2 – Demodulator)
This is the LRIT Specification (theoretically):

Demodulator in GNU Radio

So first things first. We successfully acquired a LRIT signal from GOES.
Our first step is to demodulate the baseband signal from a SDR like
Airspy or RTLSDR and transform this to a symbol stream. For the easy of
use, I will use GNU Radio for assembling the demodulator.

According to the spec the LRIT signal is a BPSK (Binary Phase Shift
Keying) modulated that has a symbol rate of 293883 symbols /
second. The first thing is: How a BPSK Modulation works?

First we have two signals: The Carrier Wave and the Binary Stream that
we want to send. The Carrier Wave is a Sine signal that will carry the
modulated data along the path. It’s frequency is usually way higher than
the
e bitrate or symbol rate. In BPSK Modulation what we do is
basically change the phase of the signal. Since its a binary phase shift,
we basically only invert the phase of the signal, and that is basically invert
the local polarity of the wave. You can see in the picture below that when
we change the polarity we generate “notches” over the carrier
wave. That’s our bits being modulated into the carrier wave using a binary
phase shift. The advantage about changing phase is that we only need to
have a consistentnt phase in the receptor, not necessarily a very huge
amplitude of the signal.
BPSK Modulation

Basically what we have is:

 When our bit is 1, our carrier wave phase shift is 0 degrees.

 When our bit is 0, our carrier wave phase shift is 180 degrees
(inverted)

Simple isn’t it? So we can represente also our data in a phase

phase-diagram by
having two unitary vectors I and Q. Below there are some representations
of Phase Diagram for BPSK (a), QPSK (b) and 8 8-PSK (c).
Phase Diagram for BPSK (a), QPSK (b) and 8-PSK (c)

So for BPSK in our phase diagram we basically have: for Positive I

(quadrants 0 and 3) we have bit 0, for Negative I ( quadrants 1 and 2) we
have bit 1.

Just for reference, in QPSK (b) we would have a gray coded binary
values:

1. Quadrant we would have 00

2. Quadrant we would have 01
3. Quadrant we would have 11
4. Quadrant we would have 10

Same works for 8-PSK,

PSK, but with 3 bits instead of 2. Usually the binary
code is gray-coded
coded starting from 0 and increasing counter
counter-clockwise.
clockwise.

Demodulating BPSK Signal

How to make the reverse process (a.k.a. demodulate the signal)?

First we need to make sure that we have our carrier in the base band
(a.k.a. center of our spectrum). You might think that just tunning at
1691MHz will make sure of that, but there are some factor
factorss that we need
to account for:
 1. The SDR Tuner Cristal is not 100% precise, and there will be some
deviation from the actual signal.
2. The temperature change over the SDR will also lead to deviation
on the signal.
3. Wind, atmosphere and other factors can doppler shift the signal.

How to do it then? We need to find where our signal really is, and
translate to the base band (center of the spectrum). But we have a
problem: The LRIT signal doesn’t actually contains the carrier wave, just
the modulated signal (so its a signal that is a consequence of the
modulated carrier wave).

If we had a Carrier Wave in the signal we could use a simple PLL (Phase
Locked Loop) to track the carrier wave and translate the signal. But
Carrier Waves also consume transmission power so the total
transmission power (from the satellite) would be “wasted” in a signal just
to track. For carrier-less
less signals like LRIT we need to use a more complex
PLL system. There are several that can do that, but I will use the famous
Costas Loop as suggested by Trango in #hearsat.

Costas Loop Basic Diagram

Costas Loop is basically a double PLL system. The main idea is: If our
supposed carrier is not centered, there will be a phase error that will
accumulate over the time in the Quadrature Signal. So what we do? Use
this error to translate our signal to the base band. The actual details of
how the costas loop works will not be discussed here, but there are plenty
of documents explaining how it works in the internet. For a BPSK we will
use a 2nd Order Costas Lo Loop.
op. There is also the 3rd order costas loop for
QPSK and 4th order costas loop for 8 8-PSK.
Ok, after that we need to recovery a second information: The Symbol
Clock. The thing is: we modulated a binary signal over the carrier wave,
but if we have like a string of 10 bits one, how we would know that the
string has 10 bits, and not 5 or 7? We will need to recover the original
clock using a M&M (Mueller and Müller) algorithm. But before that, how
can we even recovery such thing?

Why we can recovery the original data clock?

We can recovery the original clock because we have two information:

1. The estimated symbol rate (in this case 293883 symbols / s)

2. The carrier phase transitions between 0 to 1 and 1 to 0

How can we recovery with just that information? Simple, we can do a

oscillator (clock generator) in the estimated symbol rate, and synchronize
with one carrier phase transitions. Then we can assume that we have a
synchronized clock. Also because of the randomization of the data (that
will be shown further in this article), the binary data will almost be random,
that means that we will have basically the same odds to have bit 1 or 0.
This will make our clock sync more consistent over time. Also M&M do an
additional thing that is the clock frequency correction. Since the symbol
rate is just an estimate, that means the value can change a little over time
so M&M can correct that for us.

After that we should have our symbol in the I vector of our IQ Sample.

GNU Radio Flow

So let’s get our hands to GNU Radio and build our demodulator. So in
GNU Radio we will have some additional blocks along the Costas Loop
and M&M Recovery. I will be using Airspy R2 / Mini so the values are for
them, but I will soon provide also a version for RTLSDR (that will just
have different values for the first steps). As suggested by Trango in
#hearsat, is best to run the sample rate a bit low in airspy to avoid USB
Packet Losses (to be honest I never had any USB packet drop with airspy
or hackrf, but this will heavily depend on the CPU and USB Controller that
you have on your machine. So its better to be safe than sorry.). For
Airspy R2 we will use 2.5 Msps, and for Airspy mini we can use 3 Msps.
Our target sample rate for the whole process is 1.25Msps (actually we
can use anything near that value). So let’s start with the osmocom block:
Osmocom Source

Let’s set the Sample rate to 3e6, the Center Frequency to 1691e6 and all
gains to 15.. For the gains setting you can experiment your own values,
but I found that I get the best SNR with everything maxed out (thats not
very common). Also Osmocom Source has a “bug” for airspy, that is it
doesn’t get the Mixer Gain Available (just because its not BB gain. That’s
stupid.). I made a patch (that was actively rrejected
ejected because of the Gain
name) to map the Mixer gain to BB gain (as it is for RTLSDR). In the
future I will probably do a new GRC Block to use with airspy with the
correct names and stuff, but for now you can compile gr gr-osmosdr
osmosdr from
source code using my fork: https://github.com/racerxdl/gr-osmosdr
osmosdr.

Decimating and filtering to desired

sample rate
The next step is to decimate to reach the 2.5e6 of sample rate. For the
airspy mini that is 15/18 of the 3e6 sample rate. So lets create a Rational
Resampler block and put 15 as interpolation and 18 as decimation. The
taps can be empty since it will auto
auto-generate.
generate. This is not very optimal, but
will work for now. I will release a better version for each SDR in th
the future.
Now we have 2.5 Msps and we need to decimate by two. But we will also
lowpass the input to something close our rate. So let’s create a Low Pass
filter with Decimation as 2
2, Sample Rate as 2.5e6, Cut Off Frequency
as symbol_rate * 2 (that is 587766), Transition Width as 50e3.
50e3

After that we will have the sample rate is 1.25e6

Automatic Gain Control and Root

Raised Cosine Filter
For better performance we should keep our signal in a constant level
regardless of the input signal. For that we will use a Automatic Gain
Control, that will do a Software Gain (basically just multiply the signal)
that does not change the resolution (so it will not give a better signal) but
will sure keep our level constant. We can use the simple AGC block from
GNU Radio.

With rate as 10e-3, Reference as 0.5

0.5, Gain as 0.5, Max Gain as 4000.
4000

Another step is the RRC Filter (Root Raised Cosine Filter). This is a filter
optimized for nPSK modulations and uses as a parameter our symbol
rate. The Filter is not very hard to generate (its a FIR with some specific
taps), but luckily GNU Radio provide a block for us.
RRC Filter

For the parameters we will use 1.25e6 as sample rate, 293883 as

Symbol Rate, 0.5 as Alpha
Alphaand 361 as Num Taps.. From these
parameters, Alpha and Symbol Rate is provided from the specification.
The number of taps you can experiment with, but I found that a good
balance between quality vs performance was at 361 taps. After that we
should have basically a signal that contains only the BPSK Modulated
signal (or
or noise in the same band). Then we can go to Synchronization
and Clock Recovery.

Synchronization and Clock Recovery

As I talked before, we will use 2nd Order Costas Loop as Carrier Wave
Recovery (synchronization) and M&M Clock Recovery to recover the
Symbol Clock. GNU Radio provides blocks for both algorithms. Let’s start
with Costas Loop.

Costas Loop

For the parameters we will only need 0.00199 as Loop

Bandwidth and 2 as Order
Order.. After that we should have our virtual carrier
in the base band. Now we only need to synchronize our samples with
clock using M&M Clock Recovery.
M&M Clock Recovery

For the M&M Parameters we will use Omega as 4.25339 that is basically
our symbols per sample rate, or sample_rate / symbol_rate. That is the
first symbolrate guess for M&M. For Gain Omega we use (alpha ^ 2) / 4,
that is alpha = 3.7e-3,
3, so our Gain Omega be 3.4225e
3.4225e-6, Mu as
0.5, Gain Mu as alpha (or 3.7e
3.7e-3), Omega Relative Limit as 5e-3.
5e

So you can notice that I called a new parameter alpha in M&M that is not
a direct parameter of the block. That alpha is a parameter to adjust how
much the M&M clock recovery can deviate from the initial guess. You can
experiment with your own values, but 3.7e
3.7e-3
3 was the best option to me.

Now at the output of M&M we will have our Complex Symbols pumped
out with the correct rate. Now we only need to extract our values.

Symbol Output from GNU Radio

So we could directly map it to Binary data now, but for reasons that I will
explain in the next part of the Hunt, I will keep our symbols as it is and
just convert to a Byte. So basically our output from GNU Radio will be a
signed byte that can vary from -128 to 127, being -128 128 as 100% chance
to be bit 0, and 127 as 100% chance to be bit 1. And the intermediate
values as the corresponding cchances.
hances. Basically I will have a byte that will
represent the probability of a bit being 0 or 1. I will explain more in the
next part of this article.

For now what we need to do is to get the Complex output from the M&M,
get only the Real Part ( Component I ) transform to byte and output (to file
or TCP Pipe).

Thats a simple two block operation. First we use Complex to Real block
that will output a Float with the Real component of the complex number,
and then convert to char multiplying by 127 (since the Complex is
normalized). After that we can use File Sink to output to a file or create a
vector stream to output to a TCP Socket. In my case I will use TCP
Socket.
The Stream to Vector just aggregates every 16 bytes before sending
through TCP. This will reduce the TCP Packet Overhead. The TCP Sink
parameters are:

 Input Type: Byte

 Address: 127.0.0.1
 Port: 5000
 Mode: Client
 Vec Length: 16

With this parameters, when we run the GRC Flow, it will try to connect to
localhost at port 5000 and send every group of 16 bytes (or if you prefer,
symbols) through TCP. The next part of this article will deal with the
software part to decode this and generate the packets for creating the
output files. Below you can check my final flow. It also has some other
blocks to have e a better flexibility changing the parameters and also to
show the waterfall / fft of the input signal as well the constelation map.
(Part 3 – Frame Decoder)
In the last chapter of GOES Satellite Hunt, I explained how I did the
BPSK Demodulator for the LRIT Signal. Now I will explain how to decode
the output of the data we got in the last chapter.

One thing that is worth mentioning is that most (if not all) weather
satellites that transmit digital signals use the CCSDS standard packet
format, or at least something based on it. For example this frame decoder
can be used (with some modifications due QPSK instead BPSK) for
LRPT Signals from Meteor Satellites (I plan to do a LRPT decoder as well
in the future, and I will post about it). I will not descri
describe
be my entire code
here, just the pieces for decoding the data. I will also not write the entire
code here, since it can be checked in github. So before start see the
picture below (again). We will some info from it as well.

LRIT Signal Specifications

Convolution Encoding, Frame

Synchronization and Viterbi
The first thing we need to do is sync our frame starts. For making easier
to find the packet start, these bit streams has something called preamble.
The preamble is basically a period when the bits se
sent
nt does not actually
contains any data, besides a information that informs the decoder that
this is the frame start. Usually the preamble consists in a fixed 32 bits
synchronization word. Most of the satellite systems use the standard
CCSDS Synchronization Word that is 0x1ACFFC1D ( or in
bits: 00011010 11001111 11111100 00011101 ). So basically we can use
this to find where our data starts. After we find these 32 bits we will have
the start of the data. Also with two sync words, we can find the period of
ourr frame (or, how many bits the frame has). But things can never be that
easy: The data is Convolution Encoded (including the Sync Word). Why is
the data Convolution Encoded? That’s simple: Its a way to correct errors
(bit swaps). So let me talk a bit of co
convolution encoding.

Convolution Encoding
Convolution Encoding basically is a process that generates parity of
original data as the output. This parity has a special feature that is
constructing a Trellis Diagram Sequence.

By Qef – Own work, Public Domain. Click on the image for details

A trellis diagram of two bits (as shown by the image), for every pair of
bits, there is only two possible next two bits. For example if my pair is 00,
the only possible next pair is 00 or 10. By convolution encoding a data,
we generate parity that follows the Trellis diagram, in this way if we have
errors in our bit stream, we can use a statistical way to find the most
possible path in the trellis diagram that represents that data, then
correcting the errors.

So how actually
tually the data is encoded? The image below will show a
example of convolution encoder:
By Teridon at English Wikipedia - Own work (Teridon), Public Domain, Link

The image shows a k=3, r=1/3 convolution encoder. The k parameter

says how many bits our buffer has. For a k=3 we will have a buffer of 3
bits. The r parameter gives us the rate of the encoder, in other words,
how many bits our encoder produces. In this case, for every 1 bit we input
to the encoder, we generate 3 bits. The encoder in the image also has
three more parameters: G1 = 111, G2 = 011, G3 = 101. These are
the generator polynomials. It’s always one generator polynomial per bit
of output. So for r = 1/2 we would have 2 polys, for 1/6 we would have 6
polys. And here is how the encoder works:

1. The buffer starts with all 0 (unless specified different)

2. The input bit enters on the left side of the buffer, and shifts all the
buffer bits to the right.
3. Each polymonial says what bits should be sum to give the
corresponding parity bit.
4. Cycle repeats.

So let me give an example. Suppose we’re already with the k buffer filled
with 101. Our next bit is 1. So we shift our buffer k right, and put our input
bit on the left-most position. So our buffer now will have 110. Now our 3
output bits will be:

 n0 = 1 + 1 + 0 = 0
 n1 = 0 + 1 + 0 = 1
 n2 = 1 + 0 + 0 = 1

So why these values? The generic way to represent that is:

  n0 = k[0] ^ G0[0] + k[1] ^ G0[1] + k[2] ^ G0[2]
 n1 = k[0] ^ G1[0] + k[1] ^ G1[1] + k[2] ^ G1[2]
 n2 = k[0] ^ G2[0] + k[1] ^ G2[1] + k[2] ^ G2[2]

Keep in mind, this is a mod 2 operation (we only have one bit output). So
basically we just get every buffer k bits, xor with the corresponding
position of the corresponding poly, and them sum to the output.

In this chapter we will use a lot of libfec routines to process our data. The
first one is to encode the sync words.

Encoding the sync word

First thing to do for frame sync is to convolution encode our sync word
( 0x1ACFFC1D ). The LRIT signal uses a k=7 and r=1/2 with the
poly’s 0x4F and 0x6D. We can use the parity function from libfec. So
here is the code I use for encoding the word: http://codepad.org/3FJ1tVIG

1 #include <stdio.h>
2 #include <stdlib.h>
#include <stdint.h>
3 #include <fec.h>
4
5 int main(int argc,char *argv[]) {
6 int polys[2] = {0x4f, 0x6d};
7 char syncword[] = {0x1A, 0xCF, 0xFC, 0x1D};
char outputword[64];
8
9 uint64_t encodedWord = 0;
10
11 char buff[2];
12
13 unsigned int encstate;
14 int wordSize = 4;
15 int processedBytes = 0;
int encodedWordPosition = 63;
16 int outputwordPosition = 0;
17
18 char c;
19
20 encstate = 0;
21 while(processedBytes < wordSize){
c = syncword[processedBytes];
22
for(int i=7;i>=0;i--){
23 encstate = (encstate << 1) | ((c >> 7) & 1);
24 c <<= 1;
25 // Soft Encoded Word
26 outputword[outputwordPosition] = 0 - parity(encstate & polys[0]);
outputword[outputwordPosition + 1] = 0 - parity(encstate & polys[1]);
27 outputwordPosition += 2;
28
29 // Encoded Word
30 encodedWord |= (uint64_t)(1-parity(encstate & polys[0])) << (encodedWordP
31 encodedWord |= (uint64_t)(1-parity(encstate & polys[1])) << (encodedWordP
32 encodedWordPosition-=2;
}
33 processedBytes++;
34 }
35
36 printf("Encoded Word: 0x%016lx\n", encodedWord);
37 printf("Soft Encoded Word: \n");
for (int i=0; i<64; i++) {
38 printf("0x%x ", outputword[i] & 0xFF);
39 }
40
41 printf("\n");
42
43 return 0;
44 }
45
46
47
48
49
50
51

You can compile with gcc using:

1 gcc encword.c -lfec -lm -o encword

This should give you the encoded syncword as 0xfca2b63db00d9794.

But since we can have a phase ambiguity of 180 degrees in BPSK, we
should also search for the inverted word that is 0x035d49c24ff2686b.
Just side note, these are actually the inverted syncwords that we will use
for searching the frame start. The 0 degree is
atually 0x035d49c24ff2686b and 180 degree is 0xfca2b63db00d9794.
But since we’ll use a correlation filter search, we should use the inverted
words to get maximum correlation.

Also the program will output the Soft Symbols for that Encoded Sync
Word, that is basically for every bit 0 it will have a byte 0, for every bit 1 it
will have a byte 0xFF (256). That’s because we are sure that these are
the correct bits (so we get the maximum values). In our decoder we’ll
convert the uint64_t encoded syncwords to a soft symbol array to use in
the correlation filter.

Frame Synchronization
Now we have the sync word, we can do the frame synchronization. I will
use a correlation system to find the most probably location for the sync
word. For that I will basically sum the difference of each XOR (UW[n] ^
data[i+n]) to a variable. The position that has the highest value (a.k.a.
highest correlation) is where our word is most probably.

1
2
3 typedef struct {
4 uint32_t uw0mc;
uint32_t uw0p;
5 uint32_t uw2mc;
6 uint32_t uw2p;
7 } correlation_t ;
8
9
10 const uint64_t UW0 = 0xfca2b63db00d9794; // 0 degrees inverted phase shift
const uint64_t UW2 = 0x035d49c24ff2686b; // 180 degrees inverted phase shift
11
12 uint8_t UW0b[64]; // The Encoded UW0
13 uint8_t UW2b[64]; // The Encoded UW2
14
15 void initUW() {
16 printf("Converting Sync Words to Soft Data\n");
17 for (int i = 0; i < 64; i++) {
UW0b[i] = (UW0 >> (64-i-1)) & 1 ? 0xFF : 0x00;
18 UW2b[i] = (UW2 >> (64-i-1)) & 1 ? 0xFF : 0x00;
19 }
20 }
21
22 uint32_t hardCorrelate(uint8_t dataByte, uint8_t wordByte) {
//1 if (a > 127 and b == 255) or (a < 127 and b ==
23 return (dataByte >= 127 & wordByte == 0) | (dataByte < 127 & wordByte == 255);
24 }
25
26
27 void resetCorrelation(correlation_t * corr) {
28 memset(corr, 0x00, sizeof(correlation_t));
29 }
30
void checkCorrelation(uint8_t *buffer, int buffLength, correlation_t *corr) {
31 resetCorrelation(corr);
32 for (int i = 0; i < buffLength - 64; i++) {
33 uint32_t uw0c = 0;
34 uint32_t uw2c = 0;
35
for (int k = 0; k < 64; k++) {
36
uw0c += hardCorrelate(buffer[i+k], UW0b[k]);
37 uw2c += hardCorrelate(buffer[i+k], UW2b[k]);
38 }
39
40 corr->uw0p = uw0c > corr->uw0mc ? i : corr->uw0p;
41 corr->uw2p = uw2c > corr->uw2mc ? i : corr->uw2p;
42
corr->uw0mc = uw0c > corr->uw0mc ? uw0c : corr->uw0mc;
43 corr->uw2mc = uw2c > corr->uw2mc ? uw2c : corr->uw2mc;
44 }
45 }
46
47
48
49
50

With this piece of code, you can run checkCorrelation(buffer, length,

corr) and get the correlation statistics. In the corr object you will have the
fields uw0p and uw2p with the positions of the highest correlation of the
buffer and in uw0mc and uw2mc you will have the correlation of that
position. With this two values you can know the position that your sync
word is, and what is the phase shift that the Costas Loop locked into (if
uw0 is 0 degrees, if uw2 is 180 degrees). To correct the data if the output
is 180 degrees phase shifted, just invert every bit in your sequence. I do
that by XOR’ing every byte with 0xFF. Now if you output this to a file and
inspect with BitDisplay you will see a pattern like this:

Synced Frames

See that the syncword is pretty clear in the bit display? You can use that
to visually identify sync words or static data inside the frames. You can
also notice that there are even more static data in the frames. We will talk
about it later. Now our data is ready to decode.

Decoding Frame Data

Now we’re ready to decode the frame data from the Convolution Code.
For that we will use an algorithm called Viterbi. Viterbi is an algorithm to
find hidden values in markov sequences. The Trellis Diagram from a
Convolution Code is basically a markov sequence with hidden state (the
hidden state is our data). Luckly the libfec includes a viterby for
convolution encoded code with k=7 and rate=1/2. We’re going to use it.
Its pretty straightforward to use it, but we need to have the frames in sync
and know the frame period. You can discover the frame period (if you’re
already didn’t) by checking the distance between sync words). In case of
LRIT the frames are 8192 bits wide (1024 bytes) from which 32 bits (4
bytes) is the sync word. Since its convolution encoded with rate 1/2, we
will have a encoded frame size of 16384 bits (or 2048 bytes). Using libfec
we can decode it like this:
#include <fec.h>
1
2 #define VITPOLYA 0x4F
3 #define VITPOLYB 0x6D
4
uint8_t codedData[16384]; // Our coded frame data as soft symbols
5 uint8_t decodedData[1024]; // Our decoded frame
6 int viterbiPolynomial[2] = {VITPOLYA, VITPOLYB};
7
8 /* */
9 void *viterbi;
set_viterbi27_polynomial(viterbiPolynomial);
10 if((viterbi = create_viterbi27(16384)) == NULL){
11 printf("create_viterbi27 failed\n");
12 exit(1);
13 }
14 // Now for each frame:
init_viterbi27(viterbi, 0);
15 update_viterbi27_blk(viterbi, codedData, 16384 + 6);
16 chainback_viterbi27(viterbi, decodedData, 16384, 0);
17 // decodedData will have the decoded stuff
18
19
20
21

So as a side note now: Do you remember in the last chapter that I said
about saving only the soft symbols? We synced using the soft symbols
and viterbi will also use the soft symbols. What is the advantage? Since
both Correlation Search and Viterbi are statistical algorithms, this
increase the odds that a value that is on the middle of the decision tree
(for example something close to 0 on a signed byte, that can be either bit
1 or bit 0) will not count much for the statistics. So a number that is on the
middle can be either 0 or 1, viterbi will use that as a *I can be any value*
in the trellis diagram and see which one is the most probably using other
values in the sequence. This increases the quality of decoding the same
as if you signal as about 2dB Higher SNR, and believe, that’s a HUGE
improvement.

Usually after decoding a frame, you might also want to check the quality
of the decoding. So a good way to measure is by re-encoding the signal
and comparing the two streams of the original encoded signal and the
expected encoded signal (the one you encoded).

1 /**/
2
void convEncode(uint8_t *data, int dataLength, uint8_t *output, int outputLen) {
3 unsigned int encstate = 0;
4 uint8_t c;
5 uint32_t pos = 0;
6 uint32_t opos = 0;
7
8 memset(output, 0x00, outputLen);
while (pos < dataLength && (pos * 16) < outputLen) {
9 c = data[pos];
10 for(int i=7;i>=0;i--){
11 encstate = (encstate << 1) | ((c >> 7) & 1);
12 c <<= 1;
output[opos] = ~(0 - parity(encstate & viterbiPolynomial[0]));
13 output[opos+1] = ~(0 - parity(encstate & viterbiPolynomial[1]));
14
15 opos += 2;
16 }
17 pos++;
}
18 }
19
20 uint32_t calculateError(uint8_t *original, uint8_t *corrected, int length) {
21 uint32_t errors = 0;
22 for (int i=0; i<length; i++) {
23 errors += hardCorrelate(original[i], ~corrected[i]);
}
24
25 return errors;
26 }
27 /**/
28
29 convEncode(decodedData, 1024, correctedData, 16384);
uint32_t errors = calculateError(codedData, correctedData, 16384) / 2;
30 float signalErrors = (100.f * errors) / 8192;
31 /**/
32
33
34
35
36
37

Then in the variable signalErrors you will have the percent of wrong bits
that had been corrected in your frame output. After that if you output to a
file and check with BitDisplay you will see something like this:
Decoded Frame View (cropped to the first bits only)

The bitdisplay actually shows the 0 bits as white and 1 bits as black. So
the overall perception is inverted. You can see the sync word pattern and
just after that a pattern that looks like a counter. This counter is actually
the frame counter. After that you can see a wide black bar and some
small patterns on the right side of this bar. This is actually the 11 bit
Virtual Channel ID. Since all channel IDs are usually lower than 64, most
of the bits are 0.

Now if you strip the first 4 bytes of each 1024 byte frame, you will have a
virtual channel frame from the satellite! In the next part I will show how to
parse this frame data and extract the packets that will in the last chapter
form our files. You can check my frame decoder code here:

https://github.com/racerxdl/open-satellite-
project/blob/225a36d4144c0fe0704eb50a8fbc428914f654c0/GOES/netw
ork/decoder_tcp.c
 (Part 4 – Packet Demuxer)
In the last chapter I showed how to get the frames from the demodulated
bit stream. In this chapter I will show you how to parse these frames and
get the packets that will on next chapter generate the files that GOES
send. I will first add C code to the code I did in the last chapter to
separated all the virtual channels by ID. But mainly this chapter will be
done in python (just because its easier, I will eventually make a C code as
well to do the stuff).

De-randomization
randomization of the data
One of the missing things on the last chapter was the frame data de
de-
randomization. The data inside the frame (excluding the sync word) is
randomized by a generator polynomial. This is done because of few
things:

 Pseudo-randomom Symbols distribute better the energy in the

spectrum
 Avoids “line-polarization”
polarization” effect when sending a continuous stream
of 1’s
 Better Clock Recovery due more changes in symbol polarity

CCSDS has a standard Polynomial as well, and the image below shows
how to generate the pseudo
pseudo-random bistream:
CCSDS Pseudo-random Bitstream Generator

The PN Generator polynomial (as shown in the LRIT spec) is x^8 + x^7 +
x^5 + x^3 + 1. You can check several PN Sequence Generators on the
internet, but since the repeating period of this PN is 255 bytes and we’re
xor’ing with our bytestream I prefer to make a lookup table with all the 255
byte sequence and then just xor (instead generating and xor). Here is the
255 byte PN:

1 char pn[255] = {

2 0xff, 0x48, 0x0e, 0xc0, 0x9a, 0x0d, 0x70, 0xbc,

0x8e, 0x2c, 0x93, 0xad, 0xa7, 0xb7, 0x46, 0xce,

3
0x5a, 0x97, 0x7d, 0xcc, 0x32, 0xa2, 0xbf, 0x3e,
4
0x0a, 0x10, 0xf1, 0x88, 0x94, 0xcd, 0xea, 0xb1,
5
0xfe, 0x90, 0x1d, 0x81, 0x34, 0x1a, 0xe1, 0x79,
6
0x1c, 0x59, 0x27, 0x5b, 0x4f, 0x6e, 0x8d, 0x9c,
7
0xb5, 0x2e, 0xfb, 0x98, 0x65, 0x45, 0x7e, 0x7c,
8 0x14, 0x21, 0xe3, 0x11, 0x29, 0x9b, 0xd5, 0x63,

9 0xfd, 0x20, 0x3b, 0x02, 0x68, 0x35, 0xc2, 0xf2,

10 0x38, 0xb2, 0x4e, 0xb6, 0x9e, 0xdd, 0x1b, 0x39,

11 0x6a, 0x5d, 0xf7, 0x30, 0xca, 0x8a, 0xfc, 0xf8,

0x28, 0x43, 0xc6, 0x22, 0x53, 0x37, 0xaa, 0xc7,

12
0xfa, 0x40, 0x76, 0x04, 0xd0, 0x6b, 0x85, 0xe4,
13
0x71, 0x64, 0x9d, 0x6d, 0x3d, 0xba, 0x36, 0x72,
14
0xd4, 0xbb, 0xee, 0x61, 0x95, 0x15, 0xf9, 0xf0,
15
0x50, 0x87, 0x8c, 0x44, 0xa6, 0x6f, 0x55, 0x8f,
16
0xf4, 0x80, 0xec, 0x09, 0xa0, 0xd7, 0x0b, 0xc8,
17 0xe2, 0xc9, 0x3a, 0xda, 0x7b, 0x74, 0x6c, 0xe5,

18 0xa9, 0x77, 0xdc, 0xc3, 0x2a, 0x2b, 0xf3, 0xe0,

19 0xa1, 0x0f, 0x18, 0x89, 0x4c, 0xde, 0xab, 0x1f,

20 0xe9, 0x01, 0xd8, 0x13, 0x41, 0xae, 0x17, 0x91,

0xc5, 0x92, 0x75, 0xb4, 0xf6, 0xe8, 0xd9, 0xcb,

21
0x52, 0xef, 0xb9, 0x86, 0x54, 0x57, 0xe7, 0xc1,
22
0x42, 0x1e, 0x31, 0x12, 0x99, 0xbd, 0x56, 0x3f,
23
0xd2, 0x03, 0xb0, 0x26, 0x83, 0x5c, 0x2f, 0x23,
24
0x8b, 0x24, 0xeb, 0x69, 0xed, 0xd1, 0xb3, 0x96,
25
0xa5, 0xdf, 0x73, 0x0c, 0xa8, 0xaf, 0xcf, 0x82,
26 0x84, 0x3c, 0x62, 0x25, 0x33, 0x7a, 0xac, 0x7f,
27 0xa4, 0x07, 0x60, 0x4d, 0x06, 0xb8, 0x5e, 0x47,

28 0x16, 0x49, 0xd6, 0xd3, 0xdb, 0xa3, 0x67, 0x2d,

29 0x4b, 0xbe, 0xe6, 0x19, 0x51, 0x5f, 0x9f, 0x05,

30 0x08, 0x78, 0xc4, 0x4a, 0x66, 0xf5, 0x58

};
31

32

33

34

And for de-randomization just xor’it with the frame (excluding the 4 byte
sync word):
1 for (int i=0; i<1020; i++) {

2 decodedData[i] ^= pn[i%255];

3 }

Now you should have the de

de-randomized frame.

Reed Solomon Error Correction

Other of the things that were missing on the last part is the Data Error
Correction. We already did the Foward Error Correction (FEC, the viterbi),
but we also can do Reed Solomon. Notice that Reed Solomon is
completely optional if you have good SNR (that is better than 9dB and
viterbi less than 50 BER) since ReedSolomon doesn’t alter the data. I
prefer to use RS because I don’t have a perfect signal (although my
average RS corrections are 0) and I want my packet data to be
consistent. The RS doe
doesn’t
sn’t usually add to much overhead, so its not big
deal to use. Also the libfec provides a RS algorithm for the CCSDS
standard.

I will assume you have a uint8_t buffer with a frame data of 1020 bytes
(that is, the data we got in the last chapter with the sync word excluded).
The CCSDS standard RS uses 255,223 as the parameters. That means
that each RS Frame has 255 bytes which 223 bytes are data and 32
bytes are parity. With this specs, we can correct any 16 bytes in our 223
byte of data. In our LRIT Frame we have 4 RS Frames, but the structure
are not linear. Since the Viterbi uses a Trellis diagram, the error in Trellis
path can generate a sequence of bad bytes in the stream. So if we had a
linear sequence of RS Frames, we could corrupt a lot of bytes fro from one
frame and lose one of the RS Frames (that means that we lose the entire
LRIT frame). So the data is interleaved by byte. The image below shows
how the data is spread over the lrit frame.

ReedSolomon Interleaving

For correcting the data, we need to de-interleave

interleave to generate the four RS
Frames, run the RS algorithm and then interleave again to have the frame
data. The [de]interleaving process are very simple. You can use these
functions to do that:

2 #define PARITY_OFFSET 892

3 void deinterleaveRS(uint8_t *data, uint8_t *rsbuff, uint8_t pos, uint8_t I) {

// Copy data
4
for (int i=0; i<223; i++) {
5
rsbuff[i] = data[i*I + pos];
6
}
7
// Copy parity
8
for (int i=0; i<32; i++) {
9 rsbuff[i+223] = data[PARITY_OFFSET + i*I + pos];

10 }

11 }

12

13 void interleaveRS(uint8_t *idata, uint8_t *outbuff, uint8_t pos, uint8_t I) {

// Copy data
14
for (int i=0; i<223; i++) {
15
outbuff[i*I + pos] = idata[i];
16
}
17
// Copy parity - Not needed here, but I do.
18
for (int i=0; i<32; i++) {
19 outbuff[PARITY_OFFSET + i*I + pos] = idata[i+223];

20 }

21 }

22

For using it on LRIT frame we can do:

1 #define RSBLOCKS 4

2 int derrors[4] = { 0, 0, 0, 0 };
3 uint8_t rsWorkBuffer[255];

4 uint8_t rsCorrectedData[1020];

5
for (int i=0; i<RSBLOCKS; i++) {
6
deinterleaveRS(decodedData, rsWorkBuffer, i, RSBLOCKS);
7
derrors[i] = decode_rs_ccsds(rsWorkBuffer, NULL, 0, 0);
8
interleaveRS(rsWorkBuffer, rsCorrectedData, i, RSBLOCKS);
9
}
10

In the variable derrors we will have how many bytes it was corrected for
each RS Frames. In rsCorrectedData we will have the error corrected
output. The value -1 in derrors it means the data is corrupted beyond
correction (or the parity is corrupted beyond correction). I usually drop the
entire frame if all derrors are -1,
1, but keep in mind that the corruption can
happen in the parity only (we can have corrupted bytes in parity that will
lead to -1
1 in error correction) so it would be wise to not do like I did. After
that we will have the corrected LRIT Frame that is 892 bytes wide.

Virtual Channel Demuxer

Now we will demux the Virtual Channels. I current save all virtual channel
payload (the 892 bytes) to a file called channel_ID.bin then I post
process with a python script to separate the channel packets. Parsing the
virtual channel header has also some advantages now that we can see if
for some reason we skipped a frame of the channel, and also to discard
the empty frames (I will talk abo
about it later).

VCDU Header

Fields:

 Version Number – The Version of the Frame Data

  S/C ID – Satellite ID
 VC ID – Virtual Channel ID
 Counter – Packet Counter (relative to the channel)
 Replay Flag – Is 1 if the frame is being sent again.
 Spare – Not used.

Basically we will only use 2 values from the header: VCID and Counter.

1
uint32_t swapEndianess(uint32_t num) {
2
return ((num>>24)&0xff) | ((num<<8)&0xff0000) | ((num>>8)&0xff00) | ((num<<24
3
}
4

5 (...)

6 uint8_t vcid = (*(rsCorrectedData+1)) & 0x3F;

8 // Packet Counter from Packet

9 uint32_t counter = *((uint32_t *) (rsCorrectedData+2));

counter = swapEndianess(counter);
10
counter &= 0xFFFFFF00;
11
counter = counter >> 8;
12

I usually save the last counter value and compare with the current one to
see if I lost any frame. Just be carefull that the counter value is per
channel ID (VCID). I actually never got any VCID higher than 63, so I
store the counter in a 256 int32_t array.

One last thing I do in the C code is to discard any frame that has 63 as
VCID. The VCID 63 only contains Fill Packets, that is used for keeping
the satellite signal continuous, even when not sending anything. The
payload of the frame will always contain the same sequence (that can be
sequence of 0, 1 or 01).

Packet Demuxer
Having our virtual channels demuxed for files channel_ID.bin, we can do
the packet demuxer. I did the packet demuxer in python because of the
easy of use. I plan to rewrite in C as well, but I will explain using python
code.

Channel Data

Each channel Data can contain one or more packets. If the Channel
contains and packet end and another start, the First Header Pointer (the
11 bits from the header) will contain the address for the first header inside
the packet zone.

First thing we need to do is read one frame from a channel_ID.bin file,

that is, 892 bytes (6 bytes header + 886 bytes data). We can safely
ignore the 6 bytes header from VCDU now since we won’t have any
usefulness for this part of the program. The spare 5 bits in the start we
can ignore, and we should get the FHP value to know if we have a packet
start in the current frame. If we don’t, and there is no pending packet to
append data, we just ignore this frame and go to the next one.
The FHP value will be 2047 (all 1’s) when the current frame only contains
data related to a previous packet (no header). If the value is different than
2047 then we have a header. So let’s handle this:

1 data = data[6:] # Strip channel header

2 fhp = struct.unpack(">H", data[:2])[0] & 0x7FF

3 data = data[2:] # Strip M_PDU Header

5 #data is now TP_PDU

6 if not fhp == 2047: # Frame Contains a new Packet

7 # handle new header

So let’s talk first about handling a new packet. Here is the structure of a
packet:

Packet Structure (CP_PDU)

We have a 6 byte header containing some useful info, and a user data
that can vary from 1 byte to 8192 bytes. So this packet can span across
several frames and we need to handle it. Also there is another tricky thing
here:
e: Even the packet header can be split across two frames (the 6 first
bytes can be at two frames) so we need to handle that we might not have
enough data to even check the packet header. I created a function
called CreatePacket that receives a buffer param
parameter
eter that can or not
have enough data for creating a packet. It will return a tuple that contains
the APID for the packet (or -1
1 if buffer doesn’t have at least 6 bytes) and
a buffer that contains any unused data for the packet (for example if there
was moreore than one packet in the buffer). We also have a function
called ParseMSDU that will receive a buffer that contains at least 6 bytes
and return a tuple with the MSDU (packet) header decomposed. There is
also a SavePacket function that will receive the ch channelId
annelId (VCID) and a
object to save the data to a packet file. I will talk about the SavePacket
later.

1 import struct

3 SEQUENCE_FLAG_MAP = {

4 0: "Continued Segment",

5 1: "First Segment",
6 2: "Last Segment",

7 3: "Single Data"

}
8

9
pendingpackets = {}
10

11
def ParseMSDU(data):
12
o = struct.unpack(">H", data[:2])[0]
13
version = (o & 0xE000) >> 13
14
type = (o & 0x1000) >> 12
15 shf = (o & 0x800) >> 11

16 apid = (o & 0x7FF)

17

18 o = struct.unpack(">H", data[2:4])[0]

19 sequenceflag = (o & 0xC000) >> 14

packetnumber = (o & 0x3FFF)

20
packetlength = struct.unpack(">H", data[4:6])[0] -1
21
data = data[6:]
22
return version, type, shf, apid, sequenceflag, packetnumber, packetlength, dat
23

24
def CreatePacket(data):
25
while True:
26 if len(data) < 6:
27 return -1, data

28 version, type, shf, apid, sequenceflag, packetnumber, packetlength, data = P

29 pdata = data[:packetlength+2]

30 if apid != 2047:

pendingpackets[apid] = {
31
"data": pdata,
32
33 "version": version,

34 "type": type,

"apid": apid,
35
"sequenceflag": SEQUENCE_FLAG_MAP[sequenceflag],
36
"sequenceflag_int": sequenceflag,
37
"packetnumber": packetnumber,
38
"framesdropped": False,
39
"size": packetlength
40 }

41

42 print "- Creating packet %s Size: %s - %s" % (apid, packetlength, SEQUENCE_

43 else:

44 apid = -1

45
if not packetlength+2 == len(data) and packetlength+2 < len(data): # Multiple p
46
SavePacket(sys.argv[1], pendingpackets[apid])
47
del pendingpackets[apid]
48
data = data[packetlength+2:]
49
apid = -1
50
print " Multiple packets in same buffer. Repeating."
51 else:
52 break

53 return apid, ""

54

55

56

57

With that we create a dictionary called pendingpackets that will store

APID as the key, and another dictionary with the packet data, including a
field called data that we will append data from other frames until we fill
the whole packet. Back to our read function, we will have something like
this:
...
1
if not fhp == 2047: # Frame Contains a new Packet
2
# Data was incomplete on last FHP and another packet starts here.
3
# basically we have a buffer with data, but without an active packet
4
# this can happen if the header was split between two frames
5 if lastAPID == -1 and len(buff) > 0:

6 print " Data was incomplete from last FHP. Parsing packet now"

7 if fhp > 0:

8 # If our First Header Pointer is bigger than 0, we still have

# some data to add.

9
buff += data[:fhp]
10
lastAPID, data = CreatePacket(buff)
11
if lastAPID == -1:
12
buff = data
13
else:
14
buff = ""
15

16 if not lastAPID == -1: # We are finishing another packet

17 if fhp > 0: # Append the data to the last packet

18 pendingpackets[lastAPID]["data"] += data[:fhp]

19 # Since we have a FHP here, the packet has ended.

SavePacket(sys.argv[1], pendingpackets[lastAPID])
20
del pendingpackets[lastAPID] # Erase the last packet data
21
lastAPID = -1
22

23
# Try to create a new packet
24
buff += data[fhp:]
25
lastAPID, data = CreatePacket(buff)
26 if lastAPID == -1:

27 buff = data

else:
28
buff = ""
29

30

31

32

This should handle all frames that has a new header. But maybe the
packet is so big that we got frames without any header (continuation
packets). In this case the FHP will be 2047, and basically we have three
things that can lead to that:

1. The header was split between last frame end, and the current
frame. FHP will be 2047 and after we append to our buffer we will
have a full header to start a packet
2. We just need to append the data to last packet.
3. We lost some frame (or we just started) and we got a continuation
packet. So we drop it.
...
1
else:
2
if len(buff) > 0 and lastAPID == -1: # Split Header
3
print " Data was incomplete from last FHP. Parsing packet now"
4 buff += data

5 lastAPID, data = CreatePacket(buff)

6 if lastAPID == -1:

7 buff = data

8 else:

buff = ""
9
elif len(buff)> 0:
10
# This shouldn't happen, but I put a warn here if it does
11
print " PROBLEM!"
12
elif lastAPID == -1:
13 # We don't have any pending packets, and we received

14 # a continuation packet, so we drop.

pass
15
else:
16
# We have a last packet, so we append the data.
17
print " Appending %s bytes to %s" % (lastAPID, len(data))
18
pendingpackets[lastAPID]["data"] += data
19

20

21

Now let’s talk about the SavePacket function. I will describe some of the
stuff here, but there will be also something described on the next chapter.
Since the packet data can be compressed, we will need to check if the
data is compressed, and if it is, we need to decompress. In this part we
will not handle the decompression or the file assembler (that will need
decompression).

Saving the Raw Packet

Now that we have the handler for the demuxing, we will implement the
function SavePacket. It will receive two arguments, the channel id and a
packetdict. The channel id will be used for saving the packets in the
correct folder (separating them from other channel packets). We may
have also a Fill Packet here, that has an APID of 2047. We should drop
the data if the apid is 2047. Usually the fill packets are only used to
increase the likely hood of the header of packet starts on the start of
channel data. So it “fills” the channel data to get the header in the next
packet. It does not happen very often though.

In the last step we assembled a packet dict with this structure:

1 {

2 "data": pdata,

3 "version": version,

4 "type": type,

"apid": apid,
5
6 "sequenceflag": SEQUENCE_FLAG_MAP[sequenceflag],

7 "sequenceflag_int": sequenceflag,

"packetnumber": packetnumber,
8
"framesdropped": False,
9
"size": packetlength
10
}
11

The data field have the data we need to save, the type says the type of
packet (and also if its compressed), the sequenceflag says if the packet
is:

 0 => Continued Segment, if this packet belongs to a file that has

been already started.
 1 => First Segment, if this packet contains the start of the file
 2 => Last Segment, if this packet contains the end of the file
 3 => Single Data, if this packet contains the whole file

It also contains a packetnumber that we can use to check if we skip any

packet (or lose).

The size parameter is the length of data field – 2 bytes. The two last
bytes is the CRC of the packet. The CCSDS only specify the polynomial
for the CRC, CRC-CCITT standard. I made a very small function based
on a few C functions I found over the internet:

1 def CalcCRC(data):

2 lsb = 0xFF

msb = 0xFF
3
for c in data:
4
x = ord(c) ^ msb
5
x ^= (x >> 4)
6
msb = (lsb ^ (x >> 3) ^ (x << 4)) & 255
7
lsb = (x ^ (x << 5)) & 255
8 return (msb << 8) + lsb

10 def CheckCRC(data, crc):

11 c = CalcCRC(data)

12 if not c == crc:

print " Expected: %s Found %s" %(hex(crc), hex(c))

13
return c == crc
14

15

On SavePacket function we should check the CRC to see if any data was
corrupted or if we did any mistake. So we just check the CRC and then
save the packet to a file (at least for now):
EXPORTCORRUPT = False
1
def SavePacket(channelid, packet):
2
global totalCRCErrors
3
global totalSavedPackets
4
global tsize
5
global isCompressed
6 global pixels

7 global startnum

8 global endnum

10 try:

os.mkdir("channels/%s" %channelid)
11
except:
12
pass
13

14
if packet["apid"] == 2047:
15
print " Fill Packet. Skipping"
16
return
17

18 datasize = len(packet["data"])

19
20 if not datasize - 2 == packet["size"]: # CRC is the latest 2 bytes of the payloa

21 print " WARNING: Packet Size does not match! Expected %s Found: %s" %(pack

if datasize - 2 > packet["size"]:

22
datasize = packet["size"] + 2
23
print " WARNING: Trimming data to %s" % datasize
24

25
data = packet["data"][:datasize-2]
26

27
if packet["sequenceflag_int"] == 1:
28
print "Starting packet %s_%s_%s.lrit" % (packet["apid"], packet["version"], p
29
startnum = packet["packetnumber"]
30 elif packet["sequenceflag_int"] == 2:

31 print "Ending packet %s_%s_%s.lrit" % (packet["apid"], packet["version"], pac

32 endnum = packet["packetnumber"]

33 if startnum == -1:

34 print "Orphan Packet. Dropping"

return
35
elif packet["sequenceflag_int"] != 3 and startnum == -1:
36
print "Orphan Packet. Dropping."
37
return
38

39
if packet["framesdropped"]:
40
print " WARNING: Some frames has been droped for this packet."
41 filename = "channels/%s/%s_%s_%s.lrit" % (channelid, packet["apid"], packet["ve
42 print "- Saving packet to %s" %filename

43

44

45 crc = packet["data"][datasize-2:datasize]

46 if len(crc) == 2:
47 crc = struct.unpack(">H", crc)[0]

48 crc = CheckCRC(data, crc)

else:
49
crc = False
50
if not crc:
51
print " WARNING: CRC does not match!"
52
totalCRCErrors += 1
53

54
if crc or (EXPORTCORRUPT and not crc):
55 f = open(filename, "wb")

56 f.write(data)

57 f.close()

58

59 totalSavedPackets += 1

else:
60
print " Corrupted frame, skipping..."
61

62

63

64

65

66

With that you should be able to see a lot of files being out of your
channel, each one being a packet. If you get the first packet (with the
sequenceflag = 1), you will also have the Transport Layer header that
contains the decompressed file size, and file number. We will handle the
decompression and lrit file composition in next chapter. You can check
the final code here: https://github.com/racerxdl/open-satellite-
project/blob/master/GOES/standalone/channeldecoder.py
GOES Satellite Hunt (Part 5 – File
Assembler)
0 Comments

In the last chapter of my GOES Satellite Hunt, I explained how to obtain

the packets. In this part I will explain how to aggregate and decompress
the packets to generate the LRIT files. This part will be somwhat quick,
because most of the hard stuff was already done in the last part. Sadly
the decompression algorithm is a modified RICE algorithm, and the Linux
version of the library provided by NOAA cannot be used anymore
because of incompatibilities between GCC ABIs ( The NOAA library has
been compiled with GCC 2). Until I reverse engineer and create a open
version of the decompression algorithm, I will use the workaround I will
explain here.

In the packets before we have a flag called continuation flag that will
specify if the packet is a start, continuation or end packet. It will also
say if the packet itself is a single file. Usually the file header is entire
contained on the first packet. One of the things we need to do is
aggregate the start, continuation and end packets into a single file. This is
easy since the when a start packet appears inside a channel with the
same APID, the entire file will be transmitted at once, so the packets that
will follow will be either continuation or end before a new start comes. So
basically we just need to find a Packet Start with a certain APID and then
grab and aggregate all of the following packets that has the same APID.
But the packet content may also be compressed
using LritRice.lib (provided by NOAA). To check if we need to
decompress the packet or not, we need to check the header of the start
packet (the header will always be decompressed).

File Header Processing

The LRIT File has several headers. The first one is from the transport
layer that says what is the file number and what is the total
decompressed size. i usually ignore this data because its only used for
validation of the finished file. This header has 10 bytes.
LRIT File

Discarding the first 10 bytes (2 bytes for file counter and 8 bytes for the
length), you will have the Primary Header. The primary header has
basically just the size of the LRIT File header. We will need to parse all
the headers (including the secondary) in order to find if the continuation
packets will be compressed and if they are, what are the parameters to
decompress (yes, they can change). I created
a packetmanager.py script to export few helper functions to parse the
header inside channeldecode.py
channeldecode.py.. There are several header types with
different lengths, but they do have two parameters in common:

 Type – 1 byte (uint8_t)

 Size – 2 byte (uint16_t)

So what I usually do, is to grab the first 3 bytes of a header, check the
size (the size includes these 3 bytes) and then fetch more size – 3 bytes
to the buffer. With this buffer I pass to another function that will parse the
header data and return a obje
object with their parameters. This is
my parseHeader function:

1 def parseHeader(type, data):

2 if type == 0:
filetypecode, headerlength, datalength = struct.unpack(">BIQ", data)
3 return {"type":type, "filetypecode":filetypecode, "headerlength":headerlength, "
4 elif type == 1:
5 bitsperpixel, columns, lines, compression = struct.unpack(">BHHB", data)
data
6 return {"type":type, "bitsperpixel":bitsperpixel, "columns":columns, "lines":lin
7
8 elif type == 2:
projname, cfac, lfac, coff, loff = struct.unpack(">32sIIII", data)
9 return {"type":type, "projname":projname, "cfac":cfac, "lfac":lfac, "coff":coff,
10
11 elif type == 3:
12 return {"type":type, "data":data}
13
14 elif type == 4:
return {"type":type, "filename":data}
15
16 elif type == 5:
17 days, ms = struct.unpack(">HI", data[1:])
18 return {"type":type, "days":days, "ms":ms}
19
20 elif type == 6:
21 return {"type":type, "data":data}
22
elif type == 7:
23 return {"type":type, "data":data}
24
25 elif type == 128:
26 imageid, sequence, startcol, startline, maxseg, maxcol, maxrow = struct.unpack("
27 return {"type":type, "imageid":imageid, "sequence":sequence, "startcol":startcol
28
elif type == 129:
29 signature, productId, productSubId, parameter, compression = struct.unpack(">4sH
30 return {"type":type, "signature":signature, "productId":productId, "productSubId
31
32 elif type == 130:
33 return {"type":type, "data":data}
34
35 elif type == 131:
flags, pixel, line = struct.unpack(">HBB", data)
36 return {"type":type, "flags":flags, "pixel":pixel, "line":line}
37
38 elif type == 132:
39 return {"type":type, "data": data}
40 else:
41 return {"type":type}
42
43
44
45
46
47

And since we should read all headers, here is

the getHeaderData function:

1 def getHeaderData(data):
headers = []
2 while len(data) > 0:
3 type = ord(data[0])
4 size = struct.unpack(">H", data[1:3])[0]
5 o = data[3:size]
6 data = data[size:]
td = parseHeader(type, o)
7 headers.append(td)
8 if td["type"] == 0:
9 data = data[:td["headerlength"]-size]
10 return headers
11
12

With that, we have enough stuff for using in our channeldecoder.py and
know if the file is compressed. Basically we can do a simple import
packetmanager and use the packetmanager.py functions.

LritRice Compression
Usually for images, the data is compressed using LritRice.lib. Although
RICE compression is a open standard (NASA’s fitsio library has
compression and decompression of RICE), the LritRice use a modified
version. With time I will reverse engineer and create a open version that
will be able to decompress LRIT data, but for now I had to do a
workarround. Since the LritRice from linux is “broken”, I made a very
nasty workarround:

Make a windows application to decompress and run through wine.

The project of decompressor is available

here: https://github.com/racerxdl/open-satellite-
project/tree/master/GOES/decompressor, I will soon release some
binaries for those who don’t want to compile the application themselves.
But for those who want, just open the visual studio solution and hit
compile, it should generate a decompressor.exe that we will be using
together with wine and python (or if you’re at windows, just with python).

The decompressor is made to receive some arguments and has two ways
of operation:

 Single File Decompression: decompressor.exe Pixels Filename

 Multi File Decompression: decompressor.exe Prefix StartNumber
EndNumber [DebugMode]

We’re gonna use the Multi File decompression. It does basically the same
as single file, but iterates over several files and decompress all of them
into a single file. So the output file will be a single file with all of the
original files together (so our final LRIT file). The StartNumber should be
the number of the first continuation packet (so not the header packet).
The Multi File Decompression will look into StartNumber-1 to EndNumber
files, being the StartNumber-1 just rewrited to the output file that will have
a _decomp suffix. So in our channeldecoder.py we need to do few
steps.

First let’s check if either the packet stream will need to be decompressed
or just appended. If they just need to be appended, we only do that. Text
files and DCS files are usually not compressed.

In our savePacket function, if the packet is a start packet, we should run

the getHeaderDatafrom packetmanager and check the compression
flags.

1
if packet["sequenceflag_int"] == 1:
2
print "Starting packet %s_%s_%s.lrit" % (packet["apid"], packet["version"], pack
3 startnum = packet["packetnumber"]
4 p = packetmanager.getHeaderData(data[10:])
5 for i in p:
6 if i["type"] == 1 or i["type"] == 129:
7 isCompressed = not i["compression"] == 0
if i["type"] == 1:
8 pixels = i["columns"]
9

In headers of type 1 (Image Structure Header) or type 129 (NOAA

Specific Header) both describe if its compressed or not. If its an image,
we will have the compression flag set on Image Structure Header. If its
another file, it will be in NOAA Specifc header. If the data is compressed,
we need to grab the columns parameter that will be used as
the Pixels parameter in decompressor. If the decompression is enabled,
all further packets including the end packet will need to be
decompressed. So for running decompressor we also need what is the
packetnumber of the first packet (that will be the start packet + 1) and the
number of the latest packet. So if the continuation flag says that the
current packet is the latest, we need to save the number:

1
elif packet["sequenceflag_int"] == 2:
2
print "Ending packet %s_%s_%s.lrit" % (packet["apid"], packet["version"], packe
3 endnum = packet["packetnumber"]
4 if startnum == -1:
5 print "Orphan Packet. Dropping"
6 return
7 elif packet["sequenceflag_int"] != 3 and startnum == -1:
print "Orphan Packet. Dropping."
8 return
9

I also set the startnum as -1 when there is no received start packet, so I

can know if I have any orphan continuation / end packets. If that’s the
case, we just drop (if we don’t have the headers we cannot know whats
the content). Now we need to handle the output filename. If its
compressed we won’t be appending each packet to a final file, instead we
will create a file that contains the packet number on it (so the
decompressor can run over it). But if the data isn’t compressed, we can
just append to the final file, so our final file shouldn’t have the packet
number.

1 if isCompressed:
2 filename = "channels/%s/%s_%s_%s.lrit" % (channelid, packet["apid"], packet["ver
3 else:
4 filename = "channels/%s/%s_%s.lrit" % (channelid, packet["apid"], packet["versio

Now, in aspect of saving the file. If its not compressed we need to open
for appending, if it is we just save by skipping the 10 first bytes.

1 firstorsinglepacket = packet["sequenceflag_int"] == 1 or packet["sequenceflag_int"]

2 if not isCompressed:
3 f = open(filename, "wb" if firstorsinglepacket else "ab")
4 else:
f = open(filename, "wb")
5

For running the decompressor I created a function

called Decompressor that will receive the parameters and run wine to
process the file. It will also delete the original compressed packets, since
everything should be at a _decomp file.

1
2 from subprocess import call
3
def Decompressor(prefix, pixels, startnum, endnum):
4 startnum += 1
5 call(["wine", "Decompress.exe", prefix, str(pixels), str(startnum), str(endnu
6 for i in range(startnum-1, endnum+1):
7 k = "%s%s.lrit" % (prefix, i)
if os.path.exists(k):
8 os.unlink(k)
9 return "%s_decomp%s.lrit" % (prefix, startnum-1)
10

Now we can do the following to have the things decompressed:

1 if (packet["sequenceflag_int"] == 2 or packet["sequenceflag_int"] == 3):

2 if isCompressed:
3 if startnum != -1:
4 decompressed = Decompressor("channels/%s/%s_%s_" % (channelid, packet["apid"

The decompressed var will have the final filename of the decompressed
file.

File Name from Header

Some of the files has a filename in the header. So if they have, we can
rename it. The header that contains the filename is header type 4
(Annotation Record). so I created a funcion
called manageFile inside packetmanager.py to do the work of the
filename.

1
2
3 def manageFile(filename):
f = open(filename, "r")
4
5 try:
6 k = readHeader(f)
7 type, filetypecode, headerlength, datalength = k
8 except:
9 print " Header 0 is corrupted for file %s" %filename
return
10
11 newfilename = filename
12 while f.tell() < headerlength:
13 data = readHeader(f)
14 if data[0] == 4:
#print " Filename is %s" % data[1]
15 newfilename = data[1]
16 break
17 f.close()
18 if filename != newfilename:
19 print " Renaming %s to %s/%s" %(filename, os.path.dirname(filename), newfi
os.rename(filename, "%s/%s" %(os.path.dirname(filename), newfilename))
20 else:
21 print " Couldn't find name in %s" %filename
22
23

This code will search for a filename in header, if it finds, it will rename the
input filename to whatever is in the header. If not, it will just keep the
same name. So in the channeldecoder.py I can just do this to have
everything processed:

1
if (packet["sequenceflag_int"] == 2 or packet["sequenceflag_int"] == 3):
2 if isCompressed:
3 if USEDECOMPRESSOR and startnum != -1:
4 decompressed = Decompressor("channels/%s/%s_%s_" % (channelid, packet["apid"
5 packetmanager.manageFile(decompressed)
6 else:
print "File is not compressed. Checking headers."
7 packetmanager.manageFile(filename)
8

After that, you should have all files with the correct naming (if they have in
the header) and decompressed! The filenames are usually
like gos13chnIR04rgnNHseg001res04dat308034918927.lrit.
Viewing the files content
I still need to do some program to parse, but at least for now there is
the xrit2pic that can parse some of the GOES LRIT (and other satellites
LRIT) files. If you want to make your own parser, most of the files are
easy to process. The Text files are just raw text data (so just skip the
headers), the images are in raw format (check the headers to see how
they’re composed). Some aditional details about the headers that I
mapped are here: https://github.com/racerxdl/open-satellite-
project/blob/master/GOES/standalone/packetmanager.py#L172-L260

In a future article I will make a User Guide to my LRIT Decoder. For now I
want to make it better and with a more user friendly interface, so this
articles are intended to someone who wants to understand how the
protocol works. These are some data I got from GOES 13:

GOES 13 Full Disk Image

Meteosat Small Fulldisk image
WEFAX messages

Text Messages:

————————————–
LRIT Admin Message #011
Start:14-April-2010
End:20-December-2018
Distribution: East and West
Subject: LRIT contact information
————————————–
The LRIT Systems team, in an effort to be more responsive
to the user community, would like for users to have
contact information. In the event that a user notices any
long term trends or anomalies in the LRIT data stream, or
has suggestions or comments. We ask that contact be made
via email to LRIT@noaa.gov.

If more immediate matters arise, that the user deems as

urgent, we advise the use of the following operational
facility phone number: 301-817-3880.
————————————–

Ending
This is the last chapter of my GOES Satellite Hunt. For sure there will be
more things that I will right about it but at least with this Article you should
be able to create your own Demodulator / Decoder for LRIT signals. I still
have some stuff todo, but I will post my progress here in this blog.