Troubleshooting Updates,

RMA Process
&
M120 Overview
2009 Q1
Jason Abercrombie
Resident Engineer

Copyright 2008 Juniper Networks, Inc.

Agenda
Troubleshooting Updates
LCHIP
RXXG

RMA Process
M120 Overview

Craft Interface
Cooling
Flexible PIC Concentrator (FPC) and Compact FPC (cFPC)
Control Board
Routing Engine
Forwarding Engine Board
Power Distribution

Copyright 2008 Juniper Networks, Inc.

LCHIP Update
LCHIP errors can be expected if...
- An interface is flapping interface on the FPC experiencing the LCHIP errors
- The FPC has bounced (should also see HDRF CRC errors in DESRD)
Troubleshooting Review:
Error-checking on M320 and T640 occurs once, on egress FPCs LCHIP
The error can occur anywhere along the data path
1 or more ingress FPCs Fabric (SIBs) Egress FPC
Goal: Find the faulty SIB or FPC
Work from least impact to most impact
1. Cycle the SIBs one by one, checking if errors subside
2. Cycle the FPCs one by one, checking if errors subside

Copyright 2008 Juniper Networks, Inc.

RXXG Update
Jan 13 06:35:58 2009

cer-pcor-03 fpc7 .pm3393.7.1. RXXG: %PFE-3: Packet exceeds the maximum frame size 1526

Jan 13 06:35:59 2009

cer-pcor-03 fpc7 .pm3393.7.1. RXXG: %PFE-3: A line interface error is detected

These messages appear when LINE_ERRI bit is set on PMC pm3393 MAC chip.
LINE_ERRI
The LINE_ERRI bit is set when a Line Interface error is detected. Failure modes are (1)
Breakdown in alternating SOP-EOP sequence (2) Invalid byte(s) between SOP and EOP
not part of a completely invalid word. (3) Reception of frames less than 14 bytes.
PR 80772 RXXG messages are reported for invalid frames
PR 100022 RXXG messages logged too often
PR (Internal) XENPAK triggers confusing messages as a result of packet loss
RXXG messages are a symptom of a real failure that has occurred at the MAC layer.

Copyright 2008 Juniper Networks, Inc.

RXXG Update
Cases opened in 2008
Case Number and Routers

Solution

2008-0416-0238 atl-pcor-02

Pads applied on local links. Errors stopped

2008-0813-0055 dca-pcor-01

Closed after RMA of T640-FPC3-E

2008-0917-0354 atl-pcor-02 dca-pcor-01

Closed with no information on resolution

2008-0917-0406 dcx-core-01 dca-pcor-02 bur-pcor-01

Correlation found between errors and flaps

2008-1025-0089 phx-core-02

NCC replaced a MUX card on transport

2008-1110-0731 dca-pcor-2

Logging stopped spontaneously

2008-1117-0452 slkc-agw1

Closed after CFTS cited PRs 80772 and 100022

2008-1121-0237 cer-edge-12

Closed after JTAC cited internally tracked known issue (229064)

2008-1210-0704 cls-core-02

Closed after JTAC cited PR 100022

2008-1211-0683 cer-pcor-01

JTAC cited PR 100022. Closed after NCC replaced a card

2008-1211-0692 dca-core-01

Errors subsided 1/28 after cable removed and replaced

2008-1215-0316 jfk-pcor-01

JTAC suggested to bounce PIC and test transport. Closed after inactivity

2008-1224-0224 dca-pcor-01

Errors stopped after fibers cleaned and PIC replaced

Copyright 2008 Juniper Networks, Inc.

RXXG Update
792 Xenpak xcvrs in the network
42 (5.3%) reported >1 errors in 2009
Including only:
RXXG / line interface errors
RXXG / exceeds maximum frame size
Not including RXOAM messages (not errors)
41 of the 42 (97.6%) xcvrs have P/N 740-013170
766 of all 792 (96.7%) xcvrs have this P/N

Copyright 2008 Juniper Networks, Inc.

RXXG Update
XENPAK Xcvrs reporting errors in 2009
C715TF012

T07D05561

T07K46483

T06F90384

T07D05563

T07K46505

T07C93701

T07D05572

T07K46512

T07C94487

T07D05574

T07M71281

T07C96411

T07D05580

T07M71290

T07C96431

T07E25431

T07M71325

T07D04670

T07E32650

T07M71418

T07D04676

T07F53170

T07M71441

T07D04749

T07G74116

T07M71443

T07D04761

T07J04552

T07M71527

T07D04766

T07J04761

T07M71574

T07D04780

T07K39450

T07M71624

T07D04821

T07K46381

T08A15335

T07D04824

T07K46440

T08B21207

Copyright 2008 Juniper Networks, Inc.

RXXG Update
Three causes for these errors
- Bad transmitting device
- Line impairment
- Problem at receiving side
Local Side
- Collect Data
Frequency of Errors
Incrementing Counters (CRC/Align, Jabber frames, etc.)
- Clean Fiber
- Replace PIC (PIC replacement fixes the problem in most cases)
Remote Side Testing As Necessary
Line/Transport Testing As Necessary

Copyright 2008 Juniper Networks, Inc.

Agenda
Troubleshooting Updates
LCHIP
RXXG

RMA Process
M120 Overview

Craft Interface
Cooling
Flexible PIC Concentrator (FPC) and Compact FPC (cFPC)
Control Board
Routing Engine
Forwarding Engine Board
Power Distribution

Copyright 2008 Juniper Networks, Inc.

RMA Process Change

Problems:
- Growing list of our spares out of inventory
- Growing delay in getting these cards repaired and back into inventory in a timely manner
- Replacement process can go for weeks due to customer notification and internal processes
- Juniper may close the case prematurely due to inactivity
- Since DLC has only the case number to reference, the serials may not match the case, or
the card itself does not match the case (info gets lost)
Changes:
1. NMC raises the RMA w/ Juniper directly, using the DLC address as the return address
2. Once the RMA is raised, we'll request the replacement from PICSCSC and use the RMA
number (INSTEAD OF the case number to track)
3. Address changes can be made to the RMA any time
(When in doubt, CALL JTAC to confirm the address change)
4. Once DLC receives the faulty, the RMA details will be readily available and can be shipped
quickly for repair
Nutshell Version:
Instead of giving the Juniper case # for each NNS Spare, we provide the RMA # during the
initial order

Copyright 2008 Juniper Networks, Inc.

10

Agenda
Troubleshooting Updates
LCHIP
RXXG

RMA Process
M120 Overview

Craft Interface
Cooling
Flexible PIC Concentrator (FPC) and Compact FPC (cFPC)
Control Board
Routing Engine
Forwarding Engine Board
Power Distribution

Copyright 2008 Juniper Networks, Inc.

11

Components

Hardware inventory:

Item

Chassis

Midplane

REV 04

FPM Board

Serial number

Description

JN109326EAEA

M120

710-018041

RC2032

M120 Midplane

REV 06

710-011407

DM3006

M120 FPM Board

FPM Display

REV 02

710-011405

RH1106

M120 FPM Display

FPM CIP

REV 05

710-011410

RH1117

M120 FPM CIP

PEM 0

Rev 10

740-011935

TL53577

DC Power Entry Module

PEM 1

Rev 10

740-011935

TL53737

DC Power Entry Module

Routing Engine 0 REV 07

740-014082

9009004730

RE-A-2000

Routing Engine 1 REV 07

740-014082

9009004240

RE-A-2000

CB 0

REV 09

710-011403

DL2561

M120 Control Board

CB 1

REV 09

710-011403

DM5467

M120 Control Board

FPC 2

REV 03

710-015837

DM3717

M120 FPC Type 2

REV 07

750-010618

DM5012

4x G/E SFP, 1000 BASE

SFP-SX

PIC 0

Part number

Xcvr 0

REV 01

740-011613

AM0813S91DA

Xcvr 1

REV 01

740-011614

84S495H11736

SFP-LX

PIC 1

REV 25

750-001901

WP3316

4x OC-12 SONET, SMIR

PIC 2

REV 12

750-009066

WM1052

1x OC-48 SONET SFP

REV 01

740-011786

768002D00120

SFP-IR

REV 04

710-015838

DM0335

M120 FPC Mezz Board

REV 03

710-015837

DM3720

M120 FPC Type 2

REV 07

750-010618

DM2489

4x G/E SFP, 1000 BASE

SFP-SX

Xcvr 0
Board B
FPC 3
PIC 0

Xcvr 0

REV 01

740-011613

PD50TDN

Xcvr 1

REV 01

740-011614

84S495H11733

SFP-LX

REV 12

750-009066

WM1058

1x OC-48 SONET SFP

Copyright 2008 Juniper Networks, Inc.

Version

PIC 2
Xcvr 0
Board B
FPC 4

REV 01

740-011786

798002D00402

SFP-IR

REV 04

710-015838

DM0362

M120 FPC Mezz Board

REV 03

710-015835

RH1939

M120 FPC Type 1

PIC 0

REV 22

750-005634

DP0143

1x CHOC12 IQ SONET, SMIR

PIC 1

REV 13

750-003034

RG6352

4x OC-3 SONET, SMIR

Board B

REV 03

710-017980

RH1208

M120 FPC Mezz Board

FEB 0

REV 05

710-015795

RG9019

M120 FEB

FEB 1

REV 05

710-015795

DN4401

M120 FEB

FEB 2

REV 05

710-015795

DK1553

M120 FEB

FEB 3

REV 05

710-015795

DN4431

M120 FEB

FEB 4

REV 05

710-015795

DN4356

M120 FEB

FEB 5

REV 05

710-015795

DK1518

M120 FEB

Fan Tray 0

Front Top Fan Tray

Fan Tray 1

Front Bottom Fan Tray

Fan Tray 2

Rear Top Fan Tray

Fan Tray 3

Rear Bottom Fan Tray

12

Front Components
Craft Interface
Front Top Fan Tray
cFPCs
PIC
FPC
Front Bottom Fan
Tray
Copyright 2008 Juniper Networks, Inc.

13

Craft Interface
Yellow and Red
Alarm LEDs, and
Alarm Cut-off button
External clock ports
Alarm relay contacts

PEM LEDs

RE0 ports

RE1 ports

RE and CB
LEDs
FEB LEDs

FPC LEDs and

online/offline buttons
Copyright 2008 Juniper Networks, Inc.

14

Craft Interface
Yellow/Red Alarms and Cut-off

RED Critical alarm indicating possible fatal situation

Possible causes: Failure, removal, major overheat

YELLOW Warning indicating serious but non-fatal situation

Possible causes: Maintenance alert, minor overheat

Cut-off Deactivates RED and YELLOW alarms, and tests all LEDs
when pressed and held

External Clock Ports

EXT CLOCK Ports A and B accept two RJ-45s for clock input with T1 or
E1 reference clocks

Alarm Relay Contacts

Connect the M120 to external alarm devices

RE Ports

AUX Laptop, modem, etc

CONSOLE Serial system console
ETHERNET OOB FE

Copyright 2008 Juniper Networks, Inc.

15

Craft Interface
Yellow/Red Alarms and Cut-off
FPC LEDs and Buttons
Status

Steady GRN
Blinking GRN
Steady RED

FPC is functioning normally

FPC is transitioning online/offline
FPC has failed

Steady GRN
Steady RED

PEM is functioning normally

PEM has failed

Steady GRN
Steady GRN
Steady RED
Steady GRN
Blinking GRN
Steady RED

RE is master
RE is functioning
RE has failed
CB is active
CB is transitioning online/offline
CB has failed

Steady GRN
Blinking GRN
Steady GRN
Steady RED

FEB is active

PEM LEDs
Status

RE/CB LEDs
RE Master
RE Status
CB Status

FEB LEDs
Active
Status

Copyright 2008 Juniper Networks, Inc.

FEB is transitioning online/offline

FEB is functioning normally
FEB is not functioning normally

16

Cooling
Fan
Fan
Fan
Fan

Tray
Tray
Tray
Tray

0
1
2
3

Copyright 2008 Juniper Networks, Inc.

Front Top Fan Tray

Front Bottom Fan Tray
Rear Top Fan Tray
Rear Bottom Fan Tray

17

FPCs and cFPCs

FPCs host PICs, but FEBs do the Forwarding
Provide infrastructure to power and control PICs and to translate packets to and from
each PIC into a standard interface that a FEB processes
FPCs interface with the following router system components
PEMs
CBs
FEBs
PICs
Each FPC contains a translator, a crossbar connection to the FEBs, power subsystem,
and the physical PIC connectors
Up to two Compact FPCs (cFPCs) and four FPCs install vertically in the front of the
router
cFPCs are numbered top to bottom cFPC0 and cFPC1, and the remaining FPC slots
are numbered left to right from FPC2 to FPC5
The assembly contains a translation component that converts between the midplane
signals and the signals required by the types of supported PICs. The translator fully
terminates the PIC side connection, providing local flow control, buffering, and electrical
conversion.
FPCs and cFPCs are located on the front of the chassis, and provide power and
management to the PICs through the midplane. The midplane relays signals to
the FEB, inserted from the rear of the chassis, which processes the packets
Copyright 2008 Juniper Networks, Inc.

18

FPCs and cFPCs

FPC1
Rated at 4 gigabits per second (Gbps) full duplex
Supports up to four PICs
FPC2
Rated at 10 Gbps full duplex
Supports up to four PICs
FPC3
Rated at 10 Gbps full duplex
Supports one PIC, including higher-speed PICs
cFPC
Can be inserted into only FPC slot 0 or 1

A cFPC is a combination of a PIC and an FPC. It contains the interface circuitry

and the FPC as a single assembly.
Rated at 10 Gbps full duplex
Supports one 10-Gbps Ethernet or one OC192 interface per cFPC
Any combination of FPC and cFPC types can be installed in the M120 router.

Copyright 2008 Juniper Networks, Inc.

19

Rear Components
Rear Top Fan Tray
CB0
RE0
FEBs
PEMs
Rear Bottom Fan
Tray
Copyright 2008 Juniper Networks, Inc.

20

Control Board
CB works with RE to provide control and monitoring functions

Determine RE mastership
Control power and reset for the other router components
Connect FEBs and FPCs
Monitor and control fan speed
Monitor system status

Switch fabric

Provides transit traffic through the Control Board

SONET clocking module

Provides a Stratum 3 timing reference for all SONET interfaces

Redundant configuration

If two CBs are installed, one functions as the master CB and the other as
its backup. If the master fails or is removed, the backup restarts and
becomes the master.
CBs are hot-pluggable. If a CB fails and switches mastership to the
redundant CB, the Routing Engine mastership switches as well.

Copyright 2008 Juniper Networks, Inc.

21

Routing Engine
Boot Order

USB device
Internal flash disk
HDD
LAN

Copyright 2008 Juniper Networks, Inc.

22

Forwarding Engine Board

FPCs host PICs, but FEBs do the Forwarding
FEBs provide route lookup and forwarding functions
FPCs and cFPCs are located on the front of the
chassis, and provide power and management to the
PICs through the midplane. The midplane relays
signals to the FEB (inserted from the rear of the
chassis, which processes the packets.
The midplane architecture allows any FEB to carry
traffic for any FPC. Mapping of FPCs to FEBs can be
configured manually.
If a FEB fails, a backup FEB can quickly take over
packet forwarding.

Copyright 2008 Juniper Networks, Inc.

23

Forwarding Engine Board

I-chip ASIC
Provides multiple paths for PFE to PIC communication

Crossbar switch
Provides connection between FEB WAN links and
FPC WAN links

Maximum 20Gbps fabric interface

Copyright 2008 Juniper Networks, Inc.

24

Power Distribution
Non-redundant

Single PEM provides sufficient

power for a fully populated
router

Same behavior whether AC or

DC
Copyright 2008 Juniper Networks, Inc.

PEM 1

Two PEMs share load almost

equally in a fully populated
system
On failure, the remaining PEM
assumes full load without
interruption

PEM 0

Redundant

Same position
whether AC or DC
25

Summary / Questions?
Troubleshooting Updates
LCHIP
RXXG

RMA Process
M120 Overview

Craft Interface
Cooling
Flexible PIC Concentrator (FPC) and Compact FPC (cFPC)
Control Board
Routing Engine
Forwarding Engine Board
Power Distribution

Copyright 2008 Juniper Networks, Inc.

26

Copyright 2008 Juniper Networks, Inc.

27