Building
a
High
Performance
Active
Directory
Integrated
10GBE
Linux


Storage
Server
for
use
over
Single
Mode
Fiber
Networks

(SATA
Version)

By:
Ian
Evans









 Page
1






Hardware
and
software
perquisites:


• (1)
RedHat/CentOS
x_86_64
bit
operating
system
(CentOS
recommended)

• (1)
Cisco
SMF
10GB
XENPAK
(XENPAK‐10GB‐LR=)

• (1)
Supermicro
high
density
chassis
(SC846TQ‐R900B)

• (1)
Supermicro
ATX
server
motherboard
(X7DCL‐3)

• (1)
Intel
Xeon
Quad
Core
1.6GHz
Processor
(BX80563E5310P)

• (2)
ATP‐Samsung
2GB
ECC
FBDIMM’s
(AP56K72U4BHE7S)

• (1)
3Ware
24‐Port
RAID
card
(9650SE‐24M8)

• (1)
3Ware
4‐Port
RAID
card
(9650SE‐4LPML)

• (2)
3Ware
Battery
Backup
Units
(BBU‐MODULE‐03)

• (19)
Western
Digital
SATA‐II
750GB
enterprise
drives
(WD7500AYPS)

• (4)
Western
Digital
SATA
74GB
10K
enterprise
drives
(WD740ADFD)

• (1)
10GB
singlemode
fiber
uplink
(1350nm)

• (2)
Single
mode
patch
cables

• (1)
Intel
Pro
10GBE
Single
Mode
LR
Server
Adapter
(PXLA8591LR)

• Antistatic
grounding
mat
and
antistatic
wrist
strap.

These
components
can
be
purchased
from
most
electronic

stores.


The
installation
is
fairly
straightforward.

As
with
any
hardware
configuration,
it
is
important
to
read
all
of
the
manuals

prior
to
component
installation.

The
following
instructions
will
guide
you
through
the
entire
process:


Hardware
Installation


1) Unpack
the
Supermicro
chassis.

Remove
the
fans,
drive
bays,
and
plastic
shrouds.

Ensure
the
chassis
is
well

grounded
during
the
entire
installation
process.

Install
components.

2) Install
the
controllers
and
Ethernet
card
exactly
as
shown
below:



 Page
2



 3) Connect
the
multilane
cables
into
the
chassis
backplane.

The
first
4
ports
(0‐3)
will
go
to
the
4‐port
card
and

the
remaining
connectors
(4‐23)
will
go
to
the
24‐Port
card:


4) Install
the
hard
drives.

Place
the
WD
Raptors
in
SAS/SATA
#0‐#3
and
the
WD
750GB
drives
in
SAS/SATA
#4‐
#23.



5) Install
the
system
memory,
processor
and
heatsink
and
air
shroud.

6) Disable
unnecessary
devices
(e.g.
serial
and
parallel
ports)
in
the
BIOS.

7) <ALT‐3>
into
the
4‐port
3Ware
card’s
BIOS.

Select
all
of
the
available
WD
Raptor
drives
and
create
a
RAID
1+0

array
called
“OS”.

Set
the
StorSave
profile
to
“Balanced”
and
write
cache
to
“enabled”.

Select
<F8>
to
save

and
exit.

8) <ALT‐3>
into
the
24‐port
3Ware
card’s
BIOS.

Select
one
of
the
750GB
drives
and
press
<s>
to
toggle
it
as
a
hot

swap
spare.

Select
the
remaining
18
750GB
drives
and
create
a
RAID6
array
called
“Data”.

Set
the
StorSave

profile
to
“Performance”
and
the
write
cache
to
“enabled”.
Select
<F8>
to
save
and
exit.

Allow
the
RAID6

array
to
initialize
(this
may
take
several
hours).

Upon
reboot,
you
will
see
two
cards
with
functional
arrays.


Software
Installation
and
Configuration


1) Power
the
system
off
and
unplug
the
24‐port
3Ware
card.

Set
it
off
to
the
side.

2) Update
3Ware
cards
and
system
BIOS
and
reboot.

3) Start
the
RedHat/CentOS
installation
and
go
through
all
of
the
standard
installation
options.

Let

RedHat/CentOS
install
the
default
partition
and
LVM
configuration
on
the
4‐port
3Ware
controller.

Install
all

of
the
default
package
selections
and
power
the
system
off
when
the
installation
is
complete.

4) Install
the
24‐port
3Ware
controller
back
into
the
system
and
boot
to
su
‐

5) Update
the
system
and
reboot:

a. Yum
–y
update
&&
reboot

6) Install
the
packages
necessary
to
get
the
system
up
and
running:

a. Yum
–y
install
samba
gcc
kernel‐headers
kernel‐devel
autconf
automake
openssl*
gcc‐c++





 Page
3



 



7) Download
and
install
the
latest
10Gbe
network
card
drivers
from
Intel.com
and
install:




8) Modify
network
startup
script
for
10Gbe
card:




9) Run
GNU
parted
to
create
a
GPT
disk.

a. Parted
/dev/sda

b. mklabel
gpt

c. Mkpart
primary
ext3
0‐<max
partition
size>

10) Create
EXT3
filesystem
and
mount.

**THIS
STEP
MAY
TAKE
SEVERAL
HOURS**

a. Mkfs.ext3
–m0
/dev/sda1

b. Quit

c. Mkdir
/data

d. Mount
–t
ext3
/dev/sda1
/data

e. Verify
with:
Df
–hT
/data



 Page
4






11) Add
quotas
to
grub.conf:




12) Set
file
system
performance
options:




13) Add
the
following
performance
adjustments
to
/etc/sysctl.conf:














 Page
5



 14) Add
adjustments
to
/etc/rc.local:




15) Download
bonnie++
and
benchmark
file
system:

a. Yum
–y
install
bonnie++




16) Adjust
fstab
to
reflect
the
new
file
system
and
check
mounts:




17) Edit
/etc/nsswitch.conf:




18) Set
startup
scripts
so
Winbind
and
Samba
start
on
boot:




19) Install
iSCSI
target
and
client
utilities:

a. svn
checkout
svn://svn.berlios.de/iscsitarget/trunk
i. make
&&
make
install
b. yum
install
‐y
iscsi*

20) Configure
iSCSI
target
and
access:

a. Change
the
Target
ID:

i. Target
2008‐1.com.domain:storage.disk0.sys.com

b. Change
incoming
user
to
reflect
your
identifier:

i. IncomingUser
iqn.2008‐04.com.remotebox:host
(password)

c. Change
LUN
paths
to
reflect
your
partition
configuration:

i. Lun
0
Path=/dev/vg0/data,Type=fileio

d. Change
the
Alias
to
reflect
the
proper
server
name:

i. Alias
Carbon



 Page
6






21) Start
iSCSI‐Target
on
boot:




22) Check
iscsi
connection
from
remote
node:

a. iscsiadm
–m
discovery
–t
st
–p
192.168.1.2

23) Edit
/etc/krb5.conf
to
reflect
your
domain
setup












 Page
7



 24) Configure
Samba
to
use
AD
via
the
/etc/samba/smb.conf
file:






















 Page
8



 25) Add
shares
to
/etc/samba/smb.conf
as
necessary:




26) Add
the
server
into
the
domain
and
test
the
connection

a. net
ads
join
–U
<admin
account>
‐S
<domain
controller>

b. Test
domain
membership:
net
ads
testjoin.

c. Test
user
enumeration:
wbinfo
‐u.


d. Test
group
enumeration:
wbinfo
–g.



✗ An
accurate
NTP
server
is
required
to
ensure
proper
domain
time
synchronization.







 Page
9



 27) Adjust
the
rules
in
Iptables
as
necessary:


28) Reboot
and
check
the
network
and
disk
configs:
dmesg
|
less:



 Page
10