Вы находитесь на странице: 1из 60
An Investigator’s Guide to File System Internals Brian Carrier carrier@atstake.com
An Investigator’s Guide to
File System Internals
Brian Carrier
carrier@atstake.com
An Investigator’s Guide to File System Internals Brian Carrier carrier@atstake.com

Overview

Motivation

Disk Layout (MBR and Partitions)

Basic File System Model

Classical File Systems

FAT (16-bit and 32-bit)

– UNIX (FFS and EXT2FS)

Journaling File Systems

– NTFS

– EXT3FS

Other File Systems

? 2002 @stake, Inc.

2

@stake

FIRST File System Tutorial

 

@stake

Motivation

Currently, most evidence is found in the file system

 

A thorough knowledge of the file systems allow one to:

– Make better use of forensics tools

– Recover deleted content when tools do not exist

 

– Allow one to describe what your automated tools are doing

This talk is going to cover basic & advanced topics of file systems for:

 

– Manual file recovery

– Finding evidence of “wiping” tools

– Knowing how to most efficiently use tools

 

? 2002 @stake, Inc.

3

FIRST File System Tutorial

 

@stake

What is a File System?

File Systems manage data storage.

 

Organized into files, unlike memory.

Provide an addressing scheme that is easy for humans to understand, unlike memory.

 

Examples: FAT, EXT2FS, FFS, NTFS, Reiser FS,

 

? 2002 @stake, Inc.

4

FIRST File System Tutorial

 

@stake

Tools

The @stake Sleuth Kit (TASK) will be used throughout this to show examples on real images.

 

The Coroner’s Toolkit (TCT) and TCTUTILs could also be used to collect some of the UNIX file system data.

TASK is a collection of forensics command line tools for file system analysis

The tools parse file system structures and do not rely on kernel support

? 2002 @stake, Inc.

5

FIRST File System Tutorial

 

@stake

Disk Layout (MBR and Partitions)

? 2002 @stake, Inc.

6

FIRST File System Tutorial

 

@stake

The Disk

A disk can be imagined as a long stream of millions bytes

 

A disk is typically broken up into one or more groups (or partitions)

Logical volume management systems do not directly follow this model because they can have several physical disks appear as one volume, which is then broken up into partitions.

We will cover x86 (DOS) partitions and disk labels

? 2002 @stake, Inc.

7

FIRST File System Tutorial

 

@stake

x86: Disk Layout

The bytes are grouped into sectors, usually 512-bytes each

 

Each sector is addressed, using Logical Block Addressing (LBA) or Cylinder / Head / Sector (CHS)

The first sector contains the Master Boot Record (MBR) and the Partition Table

The remaining sectors are organized into Partitions, which contain file systems

? 2002 @stake, Inc.

8

FIRST File System Tutorial

 

@stake

x86: Master Boot Record (MBR)

 

Located in first 512-bytes of disk

The first 446 bytes are x86 assembly instructions that read the partition table to identify a bootable partition

 

This is the sector that the systems reads when booting

The Partition Table starts at offset 446 and contains four 16- byte entries (48 bytes total)

The final 2-bytes contain a signature (0x55AA)

 

? 2002 @stake, Inc.

9

FIRST File System Tutorial

 

@stake

x86: Partition Table

Contains 4 entries, each can describe a partition

 

Entries are each 16-bytes long

Entries contain:

– Starting address of partition in CHS and LBA format

 

– Ending address of partition in CHS format

– Size of partition in sectors

– Flags (is it bootable?)

– Type of Partition

? 2002 @stake, Inc.

10

FIRST File System Tutorial

@stake x86: Partition Types • There are more than one partition type because only four
@stake
x86: Partition Types
• There are more than one partition type because only four
entries exist in the MBR Partition Table.
• Primary Partition: Contains 1 file system and can only
be described by one the four MBR partition table entries.
• Extended Partition: Contains 0 or 1 Secondary
Partitions and 0 or 1 additional Extended Partitions (a linked
list). Used when more than 4 partitions are needed.
• Secondary Partition: Contains 1 file system and
occupies either all or part of an Extended Partition.
? 2002 @stake, Inc.
11
FIRST File System Tutorial
@stake
x86: Partition Example
Disk
MBR
Primary
Primary
Primary
Extended
(NTFS)
(EXT2FS)
(EXT2FS)
Sec
Extended
(FAT)
Secondary
(NTFS)
? 2002 @stake, Inc.
12
FIRST File System Tutorial
 

@stake

x86: Partition Tools

 

Displaying:

– fdisk

– Partition Dump (pd) <http://heat.ca.sandia.gov/projects/software.html>

 

– GNU Parted <http://www.gnu.org/software/parted/>

 

Finding partitions when the table is gone

gpart <http://www.stud.uni-hannover.de/user/76201/gpart/>

? 2002 @stake, Inc.

13

FIRST File System Tutorial

 

@stake

disk labels: Overview

 

BSD systems use disk labels to describe a disk

The disk label contains the disk layout (partitions), disk geometry, label, model etc. (but no boot code)

 

It is located somewhere in the “beginning” of the disk

There is not a standard format, but the contents include:

– Offset (in sectors or cylinders)

 
 

– Size

– Type

 

– Flags

? 2002 @stake, Inc.

14

FIRST File System Tutorial

 

@stake

disk labels: x86-based BSD flavors

 

x86-based BSD systems (i.e. OpenBSD, FreeBSD) use both x86 partitions and disk labels

 

An x86 partition is created as the primary partition for the BSD system (so that it can be booted)

 

The disk label is located in the beginning of the BSD primary partition and is used by the kernel to identify where the partitions exist

 

The disk label can describe partitions that do not exist in the x86 partition tables

The labels can be viewed with the ‘disklabel’ command.

 

? 2002 @stake, Inc.

 

15

FIRST File System Tutorial

 

disk labels: x86-based BSD flavors example

 

@stake

 

Windows XP

OpenBSD

 
     
 
     
     
 
     
     
 

f

a

b

d

e

Only two x86 partitions are defined

The BSD disk label is located at the beginning of the OpenBSD partition (a)

 

The disk label describes 5 partitions (partition f has an offset earlier than it does)

? 2002 @stake, Inc.

 

16

FIRST File System Tutorial

 
 

@stake

disk labels: Sun

Sector 0 of the disk (slice 2) contains the Virtual Table Of Contents (VTOC) and partition descriptors:

 

– ID (boot, root, unassigned, swap, usr, backup, stand, var, home etc.)

– Permission Flag (not -mountable, Read Only)

– Starting Sector (or cylinder in some versions)

– Size of partition

Maximum of 8 or 16 partitions

Sectors 1-15 contain the bootblock code

 

The contents can be listed with:

prtvtoc /dev/rdsk/c0t0d0s2

The format command can also display the partition details

 

? 2002 @stake, Inc.

17

FIRST File System Tutorial

 

@stake

Basic File System Model

? 2002 @stake, Inc.

18

FIRST File System Tutorial

 

@stake

Very Basic File System Model

 

In general, data within a file system can fall into 1 of 4 categories

 

File System Layer

 

Descriptive data about the file system (i.e. block sizes, offsets, dirty status)

 

Content Layer

 

File content (i.e. blocks, clusters). Most data falls into this category.

 

Meta Data Layer

 

Descriptive data about files (i.e. file size, permissions, time of last access)

 

Name Layer

 

Human addressing format (i.e. file and directory names)

? 2002 @stake, Inc.

19

FIRST File System Tutorial

 

@stake

TASK tool names

 

Each tool in TASK is named based on the file system layers:

 

File System Layer:

 

fsstat

Content Layer (data):

 

dls, dcalc, dstat

Meta Data Layer (inode):

 

ils, istat, icat, ifind

Name Layer (file name)

 

fls, ffind

? 2002 @stake, Inc.

20

FIRST File System Tutorial

 

@stake

 

FAT

? 2002 @stake, Inc.

21

FIRST File System Tutorial

 

@stake

FAT12, FAT16, and FAT32 File Systems

Primitive File System – has been used since early DOS versions

Still used in Windows 9x and on floppies

Size limits of file system:

– FAT12: 32 megabytes (used in floppy disks)

 

– FAT16: 4 gigabytes

– FAT32: 8 terabytes

? 2002 @stake, Inc.

22

FIRST File System Tutorial

@stake FAT: File System Layout • The file system is organized into 3 areas: •
@stake
FAT: File System Layout
• The file system is organized into 3 areas:
• Boot Sector
Contains file system details such as sizes and offsets
• Data Area
Contains file and directory contents. Consecutive sectors in it are grouped
into clusters.
• Primary and Backup File Allocation Tables (FAT)
Contains tables that manage the clusters in the Data Area
? 2002 @stake, Inc.
23
FIRST File System Tutorial
@stake
FAT: File System Layout Diagram
Boot Sector
Data Area
File Allocation Tables
Cluster
? 2002 @stake, Inc.
24
FIRST File System Tutorial
 

@stake

FAT: Data Area

All file and directory content is stored in clusters within the Data Area

 

Each cluster is 1 or more (power of 2) consecutive 512-byte sectors. The size of which is saved in the Super Block.

Each cluster has an address and the first usable one is number 2

The File Allocation Table (FAT) manages the clusters

? 2002 @stake, Inc.

25

FIRST File System Tutorial

 

@stake

FAT: File Allocation Table

 

The File Allocation Table (FAT) contains as many entries as there are clusters in the Data Area

 

The table is indexed by the cluster number and the contents describe that cluster

Table entries can be:

– An unallocated marker, used when we need to allocate new clusters

 

– The address of the next cluster in a file or directory

 

– End of File (EOF) marker if it is the last cluster in the file or directory

 

– Bad sector marker, so it is not reallocated

 

The size of each table entry depends on the version of FAT (12-bits, 16-bits or 32-bits)

 

? 2002 @stake, Inc.

26

FIRST File System Tutorial

@stake FAT: File Allocation Table Example 0 0 3 4 5 E 0 8 9
@stake
FAT: File Allocation Table Example
0
0
3
4
5
E
0
8
9
E
0
1
2
3
4
5
6
7
8
9
• Files are located in Clusters 2, 3, 4, 5 and 7, 8, 9
• Cluster 6 is unallocated
• E is our End-Of-File marker
? 2002 @stake, Inc.
27
FIRST File System Tutorial
@stake
FAT: Directory Entries
• File and directory meta data are stored in Directory
Entry structures
• One directory entry is allocated for each file and directory
• Directory entries have a static size (32-bytes) and form a list
within the clusters allocated by the parent directory
• The clusters for the Root Directory (C:\) are statically
located in FAT12 and FAT16 at the first cluster (#2) and are
dynamically located in FAT32
• The Root Directory does not have a directory entry (i.e. it
has no meta data associated with it)
? 2002 @stake, Inc.
28
FIRST File System Tutorial
 

@stake

FAT: Directory Entry Structure

 

name in ASCII (8 bytes)

extension in ASCII (3 bytes)

attribute flags (1 byte) [Directory, Read Only, Hidden, System, Volume ID, Archive, Long Name Entry]

 

lowercase flags (1 byte)

Create time & date (5 bytes) (optional, and is 0 on some Windows 9X directories)

 

Access date (2 bytes) (no time, but is required by spec)

Written time & date (4 bytes) (optional)

 

Address of the first allocated cluster (4 bytes total)

Size of file (4 bytes)

? 2002 @stake, Inc.

29

FIRST File System Tutorial

 

@stake

FAT: Date and Time structures

 

FAT stores the actual time, not the number of seconds from a known date

 

Lower 5-bits (0x001f) of time are seconds (0-29) divided by 2

Next 6-bits (0x07e0) of time are minutes (0-59)

 

Upper 5-bits (0xf800) of time are hours (0-23)

Lower 5-bits (0x001f) of date are day (1-31)

Next 4-bits (0x01e0) of date are month (1-12)

Upper 7-bits (0xfe00) of date are year from 1980 (0-127)

 

? 2002 @stake, Inc.

30

FIRST File System Tutorial

 

@stake

FAT: Long File Name Directory Entries

 

The Directory Entry contains the file name, but is constrained to the 8.3 format.

Long File Name (LFN) Directory Entries are the same size as normal directory entries, but have a special value in the attribute flag (all bits are set).

 

Older versions of DOS happen to ignore directory entries with this flag.

Files with long names also have a short name directory entry after the LFN directory entries

Each LFN can hold 13 UNICODE letters, multiple ones exist for long names.

? 2002 @stake, Inc.

31

FIRST File System Tutorial

 

@stake

FAT: LFN Structure

sequence (1 byte) [EOF or linear value]

 

5 UNICODE letters of name (10 bytes)

attributes flag (1 byte)

reserved (1 byte)

checksum (1 byte) (equal for all LFN of a file)

 

6 UNICODE letters of name (12 bytes)

reserved (2 bytes)

2 UNICODE letters of name (4 bytes)

 

? 2002 @stake, Inc.

32

FIRST File System Tutorial

@stake FAT: LFN Example File is: abcdefghijklmnopqrstuvwxyz1234.txt Attribute Name 1234.txt LFN nopqrstuvwxyz LFN
@stake
FAT: LFN Example
File is: abcdefghijklmnopqrstuvwxyz1234.txt
Attribute
Name
1234.txt
LFN
nopqrstuvwxyz
LFN
abcdefghijklm
LFN
abcdef~1.txt
FILE
NOTES: Stored in “reverse” order and the SFN entry is the
only one with the starting sector and other meta data
? 2002 @stake, Inc.
33
FIRST File System Tutorial
@stake
FAT: Example
cluster 5
cluster 6
File
File
Root Dir
file1 - 5
dir1 - 7
Content
Content
cluster 7
cluster 8
foo.c - 8
File
bar.h - 13
Content
FAT
cluster 13
File
Content
6
E
E
9
10
E
0
0
14
E
5
6
7
8
9
10
11
12
13 14
? 2002 @stake, Inc.
34
FIRST File System Tutorial
 

@stake

FAT: File Lookup Example

 

Examine contents of c:\dir1\file1.dat

1. Identify location of the Root Directory sectors (from the super block)

 

2. Read the directory entries in the root directory sectors, looking for one with the name “dir1” and a directory attribute flag

3. When the entry is found, identify the starting cluster (16 for example)

4. Read cluster 16 and process the data as directory entries, looking for the entry with “file1” as the name and “dat” as the extension.

? 2002 @stake, Inc.

35

FIRST File System Tutorial

 

@stake

FAT: File Lookup Example

 

5. If an entry is not found, look in entry 16 in the FAT to identify the next cluster.

 

6. Repeat the lookups until the entry is found. Identify the starting cluster from the directory entry (23 for example).

7. Read the contents of cluster 23 and then lookup entry 23 in the FAT for the next cluster, 24 for example.

8. Read the contents of 24 and repeat the FAT lookups until <EOF> is reached

? 2002 @stake, Inc.

36

FIRST File System Tutorial

 

@stake

FAT: File System Layer

Two structures describe the file system

 

The Boot Sector structure describes file system attributes. The structure is different in FAT16 and FAT32.

 

– Cluster size

– Location and size of the FAT

– Start and end of the Data Area

– Magic value (0xAA55)

The File Allocation Table (FAT) manages the clusters within the Data Area

 

? 2002 @stake, Inc.

37

FIRST File System Tutorial

 

@stake

FAT: TASK & sectors

TASK uses sectors for addressing instead of clusters

 

The first cluster is hundreds of sectors into the file system, after the Super Block and FATs

Therefore, sectors are used as addresses so that we can access the contents of the FAT and Super Block

? 2002 @stake, Inc.

38

FIRST File System Tutorial

 

@stake

FAT: fsstat

 

The fsstat tool lists the important Boot Sector values and a summary of the FAT.

 
 

# fsstat –f fat fat.dd

Boot Sector Values:

 

– Volume ID

– Sector and Cluster size

– Sector of first Cluster

– Sector Range

– Locations of primary and backup FAT

FAT Data

Sequence of consecutive sectors that are allocated and either the next nonconsecutive sector or EOF.

 

? 2002 @stake, Inc.

39

FIRST File System Tutorial

 

@stake

FAT: Content Layer

 

The clusters in the Data Area contain the file and directory content data.

 

We can view the contents of the clusters using the ‘dcat’ command in TASK, or using ‘dd’.

NOTE: TASK will require the sector address and not the cluster.

 

# dcat –f fat fat.dd 390

We can get the statistics of a sector using the ‘dstat’ command:

 
 

# dstat –f fat fat.dd 390

? 2002 @stake, Inc.

40

FIRST File System Tutorial

 

@stake

FAT: Meta Data Layer

 

The Directory Entries describe the files and directories in the file system.

 

We can list of details of which using the ‘ils’ command from TASK:

 

#

ils –e –f fat fat.dd

We can get more details of a specific one using ‘istat’:

 
 

#

istat –f fat fat.dd 5

We can view the contents of the file using that directory entry, use ‘icat’:

 
 

#

icat –f fat fat.dd 5

? 2002 @stake, Inc.

41

FIRST File System Tutorial

 

@stake

FAT: TASK & Directory Entries

Directory entries in FAT are not given addresses

TASK treats every 32-bytes as a possible directory entry

The Root Directory has no directory entry, so it is given an address of 2 and the first 32-bytes in the first cluster are given the address 3. The second 32-bytes are given the address 4 etc.

Not every address will be a directory entry

? 2002 @stake, Inc.

42

FIRST File System Tutorial

 

@stake

FAT: Name Layer

All names are stored in the Directory Entry structure.

 

We can get a listing of the file and directory names using the ‘fls’ command:

# fls –rp –f fat fat.dd

? 2002 @stake, Inc.

43

FIRST File System Tutorial

 

@stake

FAT: File Allocation

Results from testing a Windows 2000 system

 

The new directory entry is placed at the end of the existing list. Entries from deleted files are not reallocated, this results in a LONG list of deleted files.

The clusters are allocated by scanning the FAT for clusters that are free. I’m not 100% sure of the algorithm yet:

– If the size is known, it will look for enough consecutive clusters (i.e. it will skip over single free ones if it needs several of them)

– If the size is unknown or it is from a redirection, it allocates single free clusters (but sometimes it skips over single ones and allocatesseveral consecutive ones - maybe due to buffering)

In the last cluster, only the sectors that contain data are overwritten (i.e. slack space)

? 2002 @stake, Inc.

44

FIRST File System Tutorial

 

@stake

FAT: File Deletion

Directory entries are unallocated by setting the first letter (which is also the first byte of the structure) to ‘0xEF’

 

No other directory entry values are changed:

 

– The name will still exist

– The times do not reflect the deletion time

 

The clusters are unallocated by setting the corresponding FAT entries to 0

 

? 2002 @stake, Inc.

45

FIRST File System Tutorial

 

@stake

FAT: Allocation & Deletion Tools

 

We can see how files are allocated by using the following tools:

 

– ‘ils –e’ to list the status of all ‘inodes’

 

– ‘fls –rp’ to list all known directory entries

– ‘fsstat’ to get the contents of the FAT

– ‘dls -le’ to list the status of sectors

Save the output of these tools, create or delete files, and then compare the new tool output

 

? 2002 @stake, Inc.

46

FIRST File System Tutorial

 

@stake

Forensics Notes

Because meta data is not in a static location, it can exist for a long time before it is overwritten

 

Deleted directory entries still contain names and starting cluster - but not a deleted time

Directory entry allocation algorithm is such that deleted entries are not overwritten

Only the Access time is required to be updated and it is only accurate to the day. Created and written times are not consistently written

Deleted data can be found in the slack space of files

 

Wiping tools will sometimes leave long file names such as “aaaaaaaaaaaaaaaaaaaaaaaaaaaaa”

 

? 2002 @stake, Inc.

47

FIRST File System Tutorial

 

@stake

UNIX File Systems (FFS and EXT2FS)

? 2002 @stake, Inc.

48

FIRST File System Tutorial

 

@stake

UNIX File Systems

Solaris, HP, and other BSD flavors use the Berkley Fast File System (FFS)

 

Linux uses the Extended 2 File System (EXT2FS) and the Extended 3 File System (EXT3FS)

EXT2FS is based on FFS

We will present FFS first and then show the differences with

 

EXT2FS

? 2002 @stake, Inc.

49

FIRST File System Tutorial

 

@stake

UNIX: File System Layout

 

The File System is divided into groups, called Cylinder Groups in FFS (Block Groups in EXT2FS)

 

All groups are the same size

 

Groups contain both meta data structures and file content

 

Each group manages its own resources

 

? 2002 @stake, Inc.

50

FIRST File System Tutorial

@stake UNIX: File System Layout Diagram Group ? 2002 @stake, Inc. 51 FIRST File System
@stake
UNIX: File System Layout Diagram
Group
? 2002 @stake, Inc.
51
FIRST File System Tutorial

UNIX FFS: Cylinder Groups

@stake

All Cylinder Groups in a given file system are the same size, which is identified in the Super Block

The first group starts at offset 0 (but the first 16 are reserved)

Each group contains a copy of the Super Block, but it increments its offset (it has an offset of 16 for the first group due to the disk labels)

Each group contains a Group Descriptor structure that describes the group. The inode and block bitmaps are stored in the group descriptor.

? 2002 @stake, Inc.

52

FIRST File System Tutorial

@stake UNIX FFS: Blocks & Fragments • Sequential disk sectors (within the same group) are
@stake
UNIX FFS: Blocks & Fragments
Sequential disk sectors (within the same group) are
organized into Blocks
• Blocks may also be broken into Fragments (which are
usually larger than a sector)
• Each Fragment is addressable
• Example:
– Sector Size: 512-bytes
– Block Size: 4096-bytes (8 sectors)
– Fragment Size: 1024-bytes (2 sectors)
• Each group has a bit-map that specifies the allocation
status of each fragment
? 2002 @stake, Inc.
53
FIRST File System Tutorial
@stake
UNIX FFS: Blocks & Fragments Diagram
0
1
2
3
4
5
6
7
512-byte
4096-byte
1024-byte
sector
block
fragment
? 2002 @stake, Inc.
54
FIRST File System Tutorial
 

@stake

UNIX FFS: inodes

File and directory meta data are stored in inode structures

 

A static number of inode structures are allocated at file system creation time

Each group contains the same number of inode structures, which are stored in an inode table.

Multiple files may use the same inode (hard-links)

 

Inodes have unique addresses, starting with the number 2

 

The Root directory (/) is always described by inode 2

Each group contains a bit-map that describes the allocation status of inodes in that group

? 2002 @stake, Inc.

55

FIRST File System Tutorial

 

@stake

UNIX FFS: Summary of the inode Structure

mode (2-bytes) [ permissions & type]

 

number of links (2-bytes)

size (8-bytes)

Last access date & time (8-bytes)

 

Last modified date & time (8-bytes) – file data modifications

 

Last change date & time (8-bytes) - file status (inode) change

User ID (4-bytes) and Group ID (4-bytes)

 

Partial list of allocated blocks (direct pointers) and list of blocks with additional pointers (indirect pointers)

 

? 2002 @stake, Inc.

56

FIRST File System Tutorial

@stake

UNIX FFS: Block Pointers

FFS was designed on research that showed that small files were more common the large files

An inode contains 12 direct, 1 single-indirect, 1 double-indirect, and 1 triple-indirect pointer

Direct Pointer: Points directly to a block that contains the file or directory content

Single-indirect Pointers: Points to a block that contains a list of direct pointers, which point to the data

Double-indirect Pointers: Points to a block that contains a list of single-indirect pointers

Triple-indirect Pointers: Points to a block that contains a list of double-indirect pointers

? 2002 @stake, Inc.

57

FIRST File System Tutorial

? 2002 @stake, Inc. 57 FIRST File System Tutorial @stake UNIX FFS: Block Pointer Diagram single

@stake

UNIX FFS: Block Pointer Diagram

single

single double direct 19 20 21 22 23 24 25 26 27 28 29 30 31

double

single double direct 19 20 21 22 23 24 25 26 27 28 29 30 31

direct

19 20 21 22 23 24 25 26 27 28 29 30 31 99 direct
19
20 21
22 23
24 25
26
27 28
29
30 31
99
direct
single
32 33
34
35 100
101
direct
200
201
32 - file
content
200 - file
content
19 - file content
19 - file
content

? 2002 @stake, Inc.

58

FIRST File System Tutorial

 

@stake

UNIX FFS: Time & Date Format

 

UNIX saves the time & date as the number of seconds since 1/1/1970 UTC.

 

? 2002 @stake, Inc.

59

FIRST File System Tutorial

 

@stake

UNIX FFS: Directory Entries

 

File and directory names are stored in Directory Entry structures that are located in blocks allocated by the parent directory (similar to FAT, but different contents)

 

Each structure contains a name and points to an inode structure

Directory Entry structures have a dynamic size and form a linked list

Multiple directory entries can point to the same inode (symbolic links)

The ‘.’ and ‘ directories

directory entries are the first two in all

? 2002 @stake, Inc.

60

FIRST File System Tutorial

@stake UNIX FFS: Directory Entry Structure • inode (4-bytes) • record length (2-bytes) • type
@stake
UNIX FFS: Directory Entry Structure
• inode (4-bytes)
• record length (2-bytes)
• type (1-byte) {This does not exist in all versions}
• name length (1-byte) {This is 2-bytes if type does not
exist}
• name (less than 255, NULL terminated)
• The record length is rounded up to a multiple of 4
? 2002 @stake, Inc.
61
FIRST File System Tutorial
@stake
UNIX FFS: Directory Entry Example
rec len
inode
name len
name
16
8357
5
file1
16
8358
5
file2
16
8359
5
file3
16
8357
file1
16
8358
file2
16
8359
file3
NOTE: The ‘.’ and ‘
entries are not shown
? 2002 @stake, Inc.
62
FIRST File System Tutorial
 

@stake

UNIX FFS: File Lookup Example

 

Lookup /dir1/file1.dat

1. Locate inode 2 in the Inode Table of Group 1.

 

2. Read the blocks allocated to inode 2 and process them as directory entries.

 

3. Analyze each directory entry for the one with the name “dir1”. Save the inode value in the directory entry, 1810 for example.

4. Calculate which group inode 1810 is part of by dividing it by the number of inodes per group, group 2 for example.

5. Locate inode 1810 in the Inode Table of Group 2.

? 2002 @stake, Inc.

63

FIRST File System Tutorial

 

@stake

UNIX FFS: File Lookup Example

 

6. Read the blocks allocated to inode 1810 and process as directory entries.

 

7. Look for an entry with the name “file1.dat” and save the associated inode value, 2020 for example.

8. Calculate which group inode 2020 is part of by dividing it by the number of inodes per group, group 3 for example.

9. Read the inode 2020 structure from the Group 3 Inode Table and the blocks allocated by it.

? 2002 @stake, Inc.

64

FIRST File System Tutorial

 

@stake

UNIX FFS: File System Layer

Two types of structures describe the file system

 

The Super Block structure describes file system attributes.

– Size of Blocks & Fragments

– Size of groups

– Number of blocks and inodes per group

– Offset of group descriptor within each group

The Group Descriptor structure within each Cylinder Group describes the group attributes

 

– Location of inode bitmap

– Location of block bitmap

? 2002 @stake, Inc.

65

FIRST File System Tutorial

 

@stake

UNIX FFS: fsstat

The fsstat tool lists important Super Block values and the groups layout

 

# fsstat –f solaris solaris.dd

Super Block Values:

– Block and fragment sizes

– Range of block addresses

– Range of inode addresses

Group Info:

– Block and inode range of each group

– Location of inode table, group descriptor, and Super Block in each group

 

? 2002 @stake, Inc.

66

FIRST File System Tutorial

 

@stake

UNIX FFS: Content Layer

 

The blocks and fragments contain the file and directory content data.

 

We can view the contents of the clusters using the ‘dcat’ command in TASK (& TCT/TCTUTILs), or using ‘dd’.

 

# dcat –f solaris solaris.dd 390

 

We can get the statistics of a sector using the ‘dstat’ command:

 
 

#

dstat –f solaris solaris.dd 390

? 2002 @stake, Inc.

67

FIRST File System Tutorial

 

@stake

UNIX FFS: Meta Data Layer

 

The inode structures describe the files and directories in the file system.

 

We can list of details of which using the ‘ils’ command:

 

#

ils –e –f solaris solaris.dd

We can get more details of a specific one using ‘istat’:

 
 

#

istat –f solaris solaris.dd 200

We can view the contents of the file using that inode with ‘icat’:

 
 

#

icat –f solaris solaris.dd 200

? 2002 @stake, Inc.

68

FIRST File System Tutorial

 

@stake

UNIX FFS: Name Layer

All names are stored in Directory Entry structures.

 

We can get a listing of the file and directory names using the ‘fls’ command:

# fls –rp –f solaris solaris.dd

? 2002 @stake, Inc.

69

FIRST File System Tutorial

 

@stake

UNIX FFS: Block Allocation

Blocks are allocated to the same Cylinder Group as the inode. If the Cylinder Group is full or near full, a new one is chosen based on disk rotation and percentage of free blocks.

 

Solaris will not fill up a group with a large file. It will fill up half of the available blocks, then move on to the next group. OpenBSD will fill up the group before moving on.

The remainder of the fragment is wiped (i.e. no slack space)

? 2002 @stake, Inc.

70

FIRST File System Tutorial

 

@stake

UNIX FFS: Inode & Directory Entry Allocation

File inodes are allocated in the same Cylinder Group as the

parent directory.

If it is full, a quadratic hash search is

performed.

Directory inodes are allocated in the next Cylinder Group with a greater than average number of free inodes

 

The first available inode (lowest address) in the group is used

Directory Entries are inserted into the list at the first available location

? 2002 @stake, Inc.

71

FIRST File System Tutorial

 

@stake

UNIX FFS: File Deletion

 

Blocks and Fragments: Blocks are unallocated in the bitmap (including indirect blocks).

 

Inode: The inode is set to unallocated in the group’s inode bitmap. The size and block pointer fields are reset to 0. The Modified & Changed times are set to the current time.

Directory Entry: The record length of the previous directory entry is increased by the length of the deleted entry. Solaris will also reset the inode value to 0.

? 2002 @stake, Inc.

72

FIRST File System Tutorial

@stake UNIX FFS: Deleted File Example rec len inode name len name 16 8357 5
@stake
UNIX FFS: Deleted File Example
rec len
inode
name len
name
16
8357
5
file1
16
8358
5
file2
16
8359
5
file3
file2 is deleted
rec len
inode
name len
name
32
8357
5
file1
16
8358
5
file2
16
8359
5
file3
32
8357
file1
16
8358
file2
16
8359
file3
? 2002 @stake, Inc.
73
FIRST File System Tutorial
@stake
UNIX FFS: Allocation & Deletion Tools
We can see how files are allocated by using the following
tools:
– ‘ils –e’ to list the status of all inodes
– ‘istat’ to get a list of all blocks and fragments that were allocated to a given
inode
– ‘fls –rp’ to list all known directory entries
– ‘dls -le’ to list the status of blocks and fragments
? 2002 @stake, Inc.
74
FIRST File System Tutorial
 

@stake

 

EXT2FS

? 2002 @stake, Inc.

75

FIRST File System Tutorial

 

@stake

UNIX EXT2FS: Block Groups

 

EXT2FS calls the groups Block Groups - They are not organized by disk geometry

 

Each group manages itself and contains a duplicate copy of the Super Block (which is always at the same offset)

Each group has a table of Group Descriptors (FFS only had the local one)

Entire blocks are allocated to the inode and block bitmaps

The Sparse Super Block option prevents the Super Block from existing in every group and the space is used for data

? 2002 @stake, Inc.

76

FIRST File System Tutorial

 

@stake

UNIX EXT2FS: inode

The EXT2FS inode contains much of the same data as FFS

 

They have a different layout though

 

The EXT2FS inode has a deletion time

? 2002 @stake, Inc.

77

FIRST File System Tutorial

 

@stake

UNIX EXT2FS: Block Allocation

 

Blocks are allocated in the same Block Group as the inode (if there is room). It tries to allocate a block where the next 7 are also free.

 

Attention is not paid to one file filling up a Block Group.

? 2002 @stake, Inc.

78

FIRST File System Tutorial

 

@stake

UNIX EXT2FS: Inode & Dir Entry Allocation

File inodes are allocated to the same block group as parent directory, unless there is no room. Then they are allocated to another based on a Quadratic Hash or linear search.

Directory inodes are allocated to the block group with the most free blocks and greater than average free inodes. The search begins at group 0, so it could be the same one as the parent directory.

In both cases, the first (lowest address) inode is used.

Directory entries are placed in the first available location in the list.

? 2002 @stake, Inc.

79

FIRST File System Tutorial

 

@stake

EXT2FS: File Deletion

Linux changed how they deleted files from the 2.2 kernel to

 

2.4

Blocks & Fragments: Unallocated in the group bitmap.

 

Inode: The inode is set to unallocated, but the block pointers and size are not reset (the size is set to 0 when a directory is deleted though).

The Modified, Changed, and Deleted times of the inode are set to the current time.

Directory Entries: The length of the previous entry is increased. In the 2.2 kernel the inode value was not cleared. As of 2.4 it is set to 0, as FFS does.

? 2002 @stake, Inc.

80

FIRST File System Tutorial

 

@stake

EXT2FS: Other Tools

debugfs runs on Linux and will let you view and change any structure

 

? 2002 @stake, Inc.

81

FIRST File System Tutorial

 

@stake

UNIX: Forensics Notes

Times still exist in unallocated inodes, but usually not the directory entry pointer (except Open & FreeBSD FFS).

 

With EXT2FS we still have access to the block pointers

We can identify meta data wiping tools because there will be zeroed inodes in between allocated inodes in a CG

Blank directory entries in the middle of a directory are also suspect

To find the blocks for an unallocated inode, start looking in the CG, not the beginning of the disk (use ‘fsstat’ and ‘dls’)

By manually parsing the directory entry structure, you can sometimes tell some order of file deletions

? 2002 @stake, Inc.

82

FIRST File System Tutorial

 

@stake

Journaling File Systems

 

? 2002 @stake, Inc.

83

FIRST File System Tutorial

 

@stake

Journaling File Systems: Motivation

 

When a system crashes or is not shutdown properly, the file system must be validated

 

The ‘fsck’ tool scans inodes, directory entries, and bitmaps to ensure that all structures agree

This can take a LONG time on large file systems

 

? 2002 @stake, Inc.

84

FIRST File System Tutorial

 

@stake

Journaling File Systems: Overview

A log is kept that contains meta data (and sometimes content) changes, atomic transactions are used.

Before the data is written, the intended updates are recorded

If the system crashes, the log is consulted and it can easily identify which transactions did not complete. It then either retries the update or turns back to the previous state.

No public forensic tools currently take advantage of these logs.

Examples: NTFS, EXT3FS, IBM JFS

 

? 2002 @stake, Inc.

85

FIRST File System Tutorial

 

@stake

 

NTFS

? 2002 @stake, Inc.

86

FIRST File System Tutorial

 

@stake

NTFS

Based on HPFS (designed by Microsoft and IBM)

 

Used in Microsoft Windows NT, 2000, & XP

 

Design is not widely published by Microsoft, but a sourceforge project exists to write a Linux driver and document it

 

NTFS is object oriented and very flexible

 

There are too many data structures to address in detail in this talk (refer to TASK or Linux source)

 

? 2002 @stake, Inc.

87

FIRST File System Tutorial

 

@stake

NTFS: File System Layout

 

3 Major Areas (similar to FAT)

Boot Sector

Contains file system details such as sizes and offsets

Master File Table (MFT)

 

Manages the files and other objects

Data Area

 

Content area for files and objects

? 2002 @stake, Inc.

88

FIRST File System Tutorial

@stake NTFS: Layout Example Boot Sector Data Area Master File Table ? 2002 @stake, Inc.
@stake
NTFS: Layout Example
Boot Sector
Data Area
Master File Table
? 2002 @stake, Inc.
89
FIRST File System Tutorial
@stake
NTFS: Master File Table
• The Master File Table (MFT) contains an entry for each
user and system file
• MFT entries are called File structures and have a static size
of 1024 bytes
• MFT entries store file and directory meta data
• Data is stored in attribute structures (objects)
• Some files will require multiple MFT entries (due to static size)
• Multiple files may use the same MFT entry (hard-links)
• MFT entries are addressed (File Reference) and have a
sequence number
? 2002 @stake, Inc.
90
FIRST File System Tutorial
 

@stake

NTFS: MFT Entry Examples

 

The MFT itself and mirror copy (0 & 1)

LogFile (2)

Root Directory (5)

Volume Bitmap (3)

Bad Clusters (8)

Quota data (9)

All files and directories (25+)

 

? 2002 @stake, Inc.

91

FIRST File System Tutorial

 

@stake

NTFS: Attributes

An attribute describes a specific property of the file or directory (i.e. the name or dates)

 

Attributes have a header and a value.

 

The header is always stored in the MFT entry

The value is stored in either the MFT entry (resident) or in the Data Area (non-resident)

 

? 2002 @stake, Inc.

92

FIRST File System Tutorial

 

@stake

NTFS: Attribute Header

The attribute header is always located in the MFT

 

It has the following fields:

– type (4-bytes) & id (2-bytes)

– length (4-bytes)

– resident flag

– name length and name (all have a name except the primary data one)

 

– If resident: A pointer to attribute stream

 

– If non-resident: A run-list of the clusters that contain the attribute data

 

– Virtual Cluster Number (VCN): The address of the cluster in a run with respect to the beginning of the attribute (0 - n), not the beginning of the file system (Logial Cluster Number - LCN)

? 2002 @stake, Inc.

93

FIRST File System Tutorial

 

@stake

NTFS: Attribute Types

Standard Information: Read Only, Archive, MAC Times, number of links (type 16)

 

File Name(s) (type 48)

Data (type 128)

Index Root (type 144) and Index Allocation (type 160) (for directory contents)

 

Security Descriptor (type 80)

Anything: For example, the icon on a Macintosh file is stored in an attribute so it gets moved with the file (more type 128)

 

Each attribute type has its own data structure

 

? 2002 @stake, Inc.

94

FIRST File System Tutorial

 

@stake

NTFS: Standard Information Attribute

 

Attribute Type 16

Dates and Times:

– Creation

– Last MFT change

– Last write

– Last access

Permissions in DOS format

version

Owner Id

Security Id

? 2002 @stake, Inc.

95

FIRST File System Tutorial

 

@stake

NTFS: File Name Attribute

 

Attribute Type 48

MFT Reference Number

MFT Sequence Number

Dates and Times:

– Creation

– Last MFT change

– Last write

– Last access

Character set

Name length and name

? 2002 @stake, Inc.

96

FIRST File System Tutorial

 

@stake

NTFS: Data Attribute

Attribute Type 128

Data attributes just contain the raw data

 

The name of the data is given in the attribute header

 

The only attribute that can not have a name is the primary data attribute, which is typically given the name $Data by default.

Directories can have data attributes in addition to their B- Tree attributes

? 2002 @stake, Inc.

97

FIRST File System Tutorial

 

@stake

TASK & NTFS Attributes

MFT entries are addressed the same way in NTFS as inodes are with UNIX file systems

 

One can give the Type and Id (if there are more than one of a given type) for a specific instance in the form of:

MFT-TYPE-ID

For example:

# icat ntfs.dd 300

# icat ntfs.dd 300-128 (both are the same)

 

# icat ntfs.dd 300-128-3 (if an ADS exists)

? 2002 @stake, Inc.

98

FIRST File System Tutorial

 

@stake

NTFS: Data Area

Data that can not fit in the MFT entry is stored in the Data Area

 

The Data Area is organized into Clusters (groups of consecutive sectors)

Cluster numbering starts with the first partition sector (unlike FAT)

The clusters are managed using the Volume Bitmap MFT Entry (#6)

? 2002 @stake, Inc.

99

FIRST File System Tutorial

 

@stake

NTFS: Directories

Directories store the file descriptors in B-trees, which are sorted by any descriptor (file name by default)

 

The B-tree nodes contain the:

 

– MFT reference number

– The structure that is being sorted by (file name structure for example)

 

The MFT entry also stores the file name in an attribute

Names are stored in UNICODE

 

The Root Directory (\) is pointed to by a static MFT entry (number 5)

 

? 2002 @stake, Inc.

100

FIRST File System Tutorial

 

@stake

NTFS: File Lookup Example

 

Lookup \dir1\file1.dat

1. Read the Boot Sector to locate the beginning of the MFT

 

2. Read the MFT Entry for the Root Directory (#5)

 

3. Find & read the Index Root and Index Allocation attributes

 

4. Process the B-Tree and analyze the file name structures for the one with the name “dir1”. Save the file reference value, 180 for example.

5. Read MFT entry 180

6. Find & read the Index Root and Index Allocation attributes

 

? 2002 @stake, Inc.

101

FIRST File System Tutorial

 

@stake

NTFS: File Lookup Example

 

7. Process the B-Tree and find the file name structure with “file1.dat”. Save the reference value, 181 for example.

 

8. Read MFT entry 181.

9. Find & read the “$Data” attribute

 

? 2002 @stake, Inc.

102

FIRST File System Tutorial

 

@stake

NTFS: File System Layer

 

Several structures describe the file system

The Super Block contains the location of the MFT and the cluster sizes

 

The MFT entry in the MFT describes the location and size of itself

The $Log MFT entry contains the atomic transaction logs

All of these can be viewed with the ‘fsstat’ command

 

#

fsstat –f ntfs ntfs.dd

? 2002 @stake, Inc.

103

FIRST File System Tutorial

 

@stake

NTFS: Content Layer

 

Data is stored in the MFT itself if it is small enough

 

Larger data is stored in clusters in the Data Area

Clusters are numbered from the beginning of the file system

The dcat tool will allow us to display the contents of any cluster

 

#

dcat –f ntfs ntfs.dd 8016

The icat tool will allow us to display the contents of attributes that are stored in the MFT:

 
 

#

icat -f ntfs ntfs.dd 76-160

? 2002 @stake, Inc.

104

FIRST File System Tutorial

 

@stake

NTFS: Meta Data Layer

 

All file and directory meta data is generally located in the MFT

 

If the value of the meta data is too large, it will be located in the Data Area

Each MFT is addressed using a File Reference Number

We can view the details of an MFT entry using istat:

 

#

istat –f ntfs ntfs.dd 5

We can list them all using ils:

 
 

#

ils –f ntfs –e ntfs.dd

The contents of the file can be viewed using icat (with no type or id value):

 
 

#

icat -f ntfs ntfs.dd 9901

? 2002 @stake, Inc.

105

FIRST File System Tutorial

 

@stake

NTFS: Naming Layer

 

File names are stored in a node of the B-Tree in the directory’s MFT entry

 

The file’s MFT entry also has all names that it uses (multiple ones if it has hard-links)

The list of known file names can be shown with fls:

 

# fls –f ntfs –rp ntfs.dd

 

? 2002 @stake, Inc.

106

FIRST File System Tutorial

 

@stake

NTFS: File Allocation

An area following the MFT is reserved for its expansion

 

MFT entries are allocated linearly

 

Names are allocated into directories based on their name and location in the B-tree

 

Only the sectors in the last cluster that are used are overwritten (slack space)

? 2002 @stake, Inc.

107

FIRST File System Tutorial

 

@stake

NTFS: File Deletion

Content Blocks are unallocated by clearing the bits in the bitmap

 

MFT entry is freed by setting the in-use flag. The attributes are unchanged. (link count never goes to 0).

No times are set to the current time

 

The name still exists in the MFT entry

Parent directory Index Tree is resorted (the data may still exist in the tree depending on the other files in the directory)

 

? 2002 @stake, Inc.

108

FIRST File System Tutorial

 

@stake

Forensics Notes

Because of MFT allocation, it is likely that unallocated entries will be reused faster than with UNIX inodes

 

No times are set when an MFT is unallocated, so time lines of file activity are not as revealing

Deleted file names will not exist as frequently as with FAT or UNIX because they will get overwritten when the tree is sorted

We can collect slack space data

 

Clean MFT entries in the middle of the MFT is likely a result of wiping tools

 

If times in Standard Info are different than File Name, wiping tools may have been used

? 2002 @stake, Inc.

109

FIRST File System Tutorial

 

@stake

NTFS: Other Tools

NTFS File Sector Information Utility (nfi) (Microsoft)

 

NTFS Info (www.sysinternals.com)

 

? 2002 @stake, Inc.

110

FIRST File System Tutorial

 

@stake

 

EXT3FS

? 2002 @stake, Inc.

111

FIRST File System Tutorial

 

@stake

EXT3FS: Overview

EXT3FS is identical to EXT2FS except it interfaces with the Journal layer

 

The journal is kept in a special file or a specific journaling device

Currently, no tools publicly exist to utilize the journal for forensics

? 2002 @stake, Inc.

112

FIRST File System Tutorial

 

Summary of Structures

 

@stake

 

FAT

NTFS

UNIX

File System

Boo Sector,

Boot Sector,

Super Block,

FAT

MFT

Group Desc

Content

Clusters in the Data Area

Clusters in the Data Area

Fragments &

Blocks in

   

groups

Meta Data

Directory

MFT Entries

Inodes

Entries

Name Layer

Directory

B-tree Index

Directory

Entries

trees

Entries

? 2002 @stake, Inc.

 

113

FIRST File System Tutorial

 

Log-based File Systems

@stake

? 2002 @stake, Inc.

 

114

FIRST File System Tutorial

 

@stake

Basics of Log-structured File Systems

 

“Classical” File Systems waste time by seeking to allocated blocks and overwriting the contents (for data or inodes)

 

Log-structured File Systems work like a log file and append new data to the “end” of the data

All data and meta data are written to the current pointer.

An inode index file points to the current location of the inode

The inode points to the current location of its data blocks

Deleted Content is not overwritten for a while

 

No Forensic Tools currently take advantage of Log FS

Example: BSD Log File System, Linux Log-Structured FS

 

? 2002 @stake, Inc.

115

FIRST File System Tutorial

 

@stake

PALM OS

Everything is a database (even applications)

 

Each database stores the dates of Creation, last modification, and last backup

When a database is deleted, it is flagged as not in use, but not deleted (until a hotsync is done)

 

See Joe Grand’s talk on the pdd Palm Acquisition tool (Wednesday)

? 2002 @stake, Inc.

116

FIRST File System Tutorial

 

@stake

Summary

Knowledge of file system structures will help interpreting the results of tools

 

Knowledge will also help write tools when ones do not exist

Evidence of wiping tools can be found by examining the structures and their location

Using allocation algorithms, one can sometimes recover deleted data more quickly

? 2002 @stake, Inc.

117

FIRST File System Tutorial

 

@stake

Software References

TASK can be found at:

http://www.atstake.com/research/tools/task

 

TCT & TCTUTILs (for UNIX only) can be found at:

www.fish.com

www.porcupine.org

www.cerias.purdue.edu/homes/carrier

? 2002 @stake, Inc.

118

FIRST File System Tutorial

 

@stake

References

 

FAT

– Microsoft FAT32 File System Specification

 

– Microsoft Long Filename Specification

FFS

The Design and Implementation of the 4.4 BSD Operating System (McKusick, Bostic , Karels , Quarterman)

 

EXT2FS

 

– The extended-2 filesystem overview (Oxman)

– Design and Implementation of the Second Extended Filesystem (Card, Ts’o, Tweedie)

 

? 2002 @stake, Inc.

119

FIRST File System Tutorial

 

@stake

References

 

EXT3FS

Transcript of Stephen Tweedie talk (olstrans .sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html)

 

NTFS

– Inside Windows 2000 - 3rd Edition (Solomon, Russinovich)

 

– Linux NTFS Project (linux-ntfs.sourceforge.net)

? 2002 @stake, Inc.

120

FIRST File System Tutorial