Вы находитесь на странице: 1из 84
James A. Whittaker +docjamesw (Google+) @docjamesw (Twitter)

James A. Whittaker

+docjamesw (Google+)

@docjamesw (Twitter)

It's a book! It's a blog! It's a way of life! Follow me on •

It's a book! It's a blog!

It's a way of life!

Follow me on

Google+ (+docjamesw) Twitter (@docjamesw)

A few words about Google

Engineering culture

Fail fast

20% time

Project mobility

The Google testing philosophy

Quality ≠ Test

Google is an engineering/computer science centric company

Quality must be owned by engineering

Test must be part of engineering

Testing won't get done unless it is unavoidable

Test is everyone's job

Actual testers are rare, scarcity creates clarity

The result The SET role

Engineering Productivity not Testing

Engineering Productivity

A separate organization beholden to Google, not to Product Areas

"To accelerate Google" Tools from IDEs, compilers, build, test Testing, release engineering, engineering education

Product area leads negotiate with Eng Prod leads for test engineering Testers are assigned to product areas but report through Eng Prod Testers responsible for their own fate: hiring, promotion, reviews

Testing roles

SWE

Feature developers

code reviews, TDD, unit tests, acceptance tests own quality for every piece of code they touch

SET

Test infrastructure developers

code reviews, test frameworks, automation

TE Test scenario developers

risk analysis, end-to-end test scripts, quality management

Testing roles

Testing roles

Crawl, walk, run

"Minimum useful product"

Don't build a large product, start small and grow it

Release the moment you have something useful

Release and iterate Make gathering feedback part of the process

Don't be afraid to take chances

Fail fast, correct as you go

Release channels Canary (daily build)

Developer (weekly build)

Test (best weekly build) --> Dogfood

Beta (latest, most stable test channel build) --> Release

Types of tests

Google is a melting pot of testers from various companies, all using a different term for a different type of test

We standardized:

Small tests

Medium tests

Large tests

Enormous tests

Types of tests

Small tests

Almost always automated Run in a mocked/faked environment Exercise code within a single function or module Focus on functional issues such as data validity and error conditions

Execute within seconds

Mostly written by SWEs

Does this code do what it is supposed to do?

Types of tests

Medium tests

Usually automated Run in a fake or real environment Test the interaction between functions that call each other directly Focus on functional issues such as data validity and error conditions Execute within a few seconds Written by SETs and SWEs

Do a set of near neighbor functions interoperate the way they are supposed to?

Types of tests

Large tests

Can be automated or manual Run in an actual production environment Test end-to-end functionality Can take minutes to hours to run Written by TEs and SETs

Does the product operate the way a user would expect and produce the desired results.

Given all this, an enormous test should be self explanatory!

Automation vs manual

Automate when

Results can be checked by a machine No human judgement or cleverness is required Repetitive So important it can't be left to chance

Manual otherwise

Shared code

The secret?

"Committers" Similar to the open source idea of a person with "commit" status on a project At Google committers have language-specific "readabilities"

Code reviews All code must be reviewed and approved by a person with readability You earn your readability by a track record of writing good code

Platform dependencies

Don't deal with them, minimize them All engineers (dev and test) use the same OS on their desktop as we use in our data centers Local = Test = Staging = Production

Platform libraries are baked into OS images

Common build files for C++, Java and Python

Test Certified

Level 1

Set up test coverage bundles

Set up a continuous build

Classify your tests as Small, Medium, and Large

Identify nondeterministic tests

Create a smoke test suite Level 2

No releases with red tests

Require smoke test suite to pass before a submit Incremental coverage by all tests >= 50%

Incremental coverage by small tests >=

10%

At least one feature tested by an integration test Level 3

Require tests for all nontrivial changes Incremental coverage by small tests >=

50%

New significant features are tested by integration tests

Level 4

Automate running of smoke tests before submitting new code

Smoke tests should take less than 30 minutes to run

No nondeterministic tests

Total test coverage should be at least 40%

Test coverage from small tests alone

should be at least 25%

All significant features are tested by integration tests Level 5

Add a test for each non-trivial bug fix

Actively use available analysis tools

Total test coverage should be at least 60%

Test coverage from small tests alone should be at least 40%

A user facing test role

SWE - feature developer coding on behalf of the product

SET - test developer coding on behalf of the developers

TE - user developer coding on behalf of the user TEs must be engineers to remain first class citizens

TEs progress up the job ladder, some become product managers, some become UX designers, some SETs, some SWEs

The 10 minute test plan

Put 3 engineers in a room, hand them a spec and give them 10 minutes to build a test plan

Interrupt them after ten minutes, tell them time is up and give them a new problem and a fresh 10 minutes

Rinse and repeat until they figure it out

The 10 minute test plan

What we learned Avoid prose in favor of bulleted lists Don't bother selling No fluff, this is not a high school term paper If it isn't actionable, leave it out Make it flow, one section leads to another Guide a tester's thinking The outcome should be test cases

A test plan that doesn't lead to test cases is a waste of time

ACC

A ttributes C omponents C apabilities

Simple and powerful model for analyzing web apps

Think of the product in terms of

Adjectives and adverbs (attributes)

Nouns (components)

Verbs (capabilities)

A CC: Attributes

Adjectives and adverbs

Ask yourself

o

Why are we building this?

o

Why is this interesting to users?

Examples from Chrome

o

Fast

o

Secure

o

Stable

o

Elegant

Sources for attributes: requirements, advertising, packaging

A CC: Attributes are easy to discover

A CC: Attributes are easy to discover

A CC: Attributes are easy to discover

A CC: Attributes are easy to discover

AC C: Components

Nouns Ask yourself

What are the major parts of the product? Examples from Chrome

URL bar

Tabs

Chrome

Webkit

from Chrome • URL bar • Tabs • Chrome • Webkit Sources for components: design docs,

Sources for components: design docs, areas in bug databases, area

owners, common sense

AC C : Capabilities

Verbs

Ask yourself

What does this product do? Examples from Chrome

Load webpages

Sync preferences

Run web apps Key for capabilities: intersection of attributes and components

If the product does something that does not match an

attribute/components intersection ignore it or take it out

BITE Record & Playback

Tool Walk Through

BITE Bug Filing

Tool Walk Through

BITE Bug Visualization

Tool Walk Through

Quality Bots