Quality Assurance - McCombs School of Business

basheddockΛογισμικό & κατασκευή λογ/κού

21 Φεβ 2014 (πριν από 3 χρόνια και 1 μήνα)

51 εμφανίσεις

CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

Quality Assurance

November 2013

Kevin Collins, Ameer Noorani, & Chip Morgan

UT Austin


MIS 374 Guest Lecture



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

2

Who is Kevin?

Kevin Collins


UT BBA MIS, 2013


Minor in Marketing


College hire with 2 summer internships and 4 months of real
-
world experience


Employer History


Austin Technology Council


Database intern, managed & maintained member data


USAA Summer Internship (Summer ‘11)


CTO, worked with web analytics (clicktrail data via
DOTcom
, mobile, etc.)


USAA Summer Internship (Summer ‘12)


FASG PD, assisted in new effort to migrate a new accounting package for
mutual funds/insurance


USAA


software developer in FASG PD working with investment projects

Hobbies


Music


Couch coaching sports


Intramural Softball


Social Media @
kevinkwc






CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

3

Who is Ameer?

Ameer Noorani


UT BBA MIS, 2008


UTSA MBA, 2014


5 years experience delivering software projects


Employer History


University of Texas at Austin


student technician for media console, hardware, and helpdesk support


USAA Summer Internships


tester, tested applications and created automation scripts for investment projects


USAA


software developer on wealth management, investments, and financial advice projects


USAA


software developer on financial crimes projects


Hobbies


Football


Swimming


Softball


Video Games





CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

4

4

Who is Chip?


Chip Morgan


UT BBA MIS, 1991


18 years experience delivering software projects for a wide range of clients in various states of peril


Employer History


Neiman Marcus


developer and QA/methodology


BSG


project consulting and staff training


BSI


product development


Context Integration


curriculum development, project consulting


GreenGrape

Technologies


technical SME, architect, developer


SunGard Consulting Services


project consulting and sales


USAA


technical architect, wealth management and financial advice


Hobbies


Beach volleyball


Disc golf


Jabber/XMPP


NAESB Smart Grid Initiative




CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

5

EVERYONE IS RESPONSIBLE FOR QUALITY!

Project Manager

Functional Analyst

Technical Lead

Quality Assurance Sr/Lead

Quality Assurance Engineer

Business Manager

Project Delivery Manager

Software Developer

Business SME

Change & Release Coordinator

UI Producer

UI Designer



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

6

Agenda

What is Quality Assurance?

Waterfall vs. Agile

Automation Testing

Test Environments

Other Types of Testing

QA Challenges & Opportunities



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

7

Quality Assurance


What is it?


Industry Definition




“A Software Quality Assurance Engineer is involved in the entire software
development process to ensure the quality of the final product. This can include
processes such as requirements gathering and documentation, source code
control, code review, change management, configuration management, release
management and the actual testing of the software.”





USAA QA Mission Statement




“Our mission is to provide value
-
added testing services and solutions for
USAA Information Technology programs and products in order to meet the
business customer's quality, cost, and time to market goals.”



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

8

Quality Assurance
-

Importance


Brand


Apple


iPhone

4 (Antenna reception issue)


Toyota


Accelerator


Adoption of New Tools and Functionalities


HealthCare.gov (current events on new Health Care site)


User Experience


Live Money Applications


USAA Members/Channel of Choice


Deployed members have limited access to Internet (mobile.usaa.com)


Applications must work!



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

9

Quality Assurance


Profile of an Ideal Tester


Passionate About Testing


Detail Oriented


Business and Technical Knowledge



Understand Impacts


Good Communication Skills


Analytical Thinking


Issue Resolution


Key Traits for a Tester

Key Traits for
Anyone

in IT



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

10

Quality Assurance


Cost of Fixing Defects



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

11

Requirements

Design

Construction

Implementation

open
-
ended

close
-
ended

TYPICAL QUESTION TYPES THROUGHOUT THE PROJECT

-

What is the current process?


-

What are your current pain
points?


-

What are your priorities?


-

Would you like to have a “Select
All” option on the results page?


-

When user selects cancel, do you
want a “Are you sure?” prompt?



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

12

QA at USAA
-

2012 Production Changes

FOR INTERNAL USE ONLY: May be shared with

System Testing:


15,194 Defects Found


242,079 Test Cases Executed


Production Monitoring:


3,025 Defects Found


275 High/2,750 Medium & Low

Release Regression Testing:


163 Defects Found


15,022 Scenarios Executed

Automated Production Validation:


196 Defects Found


94,340 Scripts Executed



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

13

Waterfall vs. Agile



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

14

14

Waterfall

Requirements

Design

Development

Testing

Deployment


Structured approach


Familiar to most developers world wide


Some organizational requirements dictate process over dates


NASA for instance






CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

15

15

Waterfall

Requirements

Design

Development

Testing

Deployment


Typically some overlap






CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

16

16

Waterfall

Requirements

Design

Development

Testing

Deployment


Date driven can yield this:






CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

17

17

What’s Agile?



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

18

Agile
-

Scrum Team Iteration

Product
Backlog

Iteration
Backlog


User

Stories Tasks

Iteration

Daily
Standup

Analysis

Design

Code

Test

Release

User Story

Iteration
Demo

Iteration
Retrospective

End of
Release

User Story



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

19

Agile
-

Iteration Retrospective



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

20

Example: Actual Project
(Waterfall vs. Agile)



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

21

Waterfall Requirement vs. Lean Agile Story

Waterfall Requirement:


The system shall notify status on Salesforce.com when an Investment Review
has been generated.



Lean Agile Story:


As an advisor I want to be notified when an Investment Review is generated
so that I can schedule time to review it
.


Validation Criteria’s:

-
Validate that a task is created in
Salesforce

for the advisor when an IR is generated

-
Validate that an email notification is sent to the advisor when an IR is generated




CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

22

Waterfall Timeline


Project Execution Plan

Project Starts
in September

Implements Next Year
as
One

Big Release to
Production



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

23

Agile Timeline


Project Execution Plan

-
Changed Project Execution Strategy from Waterfall to Agile


-
Entire Project Team (including Business Partners) Went Through Agile Training


-
Divided Project Scope into
Four

(Epic) Business Deliverables From
One

Big Release



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

24

Agile Timeline


What Actually Happened

-
Epic 2 Was Delivered
90 Days

Earlier Than Original Plan!


-
From Demos/Lab Prod Feedback, Business Realized Epic 4 Was
Not

Needed!




CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

25

Agile
-

Continuous Integration

Who Broke the Code Build!?!!


Developer’s Code is Merged Frequently Into Test Environment


No Waiting Till the End for Integration Testing


Better Chances to Catch Missed Requirements Earlier


Better Test Coverage


Better Unit Testing & Iteration Functional Testing





CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

26

Test Environments



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

27

Test Environments @ USAA

Unit Testing Environment


-
Developer’s Workstation

-

Local Environment

Integration Testing Environment


-
Developers Sandbox for Agile or
Waterfall

-
DEV Environment

System Testing/Iteration
Functional Testing Environment


-
Project Test Environment (Waterfall)

-
Iteration Functional Testing (Agile)

-
Demo Environment (Agile)

Enterprise System & Release
Testing Environment


-
Merge & System Test with
Entire
Enterprise

Efforts Targeting a
Release Date (Waterfall or Agile)

-
Runway/Release Test Environment

Production QA
Environment


-
Sneak Preview/Pilot of New
Functionality to
Live

Users

-
Validate

New Functionality

Production


-
Validate

New
Functionality is
Present

Lab Production
*(Agile)


-

A Slice of
Prod

-
Feedback from Users in
Agile Projects

-
Projects Have
Full

Control



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

28

























Lab
Prod

Large

Lab
Prod

Lab
Prod

Lab
Prod

Lab
Prod

Lab
Prod

Small

Lab
Prod

Large

Lab
Prod

Small

Lab Prod

Flow



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

29

Automation Testing



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

30

Automation Testing


What is Automation?


“Use of special software to control the execution of tests and compare actual
outcomes to predicted outcomes. Test Automation can automate repetitive
testing scenarios and/or help execute additional test scenarios which are
difficult to perform manually.”



Actual Lessons Learned by a Project:


Essential for Building in Quality on Efforts


Allows for Fewer Testing Resources


Allows for More Testing Coverage in a Iteration








CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

31

Automation Testing

Refactoring


Re
-
organizing part of the code


Used for maintenance, performance, and new functionality addition


Increases code quality & testing coverage



For example, if a

programmer wants to add new functionality to a
program, he may decide to
refactor

the program first to simplify the
addition of new functionality in order to prevent

difficulty with
maintenance.







CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

32

Because of Refactoring…

When testing in Iteration 1….


Test Iteration 1 with every code change


In Iteration 2…..


Test Iteration 2 AND Test Iteration 1


What about Iteration 14?





CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

33

Automation Testing Benefits


Reduce manual testing


Ability to Quickly Regression Test Applications


Allows for on
-
demand execution


Driven by code changes


Ensures completed stories are still working properly


Finds defects for closed stories


Dedicated Lab Automation Lead


Remove impediments


More accurate estimates


Participates in Ceremonies





CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

34

Automation Testing Challenges



Maturation


Current State: Manual Testers and Automation Testers


Future State: Manual and Automation Tester



Environment Stability


Test Data





CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

35

Testing Challenges &
Other Types of Testing



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

36

Testing Opportunities & Challenges


Mobile Testing


Browser Testing Across Various Browsers


Test Data


Perfect Test Data


Miss key scenarios


Bad Data


Rabbit hole


Test Environments Stability


Multiple Efforts Making Changes


Testing in Agile
-

Maturity



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

37

Other Types of Testing


Performance Testing


Testing for response times on applications


Testing for volume of users on applications


Tools:
Dyna

Trace, Load Runner



Security Testing


Testing for OWASP Top 10 security issues
(ex: Cross
-
Site Scripting, SQL Injection, etc.)


Ensure the security of member data in usaa.com and mobile devices


Tools: Automated and manual
(e.g. IBM Security Appscan, Webscarab, and Paros)



Accessibility Testing


Functional testing for users with disabilities
(ex: vision, color
-
blindness, etc)


Ensure accessibility of applications across channels
(ex: USAA .com, mobile, etc.)


Tools: JAWS / NVDA / WAV toolbar / Compliance Sheriff



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

38

Performance Testing


Why Should You Do It?


User Experience
(# of Seconds for Backend Service Calls)


Response & Wait Time On Application


Asynchronous vs. Synchronous System Design



System Capacity
(# of Users Accessing System)


Projected Business Growth


Average Peak Volume vs. Highest Peak Volume


Marketing Campaigns



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

39

Security Testing


Six Basic Concepts


Confidentiality


Need to know basis


Integrity


Information is correct


Authentication


Identify person/message


Authorization


Access control


Availability


System will work


Non
-
Repudiation


Message delivery





CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

40

Accessibility Testing

“Laws and public opinion frown upon discriminating against people with
disabilities”


Perceivability


Images across alternative media types, color contrast, text size


Operability


Flashing content and risks to users (epileptics)


Understandability


Comprehension of content, text legibility


Robustness


Application of technology,
javascript
, etc.





CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

41

Recap

Quality Assurance


Responsibility & Importance

Waterfall vs. Agile

Automation Testing


Reduce Manual Testing & More Test Coverage

QA Challenges & Opportunities


Environment Stability, Test Data, Browsers

Non
-
Functional Testing


Performance, Security, and Accessibility Testing

Various Test Environments & Phases


Prod QA, Lab

Prod



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

42

Questions

CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

44

44

Scrum Process



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

45

Waterfall Timeline


Project Execution (Slide 1 of 3)



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

46

Waterfall Timeline


Project Execution (Slide 2 of 3)



CONFIDENTIAL: Must not be duplicated, shared or printed without USAA EMG approval.

47

Waterfall Timeline


Project Execution (Slide 3 of 3)