xml - Meetup

bricklayerbelchedInternet and Web Development

Feb 5, 2013 (4 years and 6 months ago)

259 views


Starting from Scratch with
RC, Python, and Pysaunter


Mary Ann May
-
Pumphrey

Adobe EchoSign QA

02/28/12


From Scratch



> 1000 manual test cases in
TestLink



No Selenium automation



Very little Selenium knowledge amongst the
rest of the QA team members



> 20% of my time getting assigned to help
execute manual tests

Why RC?


Selenium
-
RC had several years


worth of bug
fixes when EchoSign decision was made in
06/11. Selenium
-
WebDriver was just about to
be released for the first time.



I had experience with RC, but not WebDriver,
and producing tests asap was a priority.

Why Python?


Selenium
-
1 supported C#, Java, Perl, PHP,
Python, & Ruby.



Selenium
-
2 quietly dropped Perl & PHP.



Java & Python seem best
-
suited for long
-
term
survival.



Why Python?


Interpreted languages are simpler to write
scripts in.



Python is easier for manual testers to learn
than Java, and we want our manual testers to
do at least some automation.



An IDE is less critical for Python; scripts can
be easily written & debugged with a
terminal/
cmd

window.



Why Pysaunter?


Priority was on getting tests developed
asap
,
rather than designing our own customized
framework.



Pysaunter

implements Page Objects.


Why Pysaunter?


Pysaunter

s creator
--
Adam Goucher
--
is a
widely
-
known and respected Selenium
contributor & consultant.



Framework supports both RC and WebDriver
tests, and provides examples of both.



Page Object Model


A test case comprises business logic,
understandable by anyone who reads it.



A page object file comprises locators,
assertion methods, action methods, strings,
etc., that are
specific

to a single page (or
even part of a single page).



Page Object Model


Selenium API calls should only appear in the
page object files, not in the test scripts
themselves.



If everything is done correctly, in theory, the
entire test suite could be switched to
something other than Selenium without any
modifications to the test scripts. (!?!)



pytest Flags

Every
EchoSign

test method has a
decorator
containing
pytest

flags, ordered from most
general to most specific, e.g.,


@
pytest.marks
('regression','search','new_search'
,


'
new_search_for_company_name'
,


'
EA
-
627')
)

def

test_new_search_for_company_name
(self):


Executing the Tests

To run just the
smoke

tests:


pysaunter.py

m smoke

v


To run
all

the tests:


pysaunter.py

m

smoke or regression



v




Structure of Test Suite


conf

conftest.py

files

l
ogs

m
odules

pytest.ini

s
cripts

support





Structure of Test Suite:

scripts & modules/pages dirs

Each file of test scripts, e.g.,



scripts/AddToDocument.py,


scripts/Upgrade.py, etc.


Has a corresponding PO file…



modules/pages/AddToDocumentPage.py,


modules/pages/UpgradePage.py, etc.








Example Test in

scripts/Registration.py

@pytest.marks('regression','negative','registration','register_without_any_in
put_values','EA
-
594')

def test_register_without_any_input_values(self):



r = RegistrationPage()


r.open_default_url()


r.do_register_without_input_values()



for msg in r.missing_messages.keys():


self.assert_(r.is_missing_message_present(msg),"No or wrong error
message present for " + msg)


Relevant PO Code:
modules/pages/RegistrationP
age.py

missing_messages = {


"missing_email": "Please enter your email address",


...


"missing_country": "Please specify your country",

}


def do_register_without_input_values(self):


self.se.click(locators['submit_button'])


def is_missing_message_present(self,key):


return self.se.is_text_present(self.missing_messages[key])


Structure of Test Suite:

modules/*py


Non
-
PO files



imap.py, retrieve.py, extract.py all deal with
fetching email



helpers.py contains very general functions

Structure of Test Suite:

files dir


EchoSign
-
specific, not part of Py.saunter



Contains various files to be uploaded,
archived, added to library, etc. by the test
cases.

Structure of Test Suite:

logs dir


latest.xml
+ dated
.xml
files contain
execution results



One dir/test with snapshots taken during run



Demo

Integration with TestLink


TestLink Keyword

manual


vs.

automated


allows manual testers to run queries to
retrieve just

manual


tests for regression.



Final pytest

mark


is always the TestLink id
of corresponding manual test, i.e.,


@pytest.marks('smoke','registration',

'register_free_account','EA
-
583')

Integration with Jenkins


CI test runs on our internal build system are
in our near
-
term plans.



In the meantime, two artifacts:


echosign_selenium.tar.gz

makes it very easy to
install the latest checked
-
in version of test suite

no P4 client needed.


list.html

shows entire list of auomated tests to
date.


Integration with FogBugz

Any bug found by Selenium automation has two
features:



Name of automated test somewhere in the
bug report



automation

tag for metrics purposes


Best Practices:

Coding Standards


Adhering to very basic items from the
Style
Guide for Python Code
:



Function & variable names: lowercase,
underscores_for_readability



Class names should be CapitalizeEachWord



Indentation should be 4 spaces per level

no tabs

Best Practices: skipif()


skipif
()
decorators, for tests that only work in
certain environments


@
pytest.mark.skipif
('
helpers.get_env
() ==
"preview"')


@
pytest.mark.skipif
('
helpers.get_env
() !=
"preview"')


Best Practices: skip()


skip()
decorators, for tests that aren

t quite
finished, or have been found to have a
problem


@pytest.mark.skip()


Best Practices: xfail()


xfail
()
decorators, for tests that are expected
to fail


@
pytest.mark.xfail
(reason="
FogBugz

13864")

Best Practices: Error Handling


Provide an error message for every assertion
in case it fails.



Put all these error messages in a dictionary at
top of class.



Use self
-
documenting dictionary keys.





Best Practices: Error Handling

error_strings = {


"is_archived_count_not": "The Archived count is not
",


.


.


.


"manage_page_not_displaying_only_archived_folder" :
"The Manage page is not displaying the Archived folder
OR is showing more folders also.

,


Best Practices: Error
-
Handling



self.assert
_

(

m.is_only_folder_displayed
("widget"),

self.error_strings
['
manage_page_not_displaying_only_
widget_folder
']

)




Best Practices: Code Reviews


Code reviews of test code required for check
-
in



Best Practices: Testing the
Test

1.
Temporarily put a
pdb.set_trace()

just
before the assertion in a new/passing test.


2.
Run the test.


3.
When the
pdb

prompt appears, delete a file
or an email or do <whatever> will best
simulate a failure situation.


1.
Ensure that the test fails.



Best Practices: Important
Tests vs. More Tests


Try to balance development of smoke tests
(important but hard) with development of
regression tests (less important but easier).



What

s Worked Well


Pysaunter creator Adam Goucher has been
very supportive, both in terms of bug fixes &
feature requests.



The Python RC API has presented virtually
no issues.


What

s Worked Well


Pysaunter

s image
-
capturing feature is very
useful. Much faster than video and usually
just as effective.



Pysaunter is

Just Enough Framework


for
our current needs.



Non
-
Selenium colleagues have been able to
add new tests soon after getting started.


Concerns


Lack of a Pysaunter community => no peer
support or best
-
practice discussion



Pysaunter

s future unclear



RC

s longevity unclear, as is LOE for
migration to WebDriver



Selenium/Python job
-
seekers seem fewer in
number than Selenium/Java job
-
seekers




Resources


Instructions

on how to install
RC/Python/Pysaunter from my blog



PushToTest
-
Sponsored Webinar:

Selenium
Python Page Objects




The Quick Python Book
,
2nd ed.





Resources


PushToTest
-
Sponsored Webinar:

Selenium
--
You Are Doing It Wrong




Python Module of the Week




The
Element
-
34 Blog



Q&A?


Now!


or…


maryann@adobe.com