Logo-English

GR

 

 

QA AUTOMATION ENGINEER

 

Parameterization or Randomization

"Insanity is doing the same thing over and over again and expecting different results".

Albert Einstein

While reviewing a test script that I had setup to run daily, I found a test case had failed and decided to investigate further.  In doing so, I used a different user to run against the same script, but ran manually due to the previous user not meeting the perquisites I have defined for that script. The solution, I found, turned out to be more difficult than I had been expecting.  After trying with the different user, the script returned 3 more different errors.

While analyzing these errors, I discovered something odd.  Why would the same script return different results for two different users?  The difference, I discovered, was that my new testing user had been registered as a male, instead of my original, which was registered as a female. My script, in a nutshell, uses a test account to login to a social network to review different languages strings that appear on the site.  The string differences on this site of a male and female user are vast.  But, as it relates to my script, I noted the 3 errors as a result of using a different gender.

How can I solve this for future testing scripts?

My script was not random enough to figure out these scenarios.  I didn't realize using different gender might result in different results.  Therefore, the need for parameterization, or randomization of my tests became a critical mindset for my script-writing approach.  But, what is parameterization?  And how do I code my testing scripts to be more flexible?

Definition.

What is it? Doing a quick internet search we can find some definitions and synonyms like Data-Driven Testing, Randomization, etc. Some of those definitions:

"parameterization is passing multiple input values instead of constant values to verify the functionality."

"The problem with testing any piece of software is that complexity blows up pretty quickly. The fact is, you can't test all possible combinations of parameters... This is where parameterization is used"

"Checking the same operations with multiple sets of data"

"Enables you to expand the scope of a basic test or component by replacing fixed values with parameters. This process, known as parameterization, it greatly increases the power and flexibility of your test or component."


In summary we can say that, parameterization is a procedure that allows the tester to review something more thoroughly through random variation of the data during the execution. But, why did I pick parameterization to talk about?

Why do we need to encourage parameterization on our tests?

I have seen a lot of blog posts related to Test Automation, most of them talk about a variety of testing tools, or how to perform different step, tricks and tips, or how to make scalable scripts, etc. However, I have found very few posts talking about the importance of parameterization or randomization while writing test scripts.

Sometimes a tester forgets that we are here to test apps, not for developing code.  Our main goal is to test and help deliver apps with the best quality and security possible. Therefore, we can't conceive automation itself as a test tool if we don't approach script writing with the thought of parameterization.  I strongly believe it can help us to save some time, and write less lines of code, if we do include a good approach to parameterization.

Therefore, when we work with our projects, we need to always try to parameterize them as much as possible.  For example,  below is a script I wrote to automate a daily test for gmail login:

1. Try to login with correct data, expecting for correct login in

2. Try to login without password, expecting failed login and error message

3. Try to login leaving email field empty, expecting failed login and error message

4. Try to login with incorrect email, expecting failed login and error message

5. Try to login with incorrect password, expecting failed login and error message

6. Try to login with both fields with incorrect values, expecting failed login and error message

And then, we would remember to test more deeply the regex that manage the validations for both fields and we would include tests like

7. Try emails with .com.xx

8. Try password with #"$% char.

and some more, our tests will look something like:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
import unittest, time, re

class Gmailexample(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.implicitly_wait(30)
        self.base_url = "https://accounts.google.com/"
        self.verificationErrors = []
        self.accept_next_alert = True
    
    def test_gmail_testcase1_correct(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Email").send_keys("exampleemail@gmail.com")
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("Passwd").send_keys("examplepassword")
        driver.find_element_by_id("signIn").click()
        self.assertEqual("Recibidos (1)", driver.find_element_by_link_text("Recibidos (1)").text)

    def test_gmail_testcase2_incorrect_email(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Email").send_keys("exampleincorrectemail@gmail.com")
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("Passwd").send_keys("examplepassword")
        driver.find_element_by_id("signIn").click()
        self.assertTrue(self.is_element_present(By.ID, "errormsg_0_Passwd"))

    def test_gmail_testcase3_incorrect_password(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Email").send_keys("exampleemail@gmail.com")
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("Passwd").send_keys("exampleincorrectpassword")
        driver.find_element_by_id("signIn").click()
        self.assertTrue(self.is_element_present(By.ID, "errormsg_0_Passwd"))
		
    def test_gmail_testcase4_noemail(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("Passwd").send_keys("examplepassword")
        driver.find_element_by_id("signIn").click()
        self.assertTrue(self.is_element_present(By.ID, "errormsg_0_Passwd"))
		
    def test_gmail_testcase5_nopassword(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Email").send_keys("exampleemail@gmail.com")
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("signIn").click()
        self.assertTrue(self.is_element_present(By.ID, "errormsg_0_Passwd"))
		
    def test_gmail_testcase6_bothincorrect(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Email").send_keys("exampleincorrectemail@gmail.com")
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("Passwd").send_keys("exampleincorrectpassword")
        driver.find_element_by_id("signIn").click()
        self.assertTrue(self.is_element_present(By.ID, "errormsg_0_Passwd"))
		
    def test_gmail_testcase7_extendedemail(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Email").send_keys("exampleemail@gmail.com.mx")
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("Passwd").send_keys("examplepassword")
        driver.find_element_by_id("signIn").click()
        self.assertTrue(self.is_element_present(By.ID, "errormsg_0_Passwd"))
		
    def test_gmail_testcase8_weirdchar_password(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Email").send_keys("exampleemail@gmail.com")
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("Passwd").send_keys("3xmpl3p$$word")
        driver.find_element_by_id("signIn").click()
        self.assertTrue(self.is_element_present(By.ID, "errormsg_0_Passwd"))

    def test_gmail_testcase8_different_domain(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
        driver.find_element_by_id("Email").send_keys("exampleemail@hotmail.com")
        driver.find_element_by_id("Passwd").clear()
        driver.find_element_by_id("Passwd").send_keys("3xmpl3p$$word")
        driver.find_element_by_id("signIn").click()
        self.assertTrue(self.is_element_present(By.ID, "errormsg_0_Passwd"))
    
    def is_element_present(self, how, what):
        try: self.driver.find_element(by=how, value=what)
        except NoSuchElementException, e: return False
        return True
    
    def is_alert_present(self):
        try: self.driver.switch_to_alert()
        except NoAlertPresentException, e: return False
        return True
    
    def close_alert_and_get_its_text(self):
        try:
            alert = self.driver.switch_to_alert()
            alert_text = alert.text
            if self.accept_next_alert:
                alert.accept()
            else:
                alert.dismiss()
            return alert_text
        finally: self.accept_next_alert = True
    
    def tearDown(self):
        self.driver.quit()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

 

However, as all testers know, we are forgetting a bunch of test cases:

a). Browsers, we have at least 3 most popular, each one with 3 or 4 versions out there

b). Max and min char allowed for each field

c). Allowed char for each field

d). Using an imcomplete domain, just username part of an email, weird domaings, etc on email field.

We don't necessarily need to modify our script, adding lines and lines of code for each test case.  This is where parameterization comes into play.  As in my initial example on this article, it wasn't necessary to modify the script, what was necessary was to modify, or add, the gender variable within the data file where this test was pulling from.

Back to the gmail example, we could generate a CSV file that looks something like:

 

email1@gmail.com,password1,success_assert
email2@gmail.com,password2,failed_assert
email3@gmail.com,password3,failed_assert
email4@gmail.com,password4,failed_assert
email5@gmail.com,password5,failed_assert
email6@gmail.com,password6,failed_assert
.
.
.
email1000@gmail.com,password1000,failed_assert

Where each line is a combination of email/password value related to a specific Test Case and we would use that file in a script looking something like:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
import unittest, time, re

class Gmailexample(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.implicitly_wait(30)
        self.base_url = "https://accounts.google.com/"
        self.verificationErrors = []
        self.accept_next_alert = True
		
		filename = 'test.csv'
		line_number = 1
		with open(filename, 'rb') as f:
			mycsv = csv.reader(f)
			mycsv = list(mycsv)
			self.username=mycsv[line_number][0]
			self.password=mycsv[line_number][1]
			self.assert=mycsv[line_number][2]
			self.verificationErrors = []
    
    def test_gmail_testcase1_correct(self):
        driver = self.driver
        driver.get(self.base_url + "/ServiceLogin?sacu=1&scc=1&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&hl=es&service=mail")
        driver.find_element_by_id("Email").clear()
		for index in range(len(self.username)):
			driver.find_element_by_id("Email").send_keys(self.username)
			driver.find_element_by_id("Passwd").clear()
			driver.find_element_by_id("Passwd").send_keys(self.password)
			driver.find_element_by_id("signIn").click()
			self.assertEqual(self.assert, driver.find_element_by_link_text(self.assert).text)

    def is_element_present(self, how, what):
        try: self.driver.find_element(by=how, value=what)
        except NoSuchElementException, e: return False
        return True
    
    def is_alert_present(self):
        try: self.driver.switch_to_alert()
        except NoAlertPresentException, e: return False
        return True
    
    def close_alert_and_get_its_text(self):
        try:
            alert = self.driver.switch_to_alert()
            alert_text = alert.text
            if self.accept_next_alert:
                alert.accept()
            else:
                alert.dismiss()
            return alert_text
        finally: self.accept_next_alert = True
    
    def tearDown(self):
        self.driver.quit()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

 

In conclusion, it's evident that parameterization helped to improve my script, and could help everyone's scripts, by allowing to take large datasets and use them in our testing scripts.  This also helps to reduce the size of a testing script and allows more flexibility by simply updating a CSV file, instead of several lines of code.

In my next post I'll show a couple of tools to help us parameterize our tests from different angles and we will avoid the hassle of figuring out data from the top of our heads.