The renewal maintenance has officially ended for Progress iMacros effective November 30, 2023.
This Wiki site will also no longer be moderated from the Progress side.
Thank you again for your business and support.
Sincerely, The Progress Team
Difference between revisions of "Web Testing"
Line 94: | Line 94: | ||
iMacros is different. It works by testing the complete web application inside the browser (Internet Explorer (IE), Firefox, iMacros Browser) and gives you 100% realistic testing data to accurately reflect true end-user experiences. This includes load testing websites with AJAX, Flash, Flex, Silverlight or Java applets. | iMacros is different. It works by testing the complete web application inside the browser (Internet Explorer (IE), Firefox, iMacros Browser) and gives you 100% realistic testing data to accurately reflect true end-user experiences. This includes load testing websites with AJAX, Flash, Flex, Silverlight or Java applets. | ||
− | This approach requires that each simulated user has his own browser instance. You can run several instances of iMacros at once, thus simulating concurrent users. This approach works very well for 1...50 users per average PC. The limit is only the number of IE or Firefox instances than can run on your machine. As a rule of thumb, simulating up to 50 concurrent users on a single machine is no problem. You can use the low cost iMacros Player license to install iMacros on many machines to increase the load at no extra cost. As an example, if you have access to five machines, you can generate a load of up to 250 concurrent users. | + | This approach requires that each simulated user has his own browser instance. You can run several instances of iMacros at once, thus simulating concurrent users. This approach works very well for 1...50 users per average PC. The limit is only the number of IE or Firefox instances than can run on your machine. As a rule of thumb, simulating up to 50 concurrent users on a single machine is no problem. You can use the low cost [http://www.iopus.com/imacros/compare/index.htm#player iMacros Player Unlimited User license] to install iMacros on many machines to increase the load at no extra cost ('''Unlimited''' Virtual Users). As an example, if you have access to five machines, you can generate a load of up to 250 concurrent users. |
You can use the STOPWATCH command to measure the web application performance in great detail during your load test. | You can use the STOPWATCH command to measure the web application performance in great detail during your load test. |
Revision as of 10:54, 4 February 2009
Performance Testing
(Related example macro: Demo-Stopwatch) (Related example script: get-response-times.vbs) iMacros includes full support for in-depth Performance Testing (Response Time Measurements).
The STOPWATCH command in iMacros allows you to measure the time that elapses between the first occurrence of the command in a macro ( = stopwatch on) and the second occurrence ( = stopwatch off). By using different identifiers in the STOPWATCH command you can create up to 100 independent measurement points in your macro.
In order to do web site response measurements you need to insert the STOPWATCH statements manually after you recorded your macro. For accurate measurements it is important to set the browsers replay speed to FAST so no artificial delays are added.
Demo Movie: Performance Web Testing with STOPWATCH
iMacros response time measurements always reflects the true user experience as they are measured using a real browser. Therefore response times measured by iMacros include loading times for browser plug-ins such as the Macromedia Flash Player or the Java runtime.
By default the measured times are saved to the Downloads\ directory of your iMacros installation (e.g. C:\Program Files\iMacros\Downloads\). The default file name is macroName_stopwatch.csv. You can instruct iMacros to save the data to a custom file name by setting the built-in variable !FILESTOPWATCH.
The values are comma separated (CSV format) so they can be viewed with any text editor, imported directly in Microsoft Excel or viewed by any other software you use. Additional information about the date and time of the measurements and the calling macro will be added to the response times.
In this example we measure response times of different parts of the iOpus homepage:
VERSION BUILD=4230323 SET !FILESTOPWATCH mydata.csv STOPWATCH ID=total URL GOTO=http://www.iopus.com/imacros/ STOPWATCH ID=1 TAG POS=1 TYPE=A ATTR=HREF:http://www.iopus.com/imacros/compare STOPWATCH ID=1 STOPWATCH ID=store TAG POS=1 TYPE=A ATTR=TXT:US$<SP>149 TAG POS=1 TYPE=INPUT:SUBMIT FORM=NAME:order ATTR=NAME:ORDER_PRODUCT_NOW&&VALUE:Order<SP>Now TAG POS=1 TYPE=A ATTR=HREF:http://www.iopus.com/store STOPWATCH ID=store STOPWATCH ID=total
This macro will create the following data in the "mydata.csv" file - obviously the response times will be different when you replay this macro. The format of the file is:
YYYY/MM/DD, HH:MM:SS, Macro name, ID, time (s)
"2007/08/3","11:56:23","mymacro","1","1.272" "2007/08/3","11:56:32","mymacro","store","8.943" "2007/08/3","11:56:32","mymacro","total","10.21"
For more information please see the Tips for Accurate Web Response Time Measurements.
Automating Response Time Measurements
(Related example script: get-response-times.vbs )
If you want to automate reponse time measurements it is likely that you will call iMacros from another application. Instead of writing the response time to a log file the data can be sent to your application via the Scripting Interface. Simply use the extract feature. The last recorded response time value is stored in the built-in !STOPWATCHTIME variable. You can use this variable as follows to transfer the response time data to the calling script or program:
VERSION BUILD=4230323
SET !FILESTOPWATCH NO
STOPWATCH ID=total URL GOTO=http://www.iopus.com/iim/iim/demo/v4 STOPWATCH ID=1 TAG POS=1 TYPE=A ATTR=HREF:http://www.iopus.com/iim/compare STOPWATCH ID=1 SET !EXTRACTADD {{!STOPWATCHTIME}} STOPWATCH ID=store TAG POS=1 TYPE=A ATTR=TXT:US$<SP>149 TAG POS=1 TYPE=INPUT:SUBMIT FORM=NAME:order ATTR=NAME:ORDER_PRODUCT_NOW&&VALUE:Order<SP>Now TAG POS=1 TYPE=A ATTR=HREF:http://www.iopus.com/store STOPWATCH ID=store SET !EXTRACTADD {{!STOPWATCHTIME}} STOPWATCH ID=total SET !EXTRACTADD {{!STOPWATCHTIME}}
Thus, the data is added to the !EXTRACT variable. Its contents can be obtained via the "iimGetLastExtract" command. In the above example it contains the three recorded response times (Example string):
1.272[EXTRACT]8.943[EXTRACT]10.21[EXTRACT]
Note: "SET !FILESTOPWATCH NO" instructs iMacros not to create a response time log file. This is useful if you intend to return the values via the Scripting Interface.
Tips for Accurate Web Response Time Measurements
- Add a CLEAR statement to your macro. This way you can make sure that the browser cache is cleared before each run. Otherwise iMacros might read the web pages from the cache and not the web server, which would most likely result in lower response times. Whether the cache is actually used depends on the Internet Explorer settings.
- Run the measurements as a loop and average several runs. The "internet speed" can fluctuate from minute to minute even on a fast connection. Therefore differences between each measurement run are normal. To get stable results it is good practice to average several runs. Several common programs such as Microsoft Excel can create averages automatically for you.
- If you compare results between different PC's, please keep in mind that the accuracy depends on the accuracy of the PC clock. This applies to all software that does time measurements on a PC.
- Under normal conditions the processor speed does not influence the measured response times. Only if the PC is so slow that the web page rendering of the browser is slowed down will the CPU speed have an influence on the measured response time. iMacros response time measurements always reflect the true user experience as they are measured using a real browser and the original browser plug-ins such as Macromedia Flash Player or SUN Java runtime.
- On very complex web applications it can happen that a STOPWATCH command around a URL GOTO command correctly measures the time for a page to open, but stops the time measurement when the page is still loading and displaying "Loading". In such a case, please use a keyword check (e. g. with a TAG or IMAGESEARCH) command to check that the page is loaded completely. This way you can measure the exact time from page access, to the completion of the page load.
Example: Instead of
STOPWATCH ID=Firstpage URL GOTO=http://www.iopus.com/imacros/demo/v6/flash/ STOPWATCH ID=Firstpage
Please use:
STOPWATCH ID=Firstpage URL GOTO=http://www.iopus.com/imacros/demo/v6/flash/ IMAGESEARCH IMAGE=flashEnter.bmp CONFIDENCE=90 STOPWATCH ID=Firstpage
Load Testing
Most load testing tools simulate users at the HTTP layer - simulating only the traffic that is generated by the browser, rather than simulating the entire browser.
iMacros is different. It works by testing the complete web application inside the browser (Internet Explorer (IE), Firefox, iMacros Browser) and gives you 100% realistic testing data to accurately reflect true end-user experiences. This includes load testing websites with AJAX, Flash, Flex, Silverlight or Java applets.
This approach requires that each simulated user has his own browser instance. You can run several instances of iMacros at once, thus simulating concurrent users. This approach works very well for 1...50 users per average PC. The limit is only the number of IE or Firefox instances than can run on your machine. As a rule of thumb, simulating up to 50 concurrent users on a single machine is no problem. You can use the low cost iMacros Player Unlimited User license to install iMacros on many machines to increase the load at no extra cost (Unlimited Virtual Users). As an example, if you have access to five machines, you can generate a load of up to 250 concurrent users.
You can use the STOPWATCH command to measure the web application performance in great detail during your load test.
Testing AJAX websites
Ajax-driven or JavaScript-heavy sites often require different strategies than when testing a regular site: elements appear, disappear and change at any time. Because of its built-in "intelligence" iMacros can automate many AJAX based websites using standard HTML based TAG commands. And it can automate absolutely all AJAX elements with DirectScreen Technology. This allows in-browser testing of even the most complicated AJAX elements, including sliders and drag & drop activity.
Related Demo-Macros:
Forum posts:
Reporting
iMacros has several reporting options. You can use the default reports or use the iMacros scripting features to create any kind of report.
1. Global logfile (lists any issues that might have occurred, such as website not available) with date and time.
2. Per macro reports, same as #1, but per macro.
3. Performance reports with the STOPWATCH command
4. Screenshots: iMacros can take screenshots during the iMacros run, so you can see what went wrong (e.g. an image missing or formatting issue).
5. Custom reports: You can create any kind of report via the Scripting Interface. This includes writing the test results to a log file (e. g. Combine-Macros.vbs) or taking a screenshot of the web browser after an error has occurred. You can also use this interface to connect iMacros directly with any kind of software or program, e. g. test planning software.
All functions and the test runs can be scheduled as well.
If you have any reporting requirement that is not covered here, please let us know.
Change User Agent
Related example: Set-User-Agent.vbs (VBS script)
Every time you access a web site the browser you use sends a string to the web server containing information about your operating system and the browser you are using. This string might, for example, look like this:
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)
Sometimes it is desirable to pretend to be a different user agent because some web sites change in behaviour or appearance depending on the user agent. iMacros can simulate all user agent strings with the -useragent command line switch. The command line switch can also be used in iimInit command of the the Scripting Interface:
iret = iim1.iimInit ("-useragent ""Nokia6230/2.0+(04.43)""")
If your user agent contains spaces, please use double quotes ("") around it.
You can see the current user agent of iMacros at http://www.iopus.com/imacros/demo/v6/user-agent.htm
In iMacros for Firefox please use the following command inside the macro:
SET !USERAGENT "New User Agent here".
So with iMacros for Firefox the user agent can be changed during the macro runtime. With Internet Explorer, the user agent is defined when Internet Explorer is started.
Web Testing FAQs
Q: How can I search for a specific keyword on a web page?
When parsing a web site you want to find out if a certain word exists on a web page to trigger further action (like saving the web page, printing the page or running the next macro).
There are many applications of the keyword search feature. For example tracking your search engine ranking, observing a (financial) news web site, get feedback on a failed login, watching web pages for changes or even monitoring trademark infringements automatically.
If the keyword does not exist, iMacros retries the keyword search automatically every second until the !TIMEOUT value is reached. If the keyword is still not found, iMacros reports an error. The automatic retry makes web testing websites with AJAX easy. For standard websites, you can also combine a keyword search with the REFRESH command to wait until a page changes. If you want to test without retry, please use SET !TIMEOUT 1 before the keyword search.
iMacros offers three options for keyword search (keyword assert):
1. TAG command 2. TAG... EXTRACT command 3. IMAGESEARCH command
Please see below for more details on each option. We also created the "Demo-Keyword-Check.iim" example macro.
Keyword Search with TAG
The TAG command searches for a HTML tag inside the web page. If found, it clicks on it. But TAG can also search for any kind of text on a web page. Thus, any TAG command is automatically a keyword assert command. For example
TAG POS=1 TYPE=STRONG ATTR=TXT:*iMacros*
searches for the word iMacros enclosed by the STRONG HTML tag. If you want to search for text regardless of the format, i.e. the enclosing HTML tag, please use the wildcard character * instead of a HTML tag:
TAG POS=1 TYPE=* ATTR=TXT:*iMacros*
This command is the same as above, but this time the command ignores the formatting of the keyword. This command takes longer, as iMacros needs to scan the complete page, not only certain html tags.
You can also "invert" the keyword search and generate an error if the keyword is found on the page:
TAG POS=1 TYPE=* ATTR=TXT:*iMacros* CONTENT=EVENT:FAIL_IF_FOUND
As the event suggests the TAG command will now generate an error if it succeeds.
Keyword Verification by Extracting Text
You can use the EXTRACT feature and make the keyword the extraction anchor. The general idea is to try and extract a certain piece of information and then check if the extraction result is the Extraction Anchor Not Found (#EANF#) message.
Example: We want to find out if the words "Order completed" are displayed on a web page. If yes, we want to print the page. To search the web page for the test phrase, create a macro called mysearch, which only has two lines:
VERSION BUILD=6140125 TAG POS=1 TYPE=* ATTR=TXT:*Order<SP>completed* EXTRACT=TXT
In this example we are searching the web page for the first occurrence (POS=1) of the keyword "Order completed". If the message #EANF# is returned then the keyword was not found as the keyword is the data extraction anchor. If the keyword was found then the EXTRACT command returns the complete text of the found HTML tag. In our example this could be "Software Order completed".
To print the web page create a macro called print_this. It has only two lines:
VERSION BUILD=6140125 PRINT
To connect both macros together create a small Windows script:
set iim1= CreateObject ("imacros") iret = iim1.iimInit() iplay = iim1.iimPlay("mysearch") extracted_text = iim1.iimGetLastExtract(1) 'test if keyword appeared on website. If iplay = 1 Then If instr (extracted_text, "#EANF#") > 0 Then MsgBox ("Sorry, keywords not found") Else iplay = iim1.iimPlay("print_this") End If End If If iplay < 0 Then MsgBox "Error!" End If
Note: The same procedure can be used to look for several keywords on a page, for example "cat", "dog" and "mouse":
The solution is to use several EXTRACT commands. So in the macro use:
TAG POS=1 TYPE=* ATTR=TXT:*mouse* EXTRACT=TXT TAG POS=1 TYPE=* ATTR=TXT:*cat* EXTRACT=TXT TAG POS=1 TYPE=* ATTR=TXT:*dog* EXTRACT=TXT
In the script you can look at each array element to see if they keyword was found:
iplay = iim1.iimPlay("wsh-extract-rate") If iplay = 1 Then If iim1.iimGetLastExtract(1) <> "#EANF#" Then MsgBox "Keyword CAT found!" If iim1.iimGetLastExtract(2) <> "#EANF#" Then MsgBox "Keyword DOG found!" If iim1.iimGetLastExtract(3) <> "#EANF#" Then MsgBox "Keyword MOUSE found!" End If
Keyword Search with IMAGESEARCH
You can use the IMAGESEARCH command to look for keywords or keyimages. Thus, instead of a text you supply an image of the word:
Example (Image of the word "iMacros"): imacros
The advantage of this command is that it works on any type of web page, including Flash, Java or ActiveX controls.
For more details please see the Image Recognition Plugin chapter.
Q: I want to test that certain images will show up on my site when a page is loaded.
(a) Check that an image is present
The Save Item function includes an image checkpoint. If a image is missing, an error will occur. Example:
CLEAR TAG POS=1 TYPE=IMG ATTR=TXT:*bee.jpg CONTENT=EVENT:SAVEITEM TAG POS=1 TYPE=IMG ATTR=TXT:*shark.jpg CONTENT=EVENT:SAVEITEM
This macro will download two images. If one image is missing on the web page the macro stops with an "image not found" error. To make sure that the missing image is not stored in the browser cache it is recommended that you use the CLEAR command before this test.
This technique is also recommended if you are automating or testing web sites with complex frames. To make sure a specific frame is completely loaded do the above test for a specific image in this frame. iMacros will wait at the TAG.....EVENT:SAVEITEM command until this image appears.
The method is easily implemented if you have information about the image such as its own URL, the link URL or the alternative text. If all you have is the image itself use the IMAGESEARCH command of the Image Recognition Plugin.
(b) Check the content of an image
Please use the IMAGESEARCH command if you need to verify the content of an image.
Q: Does iMacros support checkpoints?
Yes! Please see the Checkpoints page for more details.
Q: Does iMacros support multiple checkpoints in one macro?
By design, a macro stops when an error occurs (see checkpoints). To verify multiple checkpoints there are two strategies:
1. Split the task in several sub macros, with one checkpoint per macro. A macro can be very small, and even contain only one line. Example:
i = iim1.iimPlay ("Checkpoint1") if i < 0 then msgbox ("Checkpoint 1: Macro reports error.") i = iim1.iimPlay ("CODE:TAG POS=2 TYPE=A ATTR=TXT:*Download* ") if i < 0 then msgbox ("Checkpoint 2: Text not found.") i = iim1.iimPlay ("CODE:IMAGESEARCH IMAGE=result.bmp CONFIDENCE=90") if i < 0 then msgbox ("Checkpoint 3: Image not found.")
2. If you only need to verify text you can also use the Data Extraction features of iMacros can return the information to your script, and then process the result according to your business requirements.
Q: How to invoke 3rd party tools?
iMacros can be interfaced with any 3rd party tool. To achieve this, call both, iMacros and the 3rd party tool from a script. Any programming language can be used for this via the powerful iMacros Web Scripting Interface.
All programming languages have slightly different methods to run programs, but the basic idea is always the same. Here is an example for VB Script:
Example:
'Run iMacros Regression Test iErrorCode = iim1.iimPlay ("TestShoppingCartMacro") 'Call 3rd party tool Set objShell = CreateObject("WScript.Shell") iErrorCode = objShell.Run "3rdPartyTool.exe /cmd1 /cmd2" 'You can also run Windows command line tools (MSDOS, Batch files,...) directly iErrorCode = objShell.Run("ping www.mywebsite.com",,True)
Related Case Study: Agilent Technologies used the iMacros Scripting Interface to interface their WebTrans Software with iMacros.
If you have questions about a specific third party software, please ask our tech support.
Q: Is there any way to verify if any javascript errors have been thrown?
Yes! You can run a regression test and ensure no errors came out of Javascript. To log Javascript error messages, please make sure that the display of Javascript error message is enabled. To enable it, uncheck the "Disable Script Debugging" option in the Internet Explorer Settings Dialog. These settings are also used by the iMacros Browser.
For more details please see the Page Errors chapter and ONERRORDIALOG. Related forum post: Take screenshots when Javascript errors occur.
Q: Can iMacros run data driven tests?
Yes! iMacros can read data and use it to fill forms and/or make special decisions during the macro run. There are two methods to connect iMacros with data sources.
1. Read CSV files directly. iMacros can import data directly into the {{!COL1}}, {{!COL2}},... variables. Please see the Demo-Loop-Csv-2-Web macro.
2. Use the Scripting Interface to connect to any database.
3. Use the Scripting Interface to call iMacros from any programming or scripting language.
Q: What effect does iMacros itself have on web application response measurements?
The overhead of iMacros itself is very thin. On a 3 GHz Pentium each macro step takes only about 20 ms (0.02s) of processing time. If you compare this to the average page response time the delay caused by iMacros itself is typically much less than 2%. Note that a few commands need more processor time, for example the CLEAR, IMAGESEARCH or IMAGECLICK commands.
Since the iMacros Browser is an IE-compatible browser realistic viewer experience results are guaranteed. For example, your performance measurement results include the browser rendering time.
Q: How can I set up a 24 hours a day, 7 days a week (non-stop) operation with iMacros?
Many applications of iOpus iMacros require continuous operation of the software. Examples are the use of iMacros as part of payment gateway, extracting large volumes of information or web testing applications in general.
Problem: iMacros has the ability to remotely control the web browser and a wide variety of third-party browser plug-ins, such as Java virtual machine (Java applets), Adobe Acrobat Reader (PDF viewer), Macromedia Shockwave (Flash applets) and many others. By design most of these controls are not intended for 24x7 operations and running them repeatedly for several days can lead to undesirable effects, such as increased memory consumption ("memory leaks").
Solution: The Scripting Interface is a compact and reliable component that controls the iMacros browser and thus, all its plug-ins. The Scripting Interface was designed to automate web testing and at the same time to compensate for problems of the Windows Internet components used by the iMacros browser and third-party browser plug-ins.
The image shows stable CPU and memory usage over time running the sample code listed below with the "Demo-Flash" macro. The left part is a task manager snapshot taken on day 1 and the right part is taken on day 30. The snapshots were taken on a Windows Server 2003 with 2 GHz CPU and 512 MB Ram.
A periodic restart is not required. However, as with server application in general you should make sure that the application restarts correctly after the machine is rebooted or reset.
If you need to repeat a certain macro for example five million times do not start a single browser instance and use the Play (Loop) button. Instead use the Scripting Interface for the loop. In VBS code this looks like:
'Sample code A Set iim1= CreateObject ("imacros") For m = 1 to 5000000 iret = iim1.iimInit() iret = iim1.iimPlay("macro1") iret = iim1.iimExit() Next
This command sequence closes the browser after each macro and thus avoids browser memory leaks. Even if the browser or one of its plug-ins "hangs" or"crashes", the Scripting Interface is not influenced by this and will open a fresh browser instance and continue to work. However, closing and re-opening the browser (as the above example does) takes a few seconds on an average PC. So if processing speed is important for your application you can split this task in two loops and use iimExit only, for example, every 1000th loop to save time.
'Sample code B Set iim1= CreateObject ("imacros") For m = 1 to 5000 iret = iim1.iimInit() For n = 1 to 1000 iret = iim1.iimPlay ("macro1") Next iret = iim1.iimExit() Next
Related chapter: How to run iMacros unattended.
(tinyurl to this FAQ: http://tinyurl.com/6xl8h6)