Can your Selenium do that? Testing flash/flex and silverlight in web apps with iMacros

John McCaffrey, Wednesday, April 8, 2009 @ 4:00 am

Having learned a long time ago the value of automated testing tools like Selenium, jMeter, and soapUI, I'm always on the lookout for new improvements in these tools. While I love Selenium and other frameworks like it, it has the limitation of not being able to test Flash/Flex/Silverlight or Java Applets. But if you need to test flash and silverlight components of your web app, in an automated way, the  iMacros testing tool might be worth checking out.

No Free Ride

While the free version of the iMacros plugins for InternetExplorer and Firefox allow powerful web scripting similar to Selenium, to be able to do the flash/flex and silverlight, you have to get the paid version or the 30-day trial. I downloaded the trial version to see how it compares to Selenium and what kind of damage I could to.

Going through some of the online demos, I saw you can actually do an image comparison to assert that something is visible in flash, and even have the test click on that part of the app.  The process involved taking a screen shot of the app, grabbing the section you want to use as the target, and then specifiying if you want the script to just assert that section is contained with in the app, or if you want it to also click on it.

I  followed the example to create a test that opened up a flash swf file and navigated through the app, asserting that certain image sections, like buttons and instructional messages appeared in the flash app. While the process wasn't that hard, I could see how it would be brittle as the tests would break whenever slight changes to the graphics happened.

One possible way to defend against brittle tests is to share the image assets that make up a flash app and its assertions between the flash designer and the tester. If the testing person has the access to the source code and images that make up the app, the script could refer to 'start_button.png' from the source tree, and even if the designer updates that file, the test will still pass.

You don't have to use the image detection to drive the script, you can just have it click on things according to their position, but the image detection was more interesting to me,  and I noticed that if you resized your window, the positional detection would be incorrect.

Feeding the script with User Data

One of the stand out features is having the script prompt the user for input, which I think could be extremely powerful when using the test as a data generator.  One of the ways that I've found to encourage Selenium adoption on a new project is to set up the tests so that they automate the most common tasks, and help developers and the BA/QA team generate test data to aide their manual testing.  With Selenium that required that we put some extra components into the iframe, to let the user select values that would be used within the script. I've also used selenium in the past as a way to automate certain personal tasks like filling out online surveys, or reconfiguring my stupid motorola router after a power failure, and I could see the prompting feature of iMacros making those scripts easier to create.

iMacros also lets you feed a script with a .csv file, and have it loop over and repeat for each row, which can be a good way to seed some test data in an app, while putting it under a bit of load. (Not sure how to best have two iMacros scripts going against one server like I used to with Selenium, but I bet its possible)

Iopus, the company that makes iMacros, is actively encouraging a community to share common scripts, and includes several good examples of their own, one which fires off searches against multiple sites, after prompting you for a search term.  (which I plan to create my own to blast searches against my favorite online classified sites like craigslist, oodle,  googlebase, kijiji,  listpic, and ebay)

Overall, I would say compared to Selenium, iMacros is:

• Slightly less intuitive to me in terms of syntax, assertions, and scripting capabilities
• Capable of working in Firefox and InternetExplorer
• Loaded with great examples, documentation and video tutorials
• Similar to the Selenium IDE interface for recording, editing, and  rerunning tests
• Designed with user interaction in mind (easy to record, prompting for user input, csv import)
• Taking web scripting to the next level with powerful  data extraction capabilities

Stuff I didn't look into yet

• Integrating the tests into the Continuous Integration build with either Cruise or Hudson
• How to use the Data extraction capabilities
• Invoking Javascript from the test
• Having the script take  a screenshot on failure
• Invoking the test script from  Java  or Ruby
• Drag & Drop Ajax
• Stepping through the debugger
• Firing off multiple sessions

No matter which tool you choose, having some level of automated functional and UI tests is critical. It saves everyone on the team time on menial and repetitive tasks, can be a great way for new people to learn the app, provides a suite of smoke tests you can run throughout the development cycle, and it can help you free up your QA resources to focus on higher value goals.

And what about you?

1. How do you test flash/flex/siverlight/java Applets in your apps?
2. Is there anything that Selenium can't do for you? (Other than Flash/Flex, Selenium has always been able to do what I need)
3. What parts of your UI testing have you been able to automate? (And what's your next goal?)

What do you think you could do with iMacros? Could you use it to help your Business team be more efficient? (Post a challenge, we'll see if we can tackle it with iMacros or Selenium)

Related posts:

1. Ajax Testing: Doubling Down with Selenium and JMeter
2. Skinning Flex with Illustrator CS3
3. Flash/Flex physics engines and examples
4. Coming in GWT 1.5: Selenium testing support
5. Bandwidth profiling Flex projects and more with Charles

Comments (3)

Software Development

Comments: 3 so far

• Leave a comment
• Comment feed for this post
• TrackBack URL

1. 

   A quick search through our internal mailing list archives brought up these links; I hope they’re useful.

   http://iancartwright.com/blog/2009/02/acceptance-testing-silverlight-with.html

   http://code.google.com/p/silverlight-selenium

   Comment by Sidu, Wednesday, April 8, 2009 @ 2:11 pm

2. 

   Good work, John. I was wondering how you’d figure out how to handle user-based input for Good/Bad automation in Selenium. DevHooks make a lot of sense- wish we had that when you were teaching me how to use it.

   iMacros looks like another fine tool and I hope to get a crack at it soon.

   Comment by Scot Witt, Thursday, April 9, 2009 @ 11:56 am

3. 

   Hey John, so were you successful in getting Selenium or IMacro working against a silverlight app? We have a web app built using silverlight. I tried some basic recording with Selenium and it doesn’t recognize the elements. I started to research and found some sites that say they made it work. I tried some of their code snippets but it didn’t work. So that’s when you get into whether you need “coders” to do your automation.
   Has anyone written any sample code that can be used? Or maybe a tutorial specific to silverlight?

   Comment by Greg Walters, Monday, April 13, 2009 @ 9:01 am

Leave a comment