Again – AJAX and Screen Scraping | Pathfinder Development | Software Developers

Again – AJAX and Screen Scraping

Dietrich Kappe, Wednesday, May 31, 2006 @ 9:00 am

My review on Monday of the Sprajax AJAX security scanner left me thinking about the limitations of the old school crawlers and spiders. In order to spider an AJAX application you would need to comprehensively model the behavior of a browser. That's more than just embedding Rhino or Spidermonkey into your scraper. You'd pretty much have to embed the whole browser and use it to identify all valid events that an XHTML document will accept at a given point.

The complex application state of something like an AJAX word processor could make it difficult to know when to stop spidering, i.e. when has our URL graph cycled back on itself.

There is at least one open source tool that allows one to at least record and run tests in the Firefox browser: Selenium. This terrific post by Grig Gheorghiu off of his Agile Testing blog explains some of the subtleties of using Selenium with AJAX. The tool may be a good place to start to build AJAX capable spiders and crawlers.

Topics: Ajax Tools, Google, Search

Comments (2)

Software Development

Comments: 2 so far

You can also check SWExplorerAutomation SWEA (http:\\webiussoft.com). SWEA fully supports the AJAX Web applications.

Comment by Alex, Wednesday, January 31, 2007 @ 11:41 am
Thanks guys that was extremely helpful!

sandeep verma
(http://sandeepverma.wordpress.com)

Comment by sandeep verma, Monday, May 25, 2009 @ 11:10 pm

« Next Post Blog Home Previous Post »

Subscribe via email

Subscribe via RSS

Software Development

Product Strategy

User Experience Design

Technologies and Platforms:

    Rich Internet Applications

    iPhone/Mobile

    Ruby on Rails

    Flex, Flash and Air

    .Net

    Java

    Business Rules Engines

Pathfinder News

.NET .NET Browser Control .Net Development 2d cad 2d physics 3d 3D GPS 3d mapping 3D physics 37signals Acceptance Tests Accessibility ActionMailer actionscript activerecord acts_as_ferret Add new tag Adium ADO.NET Entity Framework Adobe adobe acrobat Adobe AIR adobe flex Adobe Illustrator Advertising aggregation agile Agile Development agile iteration agile thinking AIR Ajax Ajax Applications Ajax Bookmarking Ajax Components Ajax Development Ajax Examples Ajax Experience Ajax Frameworks Ajax history management Ajax Intervention Ajax libraries Ajax library AJAX Obfuscation Ajax Performance Ajax Products Ajax toolkit Ajax Tools Ajax Widgets A list apart Amazon Amazon CDN Amazon Web Services amf Analysis Android animation annotation Announcement Announcements antennae Antipatterns Apache Apollo App Engine SDK apple apple tablet Application Architecture Application Development ap store architecture art AS3 ask a UI guy ASP.NET ASP.NET assembla asterisk Asynchronous Processing authorization awards axiis Azure Back Button backups bandwidth bandwidth profiling Barnes & Noble Barriers to Entry Beans beer Benchmarking Best Practices BitmapData.draw BJAX blackberry Blaze Advisor blender blog blogging blog widgets book review Books Borders braindump browser Browsers Bugs Business Business Concepts Business Reasons for Ajax Business Rules Engines C# c# documentation ndoc sandcastle caching campfire Canvas capistrano career Case Studies CFO change management Charles charting checklist chess Chesspresso Chicago chicagoruby chirb windycityrails Chrome Chrome OS CI CIO Claude Shannon Closure Tools Cloud Computing CloudFront CMS COBOL Cocoa code code art Code Generation code generator Code Style Color COMET Competitive Strategy Compilers Compressor Computer Science Conference Confluence Consistency Content Management continuous integration converget appliances core animation CRM cruise CruiseControl CSS cucumber Custom Application Development Custom Flex Component dashboard Data Mapper data services integration data visualization date datetime Davlik Degrafa deployment deprec Design design documentation Design Patterns design thinking Desktop desktop application development Desktop RIA Desktop Software developer Developer's Notebook development DHTML Diagnose Divide and Conquer documentation Dojo Domain Knowledge don norman Drawing Drools drupal dynamic languages ease of use EC2 Echo2 Echo3 Editorial elections enterprise iphone development Entrepreneur erb ERP error handling Estimating estimation Ethnographic Research events everyblock Evolution Excel externalinterface Ext JS Extreme Programming eye tracking Facebook factory failure Fake Agile Feedback Loop ferret FileReference Firebug Firefox Firefox Extensions fixed bid fixture replacement fixturereplacement fixtures Flare Flash Flash and Air flash awards Flash Catalyst flash player flash player 10 Flash Player optimization Flash Remoting Flex Flex, Flash and Air Flex3 flex 3 flex code generator flex css flex development flexmock Flex optimization flex skins flexunit flex widgets Flickr Flock Flow Fluent forms Frameworks FriendFeed front end front end development fulltext search functional functional specs Games Gang of Four Gauge Component GDI+ gem getting things done ghost GigaOm GIS applications Git github gitignore Golf Google Google Analytics Google Analytics for Flash Google Analytics for Flex google android Google App Engine Google calendar Google Closure Tools google docs Google Gadgets Google Gears google maps Google Web Toolkit Google X Prize GORM government g phone Grails Graphics graphing Greasemonkey Griffon Griffon Plugin Development Groovy GStreamer GTD Gwittir GWT GWT 2.0 GWT Layouts GWT Rich Internet Apps h.264 haml hardware Healthcare heuristic evaluation Hibernate high-performance teams hosting HTML Hudson Hype IBM IDE Ideation IE IE6 IE7 IE8 IE detection iGoogle Illustrator illustrator cs3 ILog ILOG JRules imacros importing graphics to flex Information Architecture information visualization infrastructure Innovation Instructional Design Interaction Design interaction patterns design Internship Interview iPad ipad development iPhone iPhone Development iphone framework iPhone SDK iPod ipod touch irb iteration IT M IT Mill Toolkit iTunes jakob nielson Java javafx Javascript JavaScript frameworks Javascript Libraries Java Web Start JBoss Rules Jess Jetty JIT jmeter jnlp job Jobs jQuery JSF JSON JSP JSR-94 JsUnit JVM JWS Kanban keynote Kindle kiosk laptop Lazlo Lean Legacy Systems leopard lightweight LinkedIn LINQ LINUX log4j logging Logical Model and Conceptual Model Low Pro Mac Maintainability Malware Manufacturing mapping marketing Marketplace Mash Note Mashups math measurement Meebo metal metaprogramming MetaWidget Methodology Microformats microsite Microsoft midVentures25 migrations minimalism Mobile mobile platform Mocha Mock mocking mock objects modeling mod_rails monitoring Mootools mortgage calculation mouse mouse scroll mouse wheel Mozilla Music MVC MySql natural key neal ford Netbook NetNewsWire networking NewNet news newspapers new york times nfjs NHibernate nokia notebook NSURLProtocol obj-c Object-Oriented Objective-C Object Relation Mapping (ORM) ocmock Office Om Malik OmniGraffle online spreadsheets OOAD OOP opengl Open Screen OpenSocial Open Source opensource Opera Oracle OReilly origami ORM os osx OS X Packer pagination pairing Pair Programming palm papervision3d Pathfinder Development Pathfinder Events Pathfinder General Patterns pdf pdf application development Peer Creation Performance Personas PGN Photoshop PHP Phusion Passenger physics physics engines planning play at work plugin plugins portableapps pragmatic prawn Predictions preloader presentation primary key privacy process Web/Tech Product Definition Product Design production environments Production Support productive programer productivity product launch product management product manager Product Strategy Progressive Enhancement project concept Project Manage project management Project Website Prototype Prototyping PureMVC Pure MVC purpose purpose built devices PV3D pyro QA qooxdoo Radiant CMS rails railscasts rails controllers rails development Rails Environment Tests rails gem rails polymorphic railsrx rails testing randori Really Simple History Refactor refactoring References regex regular expressions reporting Requirements Requirements Alice Toth requirements definition Requirements Visualization resesign Restlet RETE Review rfp ria Rich Interactions rich internet applicaiton rich internet applications risk management Robert Fogel ROI rspec ruby rubyamf Ruby on Rails Ruby on Rails Internship Ruby on Rails testing role ruport S3 SaaS Safari San Francisco Scalability Scenarios Scriptaculous Scrum SDLC Search Secretariat Security Selenium SeleniumIDE Semantic web SEO Server Side service oriented architecture shoulda Silverlight simplicity Sketch skins SOA soapUI Social Networking Software software arhcitecture software design software develoment Software Development Best Practices Software Engineering Software Maintenance Software Processes software product strategy Songbird SpiderMonkey Sprajax Spreadsheets Spring Framework StageScaleMode Standards standish starting projects Startups static typing Stencils Steve Jobs STI storytelling Story Telling strategy Stress Structured Design Struts Subclassing sun surrogate key Susan Boyle Swing tablet tabs tag taglib Tamarin Tank Engine Task Flows Task Switching tdd teams teamwork Technical Debt telephony Tellurium Test test::unit Test Driven Development Testing tether textmate The Ajax Experience throttling Tilt Component time timezone Tools touch touch screen touch screen kiosk TraceMonkey Training Trends Tumblr Tutorial Tutorials Twitter ubuntu UI UIViewController uml unit testing Unit Tests unity3d Upgrade Usability Usability Testing user centric design user driven agile user experience design User Experience Design user groups user interface User Interface Standards User Modeling User Research uxd Vaadin value velocity Venture Capital Video viral marketing Vision visual analytics visual design visual documentation Visualization VLC VML vmware Volta Waaler Surfaces wapcaplet waterfall watij watir web Web/Tech Web 2.0 web app Web Applications Web Design Web Development web forms web hosting Web Infrastructure Webkit Weblogs Web Services WebSockets Web Standards WebTest Widgets will_paginate Window Forms Development Windows windows development windows mobile WinForms Wireframes WordPress workflow work life balance xcode XML XML Metadata xp XUL Yahoo Map AS3 API YUI Zeigarnik Zeigarnik Effect ZendAMF ZK

Custom Software Developers since 1998. Hire us to build complex software that's easy to use.