- We design and build extraordinary applications for companies looking to make the next great idea a reality.
- learn more
Making GWT JSON not Quite so Painful
I've been using GWT to resurface interfaces of a variety of legacy applications -- J2EE, PHP, Rails -- and more often than not that means working with JSON returned from the server. One thing that I've found is that GWT's JSON support is kind of chatty. That is, you have to write a bunch of code like this:
JSONValue root = JSONParser.parse(json); JSONObject obj = root.isObject(); if (obj != null) { JSONValue map = obj.get("map"); if (map != null) { JSONArray arr = map.isArray(); if (arr != null) { if (arr.size() > 1) { JSONValue strval = arr.get(1); if (strval != null) { JSONString str = strval.isString(); if (str != null) { String s = str.stringValue(); // do something with the string } } } } } }
Yikes! Just to get the string in the second array item in the "map" key of the top level object. Truth is, most of the JSON I'm getting back is fairly predictable. And if it's not in that predictable form, it's an error that the client code can't recover from. I'd like to be able to write something like String result = obj.get("map").get(1).stringValue() without having to worry about NullPointerException's cropping up all over the place.
Checking in from AjaxWorld East
Just a quick update from the conference (which is now over). I gave my talk at the AjaxWorld East 2008 and got some interesting reactions. The talk really had three parts.
- What is GWT and how do you use it?
- How do I resurface an existing J2EE app?
- Where are all of the alligators?
There is always a risk that someone or other feels that too little or too much time is spent in one part or the other. In fact, while converting a JSP to produce JSON so it could be consumed by the GWT client app, I had one person raise their hand and as "what does this have to do with GWT."
So, in the end, I had half the people nodding their heads, yes, and the other half shaking their heads, no. I was peddling a very pragmatic approach, rather than a product or service, and that might have thrown some folks off.
Met a lot of cool folks, put some faces to names, got to sit on a panel with Douglas Crockford. Cool beans. More next week.
Topics: Ajax Development, GWT, JSON
Grails and JSONP: How Easy is That?
For all of those Java developers casting longing glances at their buddies doing Rails development, there is hope. Grails, which has just celebrated its 1.0 release, is a Rails-like "convention over configuration" framework that aims to do for the Java world what Rails has done for Ruby. Instead of Ruby, the dynamic programming language of choice is Groovy, which compiles to bytecode (no Groovy to Java translation) and integrates smoothly with just about any Java code you may be using.
I'll stop hyping Groovy and Grails in general here; there's plenty of good informational stuff on the Grails and Groovy home pages. I will note that if you run off the tracks a little bit, you can find yourself reading through 500-odd line stack traces of Groovy, Spring and Hibernate -- there's room for some improvement there.
One of the many nice things about Grails is it's support for JSON and XML. Let me put together a simple example that shows off some of Grails' power.
Topics: Ajax Examples, Application Development, Grails, Groovy, Java, JSON, XML
101 Ideas for JSONP - Idea #4: Scraping XML with XPath, Part 1
It's been a while since the last installment of JSONP. Lots of good stuff has happened in the meantime to make our lives easier, but some things are still brutish, ugly and complex. One of the nice things in the more recent version of the JDK is the ever improving support for XML processing. Long gone are the days when you had to hunt around for libraries and hope that your version of the JDK worked with your third party libs. There are still reasons, like performance and extended features, that would cause you to pick an alternate library, but if you just want to add a little bit of XML processing into your apps, things couldn't be easier.
New Ajax for Old Iron
In the rush to develop entirely new Web 2.0 systems based on Ajax, it is often easy to lose sight of how it can be used to improve the so-called "legacy" enterprise systems. Over at the Ext JS blog, there's a post about German developer who put together a reporting/BI application to front-end RPG code running on an AS400.
The backend serves up JSON data (together with this old article about reskinning a Spring MVC application with Tibco GI, it makes for a solid approach for refactoring existing webapps) to the Ext JS/Flash front-end. Slick. You can view a demo (unfortunately in German for you non-German speakers) here.
Technorati Tags: ajax, ext js, as400, flash, json
Topics: Ajax Applications, Ext JS, Flash, JSON
Pimp my Webapp: Turning Web 1.0 into Web 2.0, Part 1
Everyone wants the Web 2.0. But how to get there, especially if you have a dowdy old Web 1.0 application? Will I have to spend the annual GDP of a smallish Central American nation on the rewrite? There are three general approaches that evolved over the last year:
- The Christmas Tree Approach - decorate your Web 1.0 app with Ajax widgets and simple backend JSON or XML services. Pro: doesn't require major investment. Con: app still mostly suffers from the Web 1.0 downsides -- postbacks, CRUD, etc.
- The Refrosting the Cake Approach - transform all of your views from HTML to JSON or XML. Develop a client side GUI that calls your applications controllers and views. Pro: doesn't require full rewrite. Con: control flow moves from server side to client side; won't take full advantage of RIA opportunities, such as direct manipulation or document-centric interfaces; really just a nice face on CRUD.
- The From Scratch Approach - take the opportunity to rethink your application as something other than the web equivalent of a green screen. Imagine Photoshop or Word as a traditional webapp. Pro: unencumbered by the old system. Con: costly; you may lose your way in reinventing your app.
I think approach #2, Refrosting the Cake, is actually quite appealing. You can make using that dowdy old webapp quite a bit more pleasant with a quick coat of Ajax. There was a really interesting article on this written by Brian Walsh back in July of 2006. It sort of got lost as a pure Tibco GI article, but it has much wider applicability. In it, Brian uses XStream to convert the views of a SpringMVC sample application from HTML to XML. He then uses Tibco GI to build a UI that talks to the controllers and views on the back end.
I used a similar approach to convert a dowdy old application -- mwhois -- to Web 2.0. Instead of Tibco GI, I used GWT. And instead of XML, I used JSON.
The old application looked somewhat homely...
...and suffered from the kludgey flow and heavy postbacks typical of Web 1.0. I proceded to identify the different views and converted them (via mwhois' templates) into JSON.
simple search: domain=agiledev&ext=com
{ "isAvail" : false, "domain" : "agiledev", "ext" : "com", "whoisServer" : "whois.crsnic.net" }
raw record: domain=french&ext=biz&show_raw=1
{ "domain" : "french", "ext" : "biz", "raw" : "Domain Name: FRENCH.BIZ\nDomain ID: D2708502-BIZ\nSponsoring Registrar: COMMUNI GAL COMMUNICATIONS LTD.\nSponsoring Registrar IANA ID: 418\nDomain Status: clientTransferProhibited\nRegistrant ID: GC683CO965021\nRegistrant Name: jamil akhtar\nRegistrant Address1: Regent House\nRegistrant Address2: 24-25 Nutford Place\nRegistrant City: London\nRegistrant Postal Code: W1H 5YN\nRegistrant Country: Great Britain (UK)\nRegistrant Country Code: GB\nRegistrant Phone Number: +44.7729391052\nRegistrant Facsimile Number: +44.2075693152\nRegistrant Email: jamil@cityfinancialcorp.co.uk\nAdministrative Contact ID: GC683CO965021\nAdministrative Contact Name: jamil akhtar\nAdministrative Contact Address1: Regent House\nAdministrative Contact Address2: 24-25 Nutford Place\nAdministrative Contact City: London\nAdministrative Contact Postal Code: W1H 5YN\nAdministrative Contact Country: Great Britain (UK)\nAdministrative Contact Country Code: GB\nAdministrative Contact Phone Number: +44.7729391052\nAdministrative Contact Facsimile Number: +44.2075693152\nAdministrative Contact Email: jamil@cityfinancialcorp.co.uk\nBilling Contact ID: GC683CO965021\nBilling Contact Name: jamil akhtar\nBilling Contact Address1: Regent House\nBilling Contact Address2: 24-25 Nutford Place\nBilling Contact City: London\nBilling Contact Postal Code: W1H 5YN\nBilling Contact Country: Great Britain (UK)\nBilling Contact Country Code: GB\nBilling Contact Phone Number: +44.7729391052\nBilling Contact Facsimile Number: +44.2075693152\nBilling Contact Email: jamil@cityfinancialcorp.co.uk\nTechnical Contact ID: GC683CO965021\nTechnical Contact Name: jamil akhtar\nTechnical Contact Address1: Regent House\nTechnical Contact Address2: 24-25 Nutford Place\nTechnical Contact City: London\nTechnical Contact Postal Code: W1H 5YN\nTechnical Contact Country: Great Britain (UK)\nTechnical Contact Country Code: GB\nTechnical Contact Phone Number: +44.7729391052\nTechnical Contact Facsimile Number: +44.2075693152\nTechnical Contact Email: jamil@cityfinancialcorp.co.uk\nName Server: DNS.INTER.NET.IL\nName Server: NS.COMMUNIGAL.NET\nCreated by Registrar: COMMUNI GAL COMMUNICATIONS LTD.\nLast Updated by Registrar: COMMUNI GAL COMMUNICATIONS LTD.\nDomain Registration Date: Wed Mar 27 00:01:00 GMT 2002\nDomain Expiration Date: Wed Mar 26 23:59:59 GMT 2008\nDomain Last Updated Date: Mon Mar 12 16:25:34 GMT 2007\n\n>>>> Whois database was last updated on: Sun Oct 14 17:51:14 GMT 2007 <<<<\n\nNeuLevel, Inc., the Registry Operator for .BIZ, has collected this information\nfor the WHOIS database through an ICANN-Accredited Registrar. This information\nis provided to you for informational purposes only and is designed to assist\npersons in determining contents of a domain name registration record in the\nNeuLevel registry database. NeuLevel makes this information available to you\n\"as is\" and does not guarantee its accuracy. By submitting a WHOIS query, you\nagree that you will use this data only for lawful purposes and that, under no\ncircumstances will you use this data: (1) to allow, enable, or otherwise\nsupport the transmission of mass unsolicited, commercial advertising or\nsolicitations via direct mail, electronic mail, or by telephone; (2) in\ncontravention of any applicable data and privacy protection acts; or (3) to\nenable high volume, automated, electronic processes that apply to the registry\n(or its systems). Compilation, repackaging, dissemination, or other use of the\nWHOIS database in its entirety, or of a substantial portion thereof, is not\nallowed without NeuLevel's prior written permission. NeuLevel reserves the\nright to modify or change these conditions at any time without prior or\nsubsequent notification of any kind. By executing this query, in any manner\nwhatsoever, you agree to abide by these terms.\n\nNOTE: FAILURE TO LOCATE A RECORD IN THE WHOIS DATABASE IS NOT INDICATIVE\nOF THE AVAILABILITY OF A DOMAIN NAME.\n" }
global search: domain=agilesoftware&do_global=1
{ "domain" : "agilesoftware", "avail" : [{ "domain":"agilesoftware","ext":"biz" },{ "domain":"agilesoftware","ext":"be" }], "unavail" : [{ "domain":"agilesoftware","ext":"com" },{ "domain":"agilesoftware","ext":"net" },{ "domain":"agilesoftware","ext":"org" },{ "domain":"agilesoftware","ext":"co.uk" },{ "domain":"agilesoftware","ext":"info" }] }
wizard search: do_wizard=1&company=pathfinder&keyword1=uxd&keyword2=agile&ext=com
{ "whoisServer" : "whois.crsnic.net", "avail" : [{ "domain":"pathfinderuxd","ext":"com" },{ "domain":"pathfinder-uxd","ext":"com" },{ "domain":"uxdpathfinder","ext":"com" },{ "domain":"uxd-pathfinder","ext":"com" },{ "domain":"pathfinder-agile","ext":"com" },{ "domain":"agilepathfinder","ext":"com" },{ "domain":"agile-pathfinder","ext":"com" },{ "domain":"uxdagile","ext":"com" },{ "domain":"agileuxd","ext":"com" },{ "domain":"uxd-agile","ext":"com" },{ "domain":"agile-uxd","ext":"com" }], "unavail" : [{ "domain":"pathfinder","ext":"com" },{ "domain":"pathfinderagile","ext":"com" },{ "domain":"uxd","ext":"com" },{ "domain":"agile","ext":"com" }] }
Then I wrote a simple, tabbed interface using GWT and the MyGWT widgets and slapped the two together.
I'll have more details on the code details in part 2 of this article. For now, you can look at the old and new interfaces.
Technorati Tags: ajax, gwt, mygwt, tibco gi, JSON, XML, web20
Topics: Ajax Development, Best Practices, Design Patterns, Google, GWT, JSON, Tutorials
>API Keys for Ajax Services
I'm sure not everyone is developing Ajax-based front end sites and applications. There must be someone who is developing XML, JSON and JSONP web services, right? So, if you are going to offer it as a commercial service, and even if you aren't, you still want to be able to control and measure access to it. Something like those Google and Yahoo service API keys, right? I'm sure we've all worked with MD5 and SHA-1 enough to have some ideas about how that would be done, but it helps to have an example to follow.
Well, check out this article over at java.net: Creating and Using API Keys with Java Based Ajax Services. It demonstrates the client and server side pieces to using an API key. Lots of code, and shows how to generate the key from the client web site address and a secret service token. (Note: the example relies on the now outmoded MD5 to generate the token, but that could easily be switched to SHA-1 or some other one-way hash.)
Technorati Tags: ajax, json, jsonp, api key
Topics: Ajax Development, JSON, Security, Web Services
GWT with JSONP
I've been looking to add to my JSONP series with some examples using GWT. Dan Morrill over at Google has beaten me to the punch with an article on just this subject. The code for it is rather straightforward, but there are a few sticky bits that make it a little less so:
I spent quite a while debugging this, until I finally asked Scott Blum, a GWT Engineer. Scott merely asked a question: "Was the array created in the same window in which you're testing it?"
What Scott knew that I did not, is that the JavaScript classes (like Array) corresponding to the primitive types are constructed along with the window object. Because JavaScript is a prototype-oriented language, the "classes" are really just object instances with special names. These two issues combine to reveal a subtle but important issue: the Array objects from two different windows are not the same object! The expression "x instanceof y" in JavaScript boils down to something like this pseudocode: "if the 'prototype' property of 'x' is the same object as 'y', return true, else return false".
At this point, you may be wondering how multiple windows entered the discussion. The key is the
$wndvariable, which points to the hidden iframe in which the application's GWT code is loaded. The DOM object in GWT, however, points to the parent window'sdocumentobject; after all, your application code is interested in manipulating the browser window, not GWT's hidden iframe. As a result, in the code above, the<script>tag is added to the parent window, while the code using it resides in a different iframe. This means that the object is created in a window different from where the "instanceof" checks are made, thus causing the issue above.
The solution Dan chose was to add the <script> tag to the iframe that contains the GWT code.
Of course the new "cross site" compilation option produces code that loads directly into the main page, so this hack may not in fact be necessary.
The Problem with JSON
I've been tracking the whole insecure JSON debate over the past couple of weeks with some interest. I've been promoting the use of JSONP for a while, so I thought I'd provide a summary of all of the action and summarize what I myself plan to do.
Rob Yates originally proposed some best practices for the use of JSON in secure applications.
When developing a JSON api that contains data that should not be publically accessible to the world use Approach 1 i.e. return plain JSON.
Joe Walker then started the whole imbroglio with his article showing how to subvert JSON services that return arrays. In Joe's example you redefine the Javascript Array constructor, then call a JSON service using a script tag. He later blogged about how this could be done with objects, as well.
Right now these hacks only work on Firefox, and there are limits to the kind of JSON that can be eval'd outside of an assignment statement, but that is no reason to be complacent about this issue.
This guy tried his hand at defeating the above hack by grabbing a fresh copy of the Array object from an iframe, but in the end concluded that in an arms race between the hacker and the application developer on the client-side, the hacker will always win. As Joe Walker pointed out, "In the end you can't trust the environment that you are sending the data to, so you should only send data to people you trust, and that means having URLs that are not predictable."
A related topic (as pointed out in some of the posts above) is that of Cross-Site Request Forgery (CSRF).
Cross-site request forgery, also known as one click attack or session riding and abbreviated as CSRF (Sea-Surf) or XSRF, is a kind of malicious exploit of websites. Although this type of attack has similarities to cross-site scripting (XSS), cross-site scripting requires the attacker to inject unauthorized code into a website, while cross-site request forgery merely transmits unauthorized commands from a user the website trusts. Compared to XSS, CSRF attacks are not well understood by many web developers and few defense resources are available.
These two methods can be used in concert. For example, the JSON hijacking can be used to lift information such as account numbers, and then CSRF is used to construct a URL that causes some sort of malicious action in the application.
As for what I plan to do in my own secure web application development, it remains pretty consistent with what I did before the advent of Ajax:
- Use SSL, obviously.
- Don't rely on cookies for authentication and set short timeouts for user sessions. You can require a randomly generated chunk of the path of application url's for each unique session instead.
- Make sure that all of your URL's that modify data or have other side effects require POST. This doesn't eliminate risk, as an attacker can use Javascript to launch a form POST, but it does cut down on the opportunistic uses of image and other tags that perform a GET -- reducing the avenues of attack.
- Perform referrer checking. Again, this is no panacea, as an attacker can use flash or XHR to subvert this, but it raises the bar a little more and restricts the avenues of attack further.
- Use a request token that is dependent on the previous response (SHA-1, etc.) and includes a sequence number. If a request is made that is not valid, the user sessions should be immediately invalidated. (An attacker could theoretically compromise the user's machine so they could defeat even this approach, but at that point I'd have to throw up my hands.)
- Detect, log and notify on all suspicious activity.
- Use JSON only for publicly available information, i.e. if they can get it by hacking your browser, they can just as easily get it for themselves.
There are limits, however, to what can be done. As long as browsers have security holes and Javascript allows, well, what is allows, there will always be some new avenue of attack for hackers to exploit. (Let's not even get started on viruses and trojan horses on the client).
Further Reading:
You may want to read an article over at IBM on secure mashups which covers additional ground, such as the use of the fragment identifier (http://host/path#fragment). Also, some additional ideas around REST, HTTP Auth, and resetting the Javascript Array object to its original state through delete Array, can be found here.
Finally, the incomparable Bruce Schneier has a blog entry on a paper on JSON Hijacking. Read it and the comments.
101 Ideas for JSONP - Idea #3: Scraping HTML With TagSoup and XQuery
Now that we have two ways (here and here) to pull XML data into our Ajax apps via JSONP, it's time to talk about how we can get our hands on more interesting data. Our two techniques allow us to proxy services that already produce valid XML, but not everything is available in this form. Lots of groovy data is still locked away in tradition web applications.
You can see an example of this problem even over at the leader in JSONP web services, Yahoo!. While Yahoo! provides an excellent delayed stock quote service, one thing that is lacking is a corresponding ticker lookup service. They do however have a ticker lookup web app: http://finance.yahoo.com/lookup?t=S&m=US&s=IBM
Transforming this app into a service involves the usual calisthenics -- transform the crappy HTML into well formed XHTML and then extract and transform our desired data into valid XML. Now we could use XSLT to crack this nut, but every time I break out the XSLT I get flashbacks to my Recursive Function Theory course and think that I'm doing a proof. Truth be told, I've never been a big fan of functional programming; FLWR expressions just seem so much easier to write. Yes, there are times where performance dictates not using XQuery, but for this example I'm sticking with my preference.
Below is a query that will perform our desired extraction and transformation. (XQuery syntax is a little beyond the scope of this post, but for a quick tutorial see here.) Like all screen scraping, it is a little brittle. If Yahoo! decides to change the format or even the color of the header row in this return page, the thing breaks. There are other queries that will do the trick and maybe be a little less brittle, but you should build some sort of regression testing and notification into these sorts of screen scrapers for precisesly these reasons. When it breaks, you know about it and you can fix it quickly.
<entities>{ for $t in //*:table[*:tr[@bgcolor="dcdcdc"]] for $r at $position in $t/*:tr[count(child::*:td)>1] where $position != 1 return <entity> <symbol>{data($r/*:td[1])}</symbol> <name>{data($r/*:td[2])}</name> <market>{data($r/*:td[3])}</market> <industry>{data($r/*:td[4])}</industry> </entity> } </entities>
To apply this to the Yahoo! ticker symbol lookup, we use a combination of the Nux XML processing toolkit and the TagSoup parser. TagSoup is the weapon of choice for converting gnarly HTML to well formed XHTML, and Nux gives us the XQuery machinery we need. Here is the code snippet for scraping the ticker symbol information from the results page:
private String applyScript(String uri) { InputStream in = null; GetMethod get = new GetMethod(uri); get.setFollowRedirects(true); try { httpClient.executeMethod(get); in = get.getResponseBodyAsStream(); } catch (IOException e) { log.error("Problem getting uri feed " + uri, e); return JSONIdeasConstants.ERROR_XML; } try { XMLReader parser = new org.ccil.cowan.tagsoup.Parser(); // tagsoup parser Document doc = new Builder(parser).build(in); Nodes results = XQueryUtil.xquery(doc, xqueryScript); if (results.size() < 1) { return JSONIdeasConstants.ERROR_XML; } return results.get(0).toXML(); } catch (ValidityException e) { log.error("Problem getting uri feed " + uri, e); return JSONIdeasConstants.ERROR_XML; } catch (IOException e) { log.error("Problem getting uri feed " + uri, e); return JSONIdeasConstants.ERROR_XML; } catch (ParsingException e) { log.error("Problem getting uri feed " + uri, e); return JSONIdeasConstants.ERROR_XML; } }
Here xqueryScript is a String containing the above XQuery script text. The ERROR_XML is just a constant string, "<Error></Error>", that we send back to the caller in case of an error. (You may want to send back a more informative error message to the browser. We'll tackle that in a later post.) This code will end up transforming the following app output...
...into the following xml
<entities> <entity> <symbol>IBM</symbol> <name>INTL BUSINESS MACH</name> <market>NYSE</market> <industry>Diversified Computer Systems</industry> </entity> <entity> <symbol>HZK</symbol> <name>CORTS TR VI IBM DEB</name> <market>NYSE</market> <industry>N/A</industry> </entity> <entity> <symbol>HZD</symbol> <name>6.40% CORPORATE-BACK</name> <market>NYSE</market> <industry>N/A</industry> </entity> <entity> <symbol>KVM</symbol> <name>STR PD 7.0 CORTS IBM</name> <market>NYSE</market> <industry>N/A</industry> </entity> <entity> <symbol>GJI</symbol> <name>SYNTHETIC FXD INC</name> <market>NYSE</market> <industry>N/A</industry> </entity> </entities>
Submitting the query is just the same old thing we've done before, just grab the input field contents and pass it to our servlet via an insert script operation:
JSONPIdeas.lookupSymbol = function() { var query = $("#lookup").val(); JSONPIdeas.addScript("http://labs.pathf.com/JSONIdeas/TickerLookup?callback=JSONPIdeas.renderSymbols&query=" + query); }
Give the example below a try. You can see the Javascript source code involved in this here. Note that if there is only one entity, the JSON lib that converts the XML into JSON doesn't return an array, so we check for that.
---
I hope I've inspired you to write some of your own JSONP web services. Next time I'll tackle some of the security issues that JSON and JSONP raise.
101 Ideas for JSONP - Idea #2: Wrap XML in JSONP
Sometimes you have some Javascript code, either something you've already written or a third-party library (like this XSLT library) that wants and expects XML. You don't want to rewrite the whole thing, so it would be nice to be able to adapt these without doing a whole lot of work. Once you realize that you can process an XML document from a string, not just a file loaded using XHR...
JSONPIdeas.parseXML = function(xmltext) {
var doc;
// code for IE
if (window.ActiveXObject)
{
doc=new ActiveXObject("Microsoft.XMLDOM");
doc.async="false";
doc.loadXML(xmltext);
}
// code for Mozilla, Firefox, Opera, etc.
else
{
var parser=new DOMParser();
doc=parser.parseFromString(xmltext,"text/xml");
}
return doc;
}
...the solution is obvious -- just wrap the XML as a string inside a JSON object.
The approach to doing this is even simpler than for our previous example. We simply proxy the web service and escape double quotes, backslashes, newlines and carriage returns and wrap it in a JSON object:
public String getJSONWrap(String uri) {
String xmlResult = getXML(uri);
// escape the double quotes and backslashes
String replace1 = xmlResult.replaceAll("\\\\", "\\\\\\\\");
replace1 = replace1.replaceAll("\\n", "\\\\n");
replace1 = replace1.replaceAll("\\r", "\\\\r");
String replaced = replace1.replaceAll("\"", "\\\\\"");
// wrap it in a JSON object
return "{\"Result\":\"" + replaced + "\"}";
}
I've wrapped this web service -- http://www.trynt.com/ip-data-api/v1/ -- that returns ARIN information. The box below should show you your hostname and the country you are connecting from. The Javascript code that makes it go can be found here. (Note that some of the ARIN data isn't always valid XML and the service above doesn't actually deal with it, so occasionally you'll have XML parse errors, but that would be the case whether or not you used JSONP or not.)
The first two ideas are really the basics of everything else we will do using JSONP. Next time, a little fun with XQuery.
Topics: JSON
101 Ideas for JSONP - Idea #1: RSS to JSONP
I've been working with lots of Ajax projects that make use of JSONP. JSONP is basically wrapping JSON in a callback script. So a return such as the following
callback({"IBM":"82.53"});
allows you to include the call as a script tag, for example
var script = document.createElement("script");
script.src = 'http://remoteserver/stockquote?symbol=IBM&callback=renderQuotes';
script.type = 'text/javascript';
document.body.appendChild(script);
will dynamically add a script to our page which will call back an already existing function renderQuotes in our page. This technique is very useful if you are including Ajax widgets on remote sites or using bookmarklets or GreaseMonkey scripts to pull information from remote servers.
Now you can use some ready made JSONP capable web services to build your applications and mashups -- certainly Yahoo has quite a few -- but that won't get you all of the types of information that you might want. Proxying other sites and sources of information and turning the results into JSON is a good way of getting the content you need.
I'm starting a little series called "101 Ideas for JSONP," where I'll give 101 different techniques, sources and ideas for getting information into a JSONP service.
Idea #1: RSS to JSONP
RSS is an XML format for content syndication widely used by just about everybody. The JSON library from json.org is a handy little utility library that allows you to convert JSON to and from XML and Java Objects. Combine the two and you've got an easy way of providing a JSONP service.
I've got an example of this right here that takes the RSS feed for New York City resumes from Craig's List and transforms it into JSONP.
resumes
The above widget updates every 30 seconds (you can look at the Javascript file that populates it from the proxy server here). The code we use to get the RSS and transform it into JSONP is pretty basic. The one thing to note in the code below is that the org.json.XML class doesn't allow for streaming. That means we have to read the whole RSS feed into memory in order to convert it. That's really not an issue in most cases with RSS feeds, but it's a gap that I'll try to remedy at some point in the near future.
package com.pathf.demo.jsonideas;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.json.XML;
import org.json.JSONException;
import java.io.IOException;
public class RSSGetter {
private HttpClient httpClient;
private Log log = LogFactory.getLog(RSSGetter.class);
public RSSGetter() {
httpClient = new HttpClient(new MultiThreadedHttpConnectionManager());
}
public String getRSS(String uri) {
// we get it as a string, since the JSON lib doesn't handle streams
GetMethod get = new GetMethod(uri);
get.setFollowRedirects(true);
try {
httpClient.executeMethod(get);
return get.getResponseBodyAsString();
} catch (IOException e) {
log.error("Problem getting rss feed " + uri, e);
return "Problem getting RSS feed ";
}
}
public String getJSONRSS(String uri) {
try {
return XML.toJSONObject(getRSS(uri)).toString();
} catch (JSONException e) {
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
return "{\"Result\":\"Error\"}";
}
}
}
The servlet below just wraps the result of the RSS retrieval in the callback function that is passed as a parameter. If no callback parameter is passed, it reverts to plain old JSON.
package com.pathf.demo.jsonideas;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.ServletException;
import javax.servlet.ServletConfig;
import java.io.IOException;
public class NYCResumeServlet extends HttpServlet {
private RSSGetter rss = null;
private static final String NYC_RESUMES = "http://newyork.craigslist.org/res/index.rss";
protected void doGet(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) throws ServletException, IOException {
httpServletResponse.setContentType("text/javascript");
String callback = httpServletRequest.getParameter("callback");
String json = rss.getJSONRSS(NYC_RESUMES);
if (callback == null) {
httpServletResponse.getOutputStream().println(json);
} else {
httpServletResponse.getOutputStream().println(callback + "(" + json + ");");
}
}
public void init(ServletConfig servletConfig) throws ServletException {
super.init(servletConfig);
rss = new RSSGetter();
}
}
There are other ways of pulling in an RSS feed via JSONP, some of which will allow us to use the browser's machinery for operating on XML to transform and add content via remote widgets.
Topics: JSON
About Pathfinder
Recent
- Making GWT JSON not Quite so Painful
- IDEA - preconference workshop 06 Oct 08
- HTML5, Ajax history management, and The Ajax Experience 2008 Boston
- A Look Back At Past Posts
- Flash Player on iPhone gossip
- Microsoft to Jump on Board EC2
- TAE Boston 2008: The Unsexy Presentations
- The Ajax Experience 2008: Hope to see you in Beantown
- TankEngine: New plugin for Rails iPhone Development
- Symphony of Ruby on Rails and Flex through RubyAMF
Archives
- October 2008
- September 2008
- August 2008
- July 2008
- June 2008
- May 2008
- April 2008
- March 2008
- February 2008
- January 2008
- December 2007
- November 2007
- October 2007
- September 2007
- August 2007
- July 2007
- June 2007
- May 2007
- April 2007
- March 2007
- February 2007
- January 2007
- December 2006
- November 2006
- October 2006
- September 2006
- August 2006
- July 2006
- June 2006
- May 2006
- April 2006
- March 2006
Topics
.NET
2d physics
3d
3D GPS
3D physics
37signals
Accessibility
actionscript
activerecord
Add new tag
Adium
ADO.NET Entity Framework
Adobe
Adobe AIR
Advertising
agile
Agile Development
AIR
Ajax
Ajax Applications
Ajax Bookmarking
Ajax Components
Ajax Development
Ajax Examples
Ajax Experience
Ajax Frameworks
Ajax history management
Ajax Intervention
Ajax libraries
AJAX Obfuscation
Ajax Performance
Ajax Products
Ajax Tools
Ajax Widgets
Amazon
amf
Analysis
Android
Announcement
Announcements
antennae
Apollo
Application Architecture
Application Development
AS3
ASP.NET
Asynchronous Processing
awards
Back Button
Benchmarking
Best Practices
BitmapData.draw
BJAX
Blaze Advisor
blog
blogging
Books
Browsers
Business
Business Reasons for Ajax
Business Rules
C#
Canvas
Case Studies
chess
Chicago
Cloud Computing
CMS
COBOL
code art
Code Generation
Color
COMET
Conference
Consistency
Content Management
CRM
CruiseControl
CSS
Custom Flex Component
data visualization
Degrafa
Design
Design Patterns
Desktop
Desktop RIA
Developer's Notebook
Diagnose
Dojo
Domain Knowledge
Drools
EC2
Echo2
Echo3
Editorial
ERP
Ethnographic Research
events
externalinterface
Ext JS
Facebook
ferret
FileReference
Firefox
Firefox Extensions
Flash
flash awards
flash player
flash player 10
Flex
flexunit
Flock
Flow
Frameworks
front end
front end development
Games
Gauge Component
getting things done
Git
Google
Google calendar
Google Gears
Grails
Graphics
Greasemonkey
Groovy
GStreamer
GTD
Gwittir
GWT
Healthcare
Hibernate
Hudson
IDE
Ideation
IE
IE6
IE7
IE8
ILOG JRules
Information Architecture
Innovation
Instructional Design
Interaction Design
Interview
iPhone
iTunes
Java
Javascript
JavaScript frameworks
Javascript Libraries
JBoss Rules
Jess
Jetty
JIT
Jobs
jQuery
JSF
JSON
JSR-94
JsUnit
Lazlo
Legacy Systems
lightweight
LinkedIn
LINQ
Logical Model and Conceptual Model
Low Pro
Mac
Mash Note
Mashups
Meebo
MetaWidget
Methodology
Microformats
Microsoft
Mobile
Mootools
mouse scroll
mouse wheel
Mozilla
Music
MVC
MySql
NetNewsWire
Object-Oriented
Object Relation Mapping (ORM)
Office
OOP
Open Screen
Open Source
Opera
Oracle
ORM
osx
pagination
Pair Programming
papervision3d
Patterns
Peer Creation
Performance
Personas
PHP
physics
physics engines
plugin
preloader
process
Web/Tech
Product Definition
productivity
Progressive Enhancement
Project Website
Prototype
Prototyping
PV3D
QA
qooxdoo
Radiant CMS
rails
Really Simple History
References
Requirements
Requirements
Alice Toth
Requirements Visualization
Restlet
RETE
Review
Rich Interactions
ruby
rubyamf
Ruby on Rails
SaaS
Safari
San Francisco
Scalability
Scenarios
Scriptaculous
SDLC
Search
Security
Selenium
Semantic web
SEO
Server Side
Silverlight
SOA
Social Networking
Software Processes
Songbird
SpiderMonkey
Sprajax
Spreadsheets
Standards
Startups
STI
Story Telling
Struts
Tamarin
Task Flows
Test Driven Development
Testing
The Ajax Experience
Tilt Component
Tools
TraceMonkey
Training
Trends
Tumblr
Tutorial
Tutorials
Unit Tests
Usability
Usability Testing
User Experience
user experience design
user interface
User Interface Standards
User Research
UXD
Venture Capital
Video
Vision
Visualization
VLC
Volta
Web/Tech
Web 2.0
Web Design
Web Development
Webkit
Weblogs
Web Services
Web Standards
Widgets
will_paginate
Windows
Wireframes
WordPress
workflow
work life balance
XML
XML Metadata