The Web is filled with nuggets of information, some of it updated on an daily, hourly or even minute-by-minute basis. A Java applet's graphing capabilities and its ability to sit on a page and display continuously changing data make it an ideal viewport for these sorts of pages. Rather than forcing your viewers to reload to see new data, the applet can stay running and continuously pull in the required data. The best part is that you do not need direct access to the headline, weather or stock feed data: Java can access HTML pages directly, so any source of data that is available on a Web page is in theory accessible to your applets.
Unfortunately, using an applet as a dynamic filter for information is not easy. The Java security model for JDK 1.0.2 prevents applets from accessing URLs outside of the web server on which the applet itself is hosted. JDK 1.1 and the 4.0-level browsers let you sign applets and selectively permit network access, but unless you can control the browsers hitting your site (as on an intranet), you are out of luck if you want to directly access another site from your applet. Many designers solve this by writing a custom Java or C++ server that runs on the Web server and communicates with the applet over the network. The applet (legally) talks to this server, the server (which is free to go to any Web address) fetches and reads the Web page, and then hands it to the applet.
Such a design is very inefficient from both a resource and a programming standpoint, though. Your weather server, your server and your ABC news headline server each take up memory on your server, and each has to sit and listen on its own network port. Furthermore, they all basically do the same thing: they fetch a Web page, decipher the information on it in an application-specific manner, and then communicate this information to the applet. Object-oriented design encourages us to find such patterns and ask whether there's a generic way to solve the problem rather than constantly reinventing the wheel.
An "applet proxy server" is one possible solution to this problem. The idea of a proxy server comes from the world of firewalls. A proxy server sits outside of the firewall and fetches Web pages for Web browsers inside the firewall. The browsers talk to the proxy server, the proxy talks to the destination site, and it acts for (thus proxy) the browser on the Internet. An applet proxy server does the same thing if you think of the sandbox as a firewall. The proxy server runs on the Web server (which the applet can talk to) and fetches the pages, then hands the complete page to the applet to interpret how it pleases. Since the proxy server does not care what the applet does with the page, any applet can use it to get any page.
We start by defining a simple protocol that will allow the client to talk to the server. In network programming, the protocol is the language the client and the server use to communicate over the network. Examples of protocols include SMTP (sending mail), POP3 (fetching mail) and HTTP (the Web protocol). Most of these protocols have a similar format: the client sends single-line text commands indicating what they want, and the server replies with a code indicating error or success, and then maybe additional data. In our proxy protocol, there are only two commands: PROXY and QUIT. PROXY takes one argument, a URL to fetch, and returns the length of the document followed by the full text. A client can ask for as many documents as it wants, and then ends the session by sending the QUIT command. One good way to test a new network server is to telnet to the port on which it is running (here I am telnetting from a Solaris machine, jupiter, to an NT box named venus that is running the proxy server). These text-based, human-readable protocols let you act as if you are the applet client and see what the server does. Here's a sample session with the proxy server:
jupiter:kdowney: telnet venus 4005 Trying 172.16.20.5... Connected to venus. Escape character is '^]'. Applet HTTP Proxy Server 1.0, (C) WebConcepts, LLC, 1997 PROXY http://www.webconcepts.com +OK URL fetched; data to follow Content-length: 1314 ....more follows QUIT +OK logging out of proxy server Connection closed by foreign host.
The server responds with one of two codes: +OK or +ERR, with the latter indicating a mistake (each is followed by a description of the error or what has been successfully done, such as +OK logging out of proxy server or +ERR unknown directive if you type something other than QUIT or PROXY.
Now that we know what we want our generic server to do, it's just a matter of writing it. The full source for the server is available for download; we'll look at the salient features.
In the UNIX world, many multi-user network servers create additional copies of themselves to handle incoming requests from clients. If you look at the list of running programs on a Web server that runs Apache, you will see 10 copies of "httpd" running. Since Java does not have a way to create new processes, the multi-threading mechanism is used instead. Each thread handles a single client. When it is finished, it is returned to the thread pool to accept another. The default design can handle five "users" simultaneously, though it is really limited by the host server's memory and performance.
The proxy server is built on top of a generic multi-threaded server class. The name of the game, again, is creating code we can use elsewhere, and a networked server is a common enough design element that it doesn't hurt to do a little extra work. The multi-threaded server has a private Vector called threadPool that stores a collection of service threads, miniature runnable pieces of code that can be dispatched to handle clients. Since the class needs to work with any possible service thread and listen to any port, its primary constructor takes a thread class and a port as its arguments. This class must extend ServiceThread or the server will throw an exception when it starts.
You start the server by calling the init() method. This method creates threadPoolSize (the default number is 5) threads for handling clients and starts them running. Note, as mentioned above, the check to make sure the serviceThreadClass is an instance of ServiceThread.
Once each ServiceThread starts running , it loops continuously, attempting to accept() a connection from a client. When it does, it prints out a message on the console that it got a client, and then immediately passes the buck to the method service(). The HttpProxyServiceThread class simply defines service() to talk to the client according to the protocol we have already outlined. It loops, looking for "PROXY" commands and responds in kind. It cuts off the connection with the client by ending the service() call when it sees QUIT.
The meat of the proxy server is in bold below. It attempts to open the URL specified in the PROXY statement. If the connection is made properly, it copies the entire remote file into a buffer in memory. It tells the client the length of the buffer with Content-length: and then prints out the entire set of data. This is the middle tier that connects the applet to the outside world, and does the actual proxying.
public void service(InputStream in, OutputStream out) throws Exception { PrintStream os = new PrintStream(out); DataInputStream is = new DataInputStream(in); os.println("Applet HTTP Proxy Server 1.0, (C) WebConcepts, LLC, 1997"); while (true) { String commandLine = is.readLine(); if (commandLine.regionMatches(true, 0, "PROXY ", 0, 6)) { String proxyUrlString = commandLine.substring(6); System.out.println("proxying to " + proxyUrlString); // fetch the file specified in the URL and redirect it to the client URL proxyURL = new URL(proxyUrlString); InputStream urlStream = proxyURL.openStream(); // read the stream into buffer byte[] buffer = new byte[0]; byte[] chunk = new byte[4096]; int count; while ((count = urlStream.read(chunk)) >= 0) { byte [] t = new byte[buffer.length + count]; System.arraycopy(buffer, 0, t, 0, buffer.length); System.arraycopy(chunk, 0, t, buffer.length, count); buffer = t; } // write the buffer to the output stream os.println("+OK URL fetched; data to follow"); os.println("Content-length: " + buffer.length); os.write(buffer); } else if (commandLine.regionMatches(true, 0, "QUIT", 0, 4)) { os.println("+OK logging out of proxy server"); return; } else { os.println("+ERR unknown directive"); continue; } } }
On the client side, a small class, HttpProxyConnection, is used to handle communications with the server. Its primary method is fetch(), which does the actual network communication. The bold portion below handles the equivalent to service() on the client end. It sends the PROXY command, reads in the content-length, and then reads that many characters into a buffer.
public byte[] fetch(URL proxyURL) throws Exception { System.out.println("fetching " + proxyURL.toString() + " using proxy at " + proxyHost + ":" + proxyPort); Socket sock = new Socket(proxyHost, proxyPort); InputStream in = sock.getInputStream(); DataInputStream urlData = new DataInputStream(in); OutputStream out = sock.getOutputStream(); PrintStream cmdStream = new PrintStream(out); // fetch past the first line urlData.readLine(); // send a PROXY command cmdStream.println("PROXY " + proxyURL.toString()); // get the status String cmdResult = urlData.readLine(); System.out.println(cmdResult); if (cmdResult.regionMatches(0, "+OK", 0, 3)) { int contentLength = Integer.parseInt(urlData.readLine().substring(17)); byte[] dataBuffer = new byte[contentLength]; byte[] chunk = new byte[512]; int count; int bytesRead = 0; // fetch the whole piece of data into a byte array while (bytesRead <= contentLength) { count=urlData.read(chunk); bytesRead += count; byte[] t= new byte[dataBuffer.length + count]; System.arraycopy(dataBuffer, 0, t, 0, dataBuffer.length); System.arraycopy(chunk, 0, t, dataBuffer.length, count); dataBuffer= t; } // send a QUIT command cmdStream.println("QUIT"); sock.close(); return dataBuffer; } else { sock.close(); throw new Exception("proxy error: " + cmdResult); } }
Quotron is a tiny Java applet that relies on the Applet Proxy Server to contract the Yahoo stock quote service and get the value and change amount for a particular stock symbol. It can take a stock description and symbol as parameters in the <APPLET> tag, making it easy to configure. Yahoo provides a particularly simple text format (CSV, comma-separated values) that is meant for use in Excel, but also serves this process well. A URL like
http://quote.yahoo.com/d/quotes.csv?symbols=MSFT&format=sl1d1t1c1ohgv&ext=.csv
returns a screen with the contents:
"MSFT",138.5,"10/21/1997","4:01PM",+5.875,136.125,139.234,135.312,14460300
which, as you'll see below, you can parse very easily using the built-in text tools in Java to get out the stock fields you want (the second and the fifth in the case of Quotron).
The new generation of graphical user interface "painters" for Java make it relatively easy to create simple applets without knowing much about AWT, Java's GUI system. I created the interface for Quotron in Visual Café quickly, and then added the necessary code to call HttpProxyConnection to get the stock data. Since the whole point of using an applet is to take advantage of continuous updates (though not too often: with 20 minute delay on stock quotes, there is no point in polluting the network by checking the stock value every second), Quotron uses an independently-running thread that repeatedly calls getStockValue() in a loop, which is the only complication in the code.
The getStockValue() method fetches from the Yahoo URL (which has already been set up with the correct symbol in the CGI arguments) using the proxy connection. It should get back a line of comma-delimited data that includes the current stock value and its change. The rest of the method simply parses out this string to get the value and sets the text fields in the applet's user interface. It also adds a piece of logic to color-code positive or negative changes in the stock.
public String getStockValue() { InputStream stockData = null; try { HttpProxyConnection proxyConnection = new HttpProxyConnection(getCodeBase().getHost()); byte[] buffer = proxyConnection.fetch(stockServerUrl); // create a string representing the array String dataString = new String(buffer, 0); int currentValStart = dataString.indexOf(","); int currentValEnd = dataString.indexOf(",", currentValStart + 1); int valDeltaStart = dataString.indexOf(",", currentValEnd + 17); int valDeltaEnd = dataString.indexOf(",", valDeltaStart + 1); String deltaVal = dataString.substring(valDeltaStart + 1, valDeltaEnd); stockSymbolValue.setText(dataString.substring(currentValStart + 1, currentValEnd)); stockDelta.setText(deltaVal); if (Float.valueOf(deltaVal).floatValue() <0.0F) { stockDelta.setForeground(Color.red); } else { stockDelta.setForeground(Color.green); } } catch (Exception e) { System.out.println("error fetching stock!"); e.printStackTrace(); System.exit(1); } return null; }
If you have Java enabled in your browser, you can see Quotron on its demo page.
There was a reason Sun created the applet sandbox, and any mechanism that makes it easy to circumvent it should be scrutinized carefully for security holes. One possible extension of the server would be to add two protocol commands, USER and PASS, that would accept a username and password and compare them to a database of accepted application "usernames" and passwords. Only approved applets would be allowed to make a proxy request. To keep it simple, the server is also weak on error-checking: if there is a problem connecting to a URL or if a client crashes, the result is usually a dead thread. Too many such crashes and the proxy server goes down. One possible solution to this is running a "monitor thread" that keeps the number of live, running threads at a constant number.
On the applet side, the possibilities are limited by your imagination and your ability to decipher a complicated Web page for nuggets of data. Possible applets include a configurable weather station that gets and redisplays the weather from http://www.weather.com, a more powerful stock ticker that uses a textbox where the user can enter a stock symbol to track rather than a parameter in the HTML, and a headline applet like the ABC news headline ticker at http://home.netscape.com.
The downloadable package contains all of the source code, bytecode and documentation for the proxy server and quotron. It also includes a copy of this article for your reference.
The javadoc documentation tree for the MultiThreadServer and proxy classes are also browseable on-line.