Quantcast

Use Wrappers and Proxies for Basic Web Services Tracking

Get the WebProNews Newsletter:
[ Technology]

Some commercial Web services software provides sophisticated Web services accounting features, recording details of Web services transactions recognized on the wire. But sometimes developers need accounting that is more modular, much more basic, and available on a shoestring. This article explains how to use advanced function composition tasks to add basic Web services monitoring capabilities.

Technologies that predate mainstream Web services offer a dual nature in the way they fit into frameworks. The likes of ONC RPC, CORBA, and XML-RPC can be comprised either through the language-specific stubs that drive their common usage or through their wire formats. Web services consist of more than just RPC style, of course, but this style does dominate its mainstream use. What this means in practice is that developers create stubs for remote objects by writing compatible interfaces by hand, processing WSDL files, or even dynamic methods and introspection. But the services invocation then comes as a simple method call.

As these method calls get marshaled to wire formats and stream across the network, developers and administrators often want to keep track of things by maintaining detailed transaction logs in order to debug service behavior, profile performance, test security controls, and even model traffic shape for service-level improvement. A variety of vendors have lined up to provide tools to help with such monitoring, almost all of which focus on actual wire transactions. You can even get a box from some vendors that you pop into the rack in your data center, and plug into the network to gain such monitoring capabilities

Wire monitoring tools are generally expensive and require a lot of set-up and care. Considering that the idea behind Web services is that interchange can come from a diversity of end-points in a diversity of languages and platforms, wire monitoring is probably the most comprehensive approach in an enterprise environment. Then again, in reality most Web services in deployment are intra-departmental and even point-to-point. When Web services are used as an applications integration tool, such comprehensive monitoring is rarely needed. Even when such comprehensiveness is justified, it often makes sense to defer deployment of full-blown monitoring systems until production testing, which means that developers in earlier phases still need a means of keeping track of Web services for debugging and planning purposes.

It is also useful to have the ability to support Web services monitoring through the operation of the programming language stubs at the end points. The most general way of approaching this is by using techniques of function composition according to available language features. Most programmers think of such techniques as building wrappers and proxies.

Compose your methods

In functional and dynamic languages, function composition is usually straightforward at its root and flexible enough to come in numerous implementations. To demonstrate the basic technique, Listing 1 shows Python code that uses dynamic features to log all invocations of a remote method to standard error on the console.

Listing 1. Demonstration of function composition in Python for mixing logging capabilities into SOAP

import sys #Use standard system object of the interpreter
import SOAPpy #Use the third-party SOAP/WSDL toolkit SOAPpy

class logging_proxy:
&nbsp """
&nbsp Requires Python 2.2 or more recent for proper operation
&nbsp """
&nbsp def __init__(self, subject, stream):
&nbsp """
&nbsp&nbsp This is a documentation string, similar to Javadoc

&nbsp __init__ is the Class initializer (like a constructor)
&nbsp subject - the object to which we delegate
&nbsp stream - the file-like object for log output

&nbsp Special parameter:
&nbsp self - the instance upon which a method is invoked
&nbsp """
&nbsp self.subject = subject
&nbsp self.stream = stream
&nbsp return

&nbsp def __getattr__(self, attr):
&nbsp """
&nbsp Special method name that intercepts all requests for an
&nbsp attribute or method on an instance.
&nbsp attr - the name of the requested attribute/method
&nbsp """
&nbsp #getattr is a special global function which gets the named
&nbsp #attribute of an object.
&nbsp #Result is generally a data member or function object (method)
&nbsp obj = getattr(self.subject, attr)
&nbsp if callable(obj):
&nbsp&nbsp #Then it's a method, not a plain attribute
&nbsp&nbsp def log_wrapper(*args, **kwargs):
&nbsp&nbsp&nbsp #Write pre-call log
&nbsp&nbsp&nbsp self.stream.write("Invoking remote method %sn"%attr)
&nbsp&nbsp&nbsp self.stream.write("tpositional params: %sn"%repr(args))
&nbsp&nbsp&nbsp self.stream.write("tnamed params: %sn"%repr(kwargs))
&nbsp&nbsp&nbsp #Now invoke the service, passing on the parameters
&nbsp&nbsp&nbsp result = obj(*args, **kwargs)
&nbsp&nbsp&nbsp #Write post-call log
&nbsp&nbsp&nbsp self.stream.write("Method %s completedn"%attr)
&nbsp&nbsp&nbsp self.stream.write("tReturn value: %sn"%result)
&nbsp&nbsp&nbsp return result
&nbsp&nbsp return log_wrapper
&nbsp&nbsp else:
&nbsp&nbsp&nbsp return obj

WSDL_URI = "http://www.xmethods.net/sd/2001/TemperatureService.wsdl"

#Create the remote service proxy
#All the needed data comes from the WSDL
service = SOAPpy.WSDL.Proxy(WSDL_URI)
#wrap the service to the add logging feature (log to stderr)
wrapped_service = logging_proxy(service, sys.stderr)

#From here on all invocations of methods on the Web service
#Automatically go through the logging code first
zipcode = '80027'
temp = wrapped_service.getTemp(zipcode)
print 'Temperature in US postal code ', zipcode, 'is', temp, 'F'

The definition of the class logging_proxy need only occur once. You can then wrap just about any class with it to meld in the logging capability. If you’re not familiar with Python, this function might be a bit hard to follow. The heart of the technique is the definition of log_wrapper, which forms what is called a closure. For Java programmers this is reminiscent of inner classes. The closure captures the information about the actual Web service method stub to be invoked (the function object called obj) as well as the stream for the log output. The caller gets this closure when it invokes a SOAP method by name, which triggers the special method __getattr__. In usual cases of method invocation the return result is a function object, which is then executed. In this case the returned function object is a closure which encapsulates some logging code and the data needed to call the original function. The overall effect is a composition of the log_wrapper code and the actual remote method invocation. The rest of the code in the module puts the technique to use. Only one additional line is required besides the usual SOAP client code:

service = logging_proxy(service, sys.stderr)

This one-time set-up is required for each remote service. As you can see, the output stream is parameterized: You can send some logs to a file and others to a network socket. You can parameterize other details easily enough. Overall, this technique provides a huge amount of flexibility. The following snippet shows the output from Listing 1:

$ python listing1.py
Invoking remote method getTemp
&nbsp&nbsp&nbsp positional params: ('80027',)
&nbsp&nbsp&nbsp named params: {}
Method getTemp completed
&nbsp&nbsp&nbsp Return value: 21.0
Temperature in US postal code 80027 is 21.0 F

The last line represents the normal processing of the method result. The five previous lines are output from the logging code.

A proxy on your network

Another approach to monitoring is to listen at the network level. Simply set up a proxy server of your own that performs some monitoring action while forwarding traffic to and from each remote service end-point. The proxy server listens on the same protocol as the remote server, but when it gets a request it simply performs the necessary action, such as logging the request. It then forwards the request to the remote service, and passes any response back to the original client. Listing 2 is a program that operates as such a proxy for SOAP over HTTP. It’s also in Python, but the nature of this proxying approach is such that this code can be used with any Web service regardless of the language used to implement either SOAP end point.

Listing 2. Script that logs then forwards SOAP/HTTP messages

import sys
import BaseHTTPServer
import httplib

class ProxyHandler(BaseHTTPServer.BaseHTTPRequestHandler):
&nbsp def do_POST(self):
&nbsp&nbsp clen = int(self.headers.getheader('content-length'))

&nbsp #dest is the true SOAP end point address. It comes in as
&nbsp #a standard HTTP Host header, that is the host then the port
&nbsp #separated by a colon. Break this into the components
&nbsp dest = self.headers.getheader('Host')
&nbsp dest_host, dest_port = dest.split(':')

&nbsp #Read the HTTP payload
&nbsp request_body = self.rfile.read(clen)

&nbsp #Log the HTTP details to STDERR
&nbsp sys.stderr.write('Path:' + self.path + 'n')
&nbsp sys.stderr.write('Headers:n')
&nbsp for k, v in self.headers.items():
&nbsp&nbsp sys.stderr.write('t' + str(k) + ': ' + str(v) + 'n')
&nbsp sys.stderr.write('Payload:n')
&nbsp #input_body is the request SOAP envelope and contents
&nbsp sys.stderr.write(request_body)

&nbsp #Create a matching request to the actual destination
&nbsp requestor = httplib.HTTP(dest_host, dest_port)
&nbsp requestor.putrequest('POST', self.path)
&nbsp for k, v in self.headers.items():
&nbsp&nbsp requestor.putheader(k, v)
&nbsp requestor.endheaders()
&nbsp requestor.send(request_body)

&nbsp #Wait for the response from the remote host and forward it
&nbsp #as a response from this host
&nbsp (status_code, message, response_headers) = requestor.getreply()
&nbsp response_body = requestor.getfile().read()
&nbsp self.send_response(200, 'OK')
&nbsp for k, v in response_headers.items():
&nbsp&nbsp self.send_header(k, v)
&nbsp self.end_headers()
&nbsp self.wfile.write(response_body)
&nbsp return

LISTENING_PORT= 8888
#Set up to run on local machine
SERVER_ADDRESS = ('127.0.0.1', LISTENING_PORT)
httpd = BaseHTTPServer.HTTPServer(SERVER_ADDRESS, ProxyHandler)
print "Listening on port", LISTENING_PORT
#Go into a the main event loop
httpd.serve_forever()

The method do_POST on ProxyHandler is executed once the server receives SOAP by way of an HTTP POST invocation. It logs the HTTP headers and payload (that is, the SOAP message) and then makes an equivalent call to the actual remote end point. The HTTP response from the remote end point is forwarded back to the original client. This proxy code is very flexible because, if used correctly, it can determine the actual destination from the HTTP headers. In the code, this is read into the dest variable. This means that you can use this one SOAP/HTTP proxy to handle any number of requests for all variety of actual destination end points. The original clients see the same network traffic as if they were communicating to the remote server directly. Listing 3 is an example of client code that uses the SOAP/HTTP proxy.

Listing 3. SOAP client that proxies its messages for logging

import sys&nbsp&nbsp&nbsp #Use standard system object of the interpreter
import SOAPpy&nbsp&nbsp&nbsp #Use the third-party SOAP/WSDL toolkit SOAPpy

WSDL_URI = "http://www.xmethods.net/sd/2001/TemperatureService.wsdl"
#WSDL_URI = "http://localhost:8888/sd/2001/TemperatureService.wsdl"

#Create the remote service proxy
#All the needed data comes from the WSDL
service = SOAPpy.WSDL.Proxy(WSDL_URI)
service.soapproxy.http_proxy = 'localhost:8888'

zipcode = '80027'
temp = service.getTemp(zipcode)
print 'Temperature in US postal code ', zipcode, 'is', temp, 'F'

This is pretty much boilerplate remote procedure calling code except for the line:

service.soapproxy.http_proxy = 'localhost:8888'

Most SOAP libraries give you a mechanism for directing a SOAP request through a proxy. This line sets an HTTP proxy for the SOAP client, pointing it to port 8888 on the local machine. Because setting the HTTP Host header is a standard step in using an HTTP proxy, very little additional work is needed to get the proxy server to forward the traffic properly.

The main limitation with this HTTP proxy trick is that you have to add more customization to handle usage patterns that are more complex than the common request/response interaction. Luckily, such customization isn’t terribly difficult.

Wrap up

I have just scratched the surface of function composition and the capabilities of network service proxying. These themes have many variations and most languages support similar capabilities. If you’re using a functional language (such as Python), you can follow up on these compositional tricks by experimenting with currying and other such techniques. If you’re using a static object-oriented language such as Java, look into aspect-oriented programming (AOP). Programmers have lots of options for adding infrastructural capabilities to Web services without having to buy expensive wire monitoring tools.

Resources

  • Find out more about SOAPpy 0.11.1, which the author used here for the SOAP/WSDL client code. He covers this toolkit and others in his “Python Web services developer” column.
  • Intrigued by some of the techniques used in this piece? Then take a look at David Mertz’s excellent “Charming Python” column and items in the Python Cookbook mentioning composition and curry.
  • Aspect-oriented programming (AOP) is an important system for implementing similar techniques in static programming languages. If you’re a Java programmer, see “Improve modularity with aspect-oriented programming” (developerWorks, January 2002) by Nicholas Lesiecki.
  • Find more related resources on the developerWorks XML and Web services zones. You can also sign up for the weekly Web services/XML tips newsletter.
  • Browse a wide range of XML-related titles at the developerWorks Developer Bookstore.
  • Find out how you can become an IBM Certified Developer in XML and related technologies.


    First published by IBM developerWorks at http://www.ibm.com/developerWorks/.

    Uche Ogbuji is a consultant and co-founder of Fourthought Inc., a software vendor and consultancy specializing in XML solutions for enterprise knowledge management. Fourthought develops 4Suite, an open source platform for XML, RDF, and knowledge-management applications. Mr. Ogbuji is also a lead developer of the Versa RDF query language. He is a computer engineer and writer born in Nigeria, living and working in Boulder, Colorado, USA. You can contact Mr. Ogbuji at uche.ogbuji@fourthought.com.

    Use Wrappers and Proxies for Basic Web Services Tracking
    Comments Off
    About Uche Ogbuji
    Uche Ogbuji is a consultant and co-founder of Fourthought Inc., a software vendor and consultancy specializing in XML solutions for enterprise knowledge management. Fourthought develops 4Suite, an open source platform for XML, RDF, and knowledge-management applications. Mr. Ogbuji is also a lead developer of the Versa RDF query language. He is a computer engineer and writer born in Nigeria, living and working in Boulder, Colorado, USA. You can contact Mr. Ogbuji at uche.ogbuji@fourthought.com. WebProNews Writer
    Top Rated White Papers and Resources

    Comments are closed.

    • Join for Access to Our Exclusive Web Tools
    • Sidebar Top
    • Sidebar Middle
    • Sign Up For The Free Newsletter
    • Sidebar Bottom