A reloading CherryPy server with Jython

One thing that didn't work in my last attempt at running CherryPy on Jython was the autoreload feature of the development server. There were gaps in the current Jython implementation that I didn't feel motivated to start looking into.

So, I thought of an aternative solution and with a simple example I got it working very smoothly.

I decided to use modjy again. And for a lightweight embeddable servlet container I picked Jetty (5). Modjy has a "reload_on_mod" property that can be set to true if you want the wsgi application that is beeing served checked for modification each time it is called. This only works with one file right now, but could easily be extended.

Firs I unpacked modjy's modjy_webapp directory to a modjy directory and then I wrote this simple script to fire up modjy in Jetty:

from org.mortbay.jetty import Server

server = Server()
server.addListener(":8080")
server.addWebApplication("", "./modjy/")
server.start()

Modjy just worked this time without any hassle :-) So the next thing was to write my CherryPy wsgi application and configre modjy to serve it for me. Very advanced example once again ;-)

import cherrypy
class Root:
    def index(self, name="World"):
        return "Hello ", name
    index.exposed = True

handler = cherrypy.tree.mount(Root())
cherrypy.engine.start(blocking=False)

In modjy's web.xml I had to chage the "app_filenam" to the name of my application and make sure that "python.home" was set correctly and the "reload_on_mod" was set to 1.

A simple change like switching the order of "Hello" and "name" made modjy reload the application and use the modified version. Reload time for this example was next to none as expected.

I might give it a shot at getting modjy to scan a whole directory and its subdirectoies for changes and see how that works out. Oh and this example would probably run on Jython 2.2 with only a couple of minor changes to the CherryPy code... I'll check that and report back.

I've also had some success in using the following setup with Jython: Velocity templates -> CherryPy -> (Spring) -> JPA. I will probably post something about that later on.

Django... wow!

My CherryPy + Jython work has been on hold for a while again. Mostly because I've been a bit put down by the fact that we will have to wait for Jython 2.3 (or 2.5 or whatever it will be called) before we can have generators enabled by default. That and a few other issues (a lot to do at work is one of them) has made me less enthusiastic.

So last week I decided to have a look at Django. I've been meaning to do this for a while, but somehow it has seemed less approachable than other Python frameworks like TutboGears and Pylons. Once I started playing with it I instantly knew that that was a illusion. Django was extremely simple to get started with and after getting started it just revealed more and more excellent features... It impressed me enought to make me start planning a small project I'd like to implement using Django (more on that later maybe).

In short, the first things that really appealed to me was the way URL's were dispatched to the controllers (called views in Django) and a mechanism called generic views (elliminates the need to write some code that follows commonly used scenarios...).

My first impression of Django... WOW!!!

Hopefully I will have time to post some Django examples soon.

CherryPy 3 + Jython - Third attempt

OK, so it's been a while since I had time and energy to give this another shot. But Leo Soto's work on Django and Jython gave me some inspiration the other day. So I pulled down the latest svn version of Jython (since 2.2 is released now, trunk is the former 2.3 branch). Since the last time I tried this Alan Kennedy has supplied some excellent improvements to Jython's socket module, so I was hoping most of my problems would be solved.

I started off with a clean Jython and clean CherryPy (3.0.2). I realised right away that the metaclass problem mentioned in the previous post still existed, so I made the same fix as before. I also noticed that Jython is still missing a signal module, so I dummied one up for my purposes. Afters some trial and error I added three things to the Jython socket module: the _fileobject class (just copied it from CPython) and the constants AF_UNSPEC and AI_PASSIVE.

Yay! It finally worked. To my joy the simple Hello World example ran without any problems :-)

import cherrypy

class HelloWorld:
    def index(self):
        return "Hello world!"
    index.exposed = True

cherrypy.quickstart(HelloWorld())

The next step is of course to try out some more advanced CherryPy moves... more to come.

CherryPy 3 and Jython - second attempt

Well, the issue with metaclasses was due to the simple fact that I didn't have the latest version of the Jython 2.3 branch pulled out. So yesterday I was able to make some more progress.

Once I got the basic metaclass stuff working I noticed an interesting difference between CPython and Jython when it comes to metaclass handling. CherryPy does something like this (although not exactly like this):

def __init__(cls, name, bases, dct):
  # ...snip
  dctnames = dct.keys()
  if name.endswith("__doc"):
    if hasattr(cls, name):
      delattr(cls, name)
                
  val = dct[name]
  # ...snip

In CPython this works fine, but in Jython this results in a KeyError. The delattr call actually removes the attribute from the same dct instance. If this is a bug in Jython or just an implementation detail where CherryPy relies on the CPython way I'm not sure...

I temporarily solved this by moving the assinment to val before the delattr call.

Apart from this there were some stuff missing from the socket module in Jython that I had to add (more on this later - possibly a jython-patch).

In the end I didn't succeed in getting the CherryPy server working this time either... I ran into a similar problem that I described in an earlier post (Port not bound), but was able to ignore that and still get some requests sent to the server. Unfortunately all I got in return to the browser was 'unpacking sequence too long' (but I did get the favicon shown in the browser ;-)). More to come...

CherryPy 3 and Jython - first attempt

I took a first stab at getting CherryPy 3 working with Jython yesterday. As expected I ran into some issues. First I noticed the signal module was missing in Jython. This shouldn't be crucial to CP's functionality, so I just dummied a signal module with a simple "do nothing" signal function in it. This got me a little bit further, but then I stubled upon a problem I couldn't solve that easily. CP uses Metaclasses for some of its types to generate docstrings for attributes. As it turns out the Jython 2.3 code (from svn) I was using didn't really play well with metaclasses. The 2.2beta1 version that is out handles metaclasses (decently), but there is something missing in the 2.3 branch... I'll have to look into exactly what. Hopefully it can be added soon. Before that I won't do any more CP3 - Jython work...

I've been glancing a bit at Django these last few days as well. My first impression when I looked at it a few months ago was that it was inferior to turbogears - but that was from a very quick first look. Now I'm really starting to think it looks great... more to come of that...

Improvised AOP with Jython

I'm currently working on a JEE application that exposes Web services to among others a Swing client. The web services can be configured to return serialized java objects within the SOAP responses... I won't get into detail about the why? here, it's a long story :-)

I started using jython to test these interfaces the other day and ended up with a sort of simple AOP solution.

The client uses Spring to instantiate a web service proxy for a specified service interface. Every method of the interface returns an instance of SerializedResponse, which has a getSerializedResponse method that returns a byte array that the client needs to deserialize.

Since I wanted to do this under the covers I created a very simple python class (with a really ambitious name :-)):

class aop_proxy:
  def __init__(self, target, interceptors=[]):
    self.target = target
    self.interceptors = interceptors

  # replaces the method requested with an 
  # interceptor chain
  def __getattr__(self, name):
    real_method = getattr(self.target, name)
    # the real method should be called last...
    func = real_method
    for ic in self.interceptors:
      # all 'interceptors' must be functions returning 
      # a function
      func = ic(func)
    def interceptor_wrapper(*args, **kwargs):
      return func(*args, **kwargs)
    return interceptor_wrapper

This is a proxy that replaces each method of the target object with a chain of interceptors "around" that actual method. The only interceptor I needed in my case was one that took the result of the original method call and passed it through a deserialize method of a SerializationUtil java class (verbose for visibility here):

def deserialization_interceptor(method):
  def intercept(*args, **kwargs):
    res = method(*args, **kwargs)
    serres = res..getSerializedResponse()
    return SerializationUtil.deserialize(res)
  return intercept

An interceptor in this aop_proxy solution consists of a function that takes a bound method as its only argument and returns a function that takes any number of arguments (and hopefully calls the supplied method somewhere - or the chain is broken). Not very elegant, but very very simple.

This is certainly nothing new in the Python world, but combining this with existing java code with jython really works excellent.

This has given me a really useful tool in my day to day development :-)

CherryPy + Jython + modjy + JBoss = true

There has been some talk on the jython-dev list about setting compelling goals for the continued development of Jython. One that many people agree upon is to get one of the "top-notch" Python frameworks running on Jython. Django was mentioned, but TurboGears came out as the most likely (first) candidate - mostly due to the fact that it would be easier to develop Jython support for it stepwise because it is basically a meta-framework (I said "basically"...). The obvious first step in getting TG running on Jython would be to get CherryPy running. This has been done before according to some posts on the jython-dev list, so it shouldn't be a big issue... or?

One of the main reasons I started helping out with Jython development is to one day see applications consisting of (for instance) a TurboGears web-layer talking to a core business-layer (and persistence-layer) written in Java supported with the excellent features of one of the many great JEE servers out there. So apart from getting CherryPy running on Jython I will aim at getting CherryPy running on JBoss!

OK, let's dive in:

>>> import cherrypy
Traceback (innermost last):
File "", line 1, in ?
File "d:\Henrik\jython-dev\2.3\CPythonLib\site-packages\
cherrypy\__init__.py",
line 10, in ?
File "d:\Henrik\jython-dev\2.3\CPythonLib\site-packages\
cherrypy\config.py", l
ine 8, in ?
ImportError: cannot import name cptools

Ugh... not good.

After some digging around I found that it was an import far down in the cptools.py file that tried to import 'compiler'... which doesn't seem to be lifted over to the Lib dir by the jython build script (have to check why?). I commented out a few lines that didn't look like they would effect my simple Hello World attempt (after first just trying to lift over compiler myself), crossed my fingers and tried again. Now I stumbled upon an error I saw mentioned on the jython-dev list earlier, cherrypy uses a tuple as the key to a __dict__ in one place (at least). Jython represents the __dict__ object as a PyStringMap - you do the math. So I just modified the cherrypy file with "key = str(key)" and voila the import of cherrypy was successful :-) I will see what I can do to get this working in Jython instead of changing the cherrypy code...

But after that I ran into trouble with the cherrypy reloader when it tried to do an 'os.spawnve' - not implemented in Jython (note to self: os module needs to be ported). Disabling auto-reload in cherrypy is as simple as setting it to 'production-mode'. Now I felt I was getting close... only to be faced with the cherrypy server's complaints that 'socket' doesn't have an AF_UNSPEC attribute, so I just add it (with the value 0 as in CPyhon). I really thought this was it, but after thinking for a while cherrypy told me 'NotReady: Port not bound.' but I did see log traces of my access attempts... bah... There seems to be an issue with the threading and socket code here...

One of my goals was as I mentioned to get cherrypy running in JBoss, so I decided not to bother anymore with the cherrypy server for now (although I/we will have to get it running eventually). So instead I wanted to try and feed a piece of cherrypy to modjy.

So what is modjy? "Modjy is an implementation of a WSGI compliant gateway/server for jython, built on J2EE servlets." For those unfamiliar with WSGI it stands for Web Server Gateway Interface and is described in PEP333. CherryPy is WSGI compliant and should be able to play nicely with modjy. Modjy is based on Jython 2.1, but as it turns out it has no issue working with 2.3 either.

I had a bit of trouble getting modjy to run in JBoss. A PyException was thrown when the PythonInterperter was beeing initialized, but all I could see in the output was
"Unable to import modjy_servlet: do you maybe need to set the 'modjy_jar.location' parameter?" So I included the modjy stuff directly in my war structure and added a little better error handling (read: printStackTrace ;-)). As expected it turns out that Jython can't find the modjy.py files. So I dumped them in [python_home]/Lib as a first crude test - worked fine (of course). Oh and in the process of doing this I also noticed that my python_home was set wrong and when I set it correctly and tried running everything from the modjy.jar file it worked too. To my defence it was 06:10 in the morning. So now modjy was running in JBoss... cool. What next? Well, before throwing CherryPy in the mix I wanted to just try creating my own little Hello World wsgi app. And to not push my luck I just wrote a simple handler function (modjy looks for a callable with the name handler by default). Here's my first wsgi app (and probably my last):

def handler(environ, start_response):
 writer = start_response("200 OK",
     [ ('content-type', 'text/html') ])
 writer("Hi there!")
 writer("These are the wsgi environ properties
     that are available to you:")

 for key in environ.keys():
     writer("Key = %s" % key)

That worked like a charm. So now it was time to introduce CherryPy to JBoss through their mutual friends modjy and jython ;-) Here's my first crack at a CherryPy application suitable for the task:

import cherrypy

class Hello:
 def index(self):
     return "Hello from CherryPy and JBoss"

 index.exposed = True

cherrypy.config.update({'global':
 {'server.environment':'production'}})
handler = cherrypy.tree.mount(Hello())

Well, that didn't work at all... I was a bit fooled by the CherryPy 3 docs here. I was running CherryPy 2.2.1. In CherryPy 3 'mount' returns a callable application object that is wsgi compliant, but in 2.2 I had to do this instead:

from cherrypy._cpwsgi import wsgiApp
...
cherrypy.tree.mount(Hello())
handler = wsgiApp

Not very complicated, but not very intuitive I have to admit... Well anyway... that didn't work either :-) Now I got two errors. I'll come back to the second one in a bit. The first one was:

"NotReady: The CherryPy server has stopped."

Stopped? I hadn't even started it... ah that might just be the problem. In CherryPy 2.2.1 I need to add this to my script:

cherrypy.server.start(initOnly=True,serverClass=None)

This means that I'm starting the CherryPy "engine", not the server itself. Again, this is something that is a lot clearer in version 3.

So, on to the next error:

...\WEB-INF\lib\modjy.jar\modjy_response.py", line 63, in __call__
BadArgument: Start response callback headers must contain a list of (string,string) tuples
19:45:22,953 INFO [[/cherry]] error: Start_response callable was never called

After looking closer at the callback headers mentioned, they sure looked like tuples of string, string to me :-( So I had to dig even deeper to find that modjy was trying to set a 'num_writes' attribute on a writer object that was None. I added a None check and gave it another try.

Still no luck and now I'm pretty sure I ran into a bug in modjy (another one that is). There were a few references to a start_response_callable.write_callable in modjy_impl.py. The error I got complained that this attribute didn't exist and when I looked for it I couldn't find it either... But after looking around in the code I came to the conclusion that it was in fact start_response_callable.writer that was intended, so I changed those references and fired up JBoss again and FINALLY I saw the sweet "Hello from CherryPy and JBoss" in my browser window :-)

This was my final CherryPy "application":

import cherrypy
from cherrypy._cpwsgi import wsgiApp

class Hello:
def index(self):
   return "Hello from CherryPy and JBoss"

index.exposed = True

cherrypy.config.update({'global':
 {'server.environment':'production'}})
cherrypy.tree.mount(Hello())
cherrypy.server.start(initOnly=True,serverClass=None)
handler = wsgiApp

This was really trivial stuff, but I'm really enthusiastic about continuing this work. First I will try to use a little more CherryPy features and after that I will try to get Kid working too. I will also add some simple support for using Jetty as an embedded server to make it easy to fire up a server in a python script pretty much the same way as you use the cherrypy server.