Liking the File language in Camel 1.5.0

Claus has a great write up on the new File language feature in Camel 1.5.0 - cool stuff! I'm really looking forward to the next release!

A favorite feature of mine in the next Apache Camel is a new feature to use file name patterns in the file component. What we have introduced is a new language, the file language, so you can leverage this feature anywhere in Camel where it uses Expressions, Languages etc. The file language currently support both the file and the FTP component as well.

As a lot of integrations is file based you need to be able to read (consume) or write (produce) files. Then you need to be able to specify file names. What's new in Camel 1.5.0 is that you can express this using patterns, for instance directly in the configuration of your endpoints. Prior you had to do this in Java code.

Simple example
Lets image you need to move consumed files into a backup folder after processing.

Now you can express this using the pattern: backup/${file:name}
Well then you need to use .bak as extension, and the pattern is:backup/${file:name}.bak

Okay next issue is that the backup folder should be grouped by dates using the yyyyMMdd pattern. Well the expression support thejava.text.SimpleDateFormat patterns also.
The pattern is:backup/${date:now:yyyyMMdd}/${file:name}.bak wherenow means current date. You can substitute this with different commands such as file for using the timestamp on the file instead.

Advanced example
In this sample you need to produce files (save) and you need to save the file using a generate unique filename. We use the divide and conquer pattern for this so you create a POJO class that generates the unique filename, then you as the developer have the full power how to do this. And then its very easy to unit test as its plain POJO that can be tested very easily with JUnit. Next step is to express this as a file pattern. As your POJO is a bean we use the bean langauge to invoke your pojo.

So the pattern is: myfile-${bean:myguidgenerator.generateid}.txt
Where myguidgenerator is the bean id of your POJO and generateidis the method name. You can omit the method name if there is no ambiguity which method Camel should invoke.

Camel routes
This feature is configurable directly in camel routing on your endpoints. So what we can do now is:

from("file://inbox?expression=backup/${date:now:yyyyMMdd}/${file:name}.bak").to("bean:processFile");

The route will consume files and process the files in the processFile bean, that is a plain POJO class. After processing the files is moved (renamed) using the pattern in the expression. So the file is moved to the backup subfolder, for example: backup/20081014/report-october-2008.bak

More to come
Well there is a ton of new features and improvements in Camel 1.5.0. Check out the current release note.

Using Guice as a JNDI provider

In many ways a Dependency Injection framework is a replacement for looking things up in a registry like JNDI; but sometimes having some kind of registry or JNDI provider can be useful such as if you want to work with JSR250/EJB3 or some legacy code which expects JNDI to be used - or you want to use JNDI as a kinda loosely coupled registry between completely different modules of your system.

I've hacked up a simple JNDI provider which just uses Guice to create all the objects; so you stick to pure Java code for writing your Guice Modules to depenency inject your objects together then have a little properties file to enable/disable/tweak things.

When looking at Guice through spring-tinted glasses the most glaring difference (apart from the lack of XML) tends to be the absence of the ApplicationContext and a registry where you can look things up by name. This JNDI provider provides a more standard alternative to the ApplicationContext, as JNDI is in the JDK and it kinda gives you that crutch to lean on until you figure out the guicier way of doing your dependency injection :). 

One neat thing is it combines JNDI naming with the @Named injection points in Guice and allows @Resource from JSR250/EJB3 all configured via the same jndi.properties file 

Adding support for @PostConstruct, @PreDestroy and @Resource to Guice

I do like guice as a Dependency Injection engine; I think it has a lot of promise; particularly if 2.0 is released soon :)

I still struggle sometimes, coming from a spring framework mindset, figuring out how to do some stuff. Though I love the @Provides support in trunk of Guice. Here's hoping 2.0 is out soon! :)

I've experimented recently adding support for ConstructorInterceptors into guice so you can add custom lifecycles like @PostContruct from JSR250/EJB3 or Spring's InitializingBean etc. 

I've also created a patch to support closing of singletons such as via @PreDestroy or spring's DisposableBean.

Slightly more wacky is a patch adding support for custom injection point annotations which allows you to easily support things like 

• @Resource from JSR 250/EJB3
• @PersistenceContext from JPA
  
• the various JAX-RS annotations like @Context, @PathParam etc
• WebBeans @In
• Spring's @Autowire
• Apache Camel's injection annotations like @EndpointInject or @Produce

Feedback greatly appreciated!

Enterprise Integration Designer preview

From Oisin's blog.

Thanks to Enterprise Integration Patterns, architects have a clear and succinct way to describe integration within complex systems. It certainly beats a lot of MS-Word documents and monster integration architecture diagrams. And, with Apache Camel, we’ve got an open source runtime that allows us to create routes by chaining EIPs together, using Spring XML or Java fluent builders. That’s fantastic, but EIPs are as much about sharing pictures as creating message routing graphs, and where are the pictures?  

We’ve made some tools that will allow you to load up Camel Spring XML files into Eclipse for visual inspection and editing. You can also create EIP diagrams in your Workbench and save them as Camel configurations. Finally, we’ve put in some debugging capabilities to allow you trace through the paths a message will take through your EIP graph.  

We are earnestly seeking feedback - check out the FUSE Integration Designer Preview page to download the Eclipse plugins, or check out the video links under the Training Videos at the bottom of the page.

Looks great! Feedback greatly appreciated

Using Camel with ServiceMix Kernel

Gert has a great post on ServiceMix Kernel and Camel showing how easy it is to take ServiceMix Kernel, install Camel into it and then deploy a route using a simple Spring XML file. Nice job Gert!

Apache ServiceMix Kernel 1.0.0 released!

via Guillaume's post

Apache ServiceMix Kernel 1.0.0 has just been released.

Apache ServiceMix Kernel is a small OSGi based runtime which provides
a lightweight container onto which various bundles can be deployed.
Amongst the list of supported features, Apache ServiceMix Kernel supports:

• hot deployment of OSGi bundles, exploded bundles or custom artifacts (spring xml configuration files support is provided)
  
• services configuration stored as property files are monitored and provided as standard OSGi configurations
  
• a centralized logging back end supported by Log4J, ServiceMix Kernel supports a number of different APIs (JDK 1.4, JCL, SLF4J, Avalon, Tomcat, OSGi)
• provisioning of libraries or applications can be done using simple commands via simple xml descriptors
  
• native OS integration as a service so that the lifecycle will be bound to your Operating System.
  
• an extensible shell console to manage services, applications and libraries
• operations on the console can be done remotely via a secured and encrypted channel
  
• a security framework based on JAAS
• new instances can be created using a single command line

This release, with the detailed release notes, is available at:
http://servicemix.apache.org/kernel/servicemix-kernel-100.html.

I'm particularly pleased to get the 1.0.0 release out; it makes it really easy to have hot-redeployable Camel routing rules - just edit your Spring XML when using the expanded bundle mode and ServiceMix will auto-redeploy your routing rules dynamically!

Whats new in ServiceMix 4 and Camel

Rod Biresch has posted a nice blog post about what's new in ServiceMix 4.x which is definitely worth a read! 

Also Claus Ibsen has added to the collection of tutorials on Camel, this time on using Axis with Camel. Nice work guys!

better stack traces in Java with log4j - including the jar file and version number!

When problems occur in open source software users tend to get stack traces. They often fire them off to some open source mailing list or forum asking for help; often without providing much in the way of detail. One of the first questions asked by support teams is often 'OK, what version are you using'.

Given that its possible to figure out the actual version being used of each class - either using the package information from the MANIFEST or by finding the jar the class came from, it seems logical to include the version information into a stack trace at the end of the line.

One day the JVM might actually do this for us :)

In the meantime, I've created a patch for log4j to add this feature; it appends an optional String to stack traces printed via log4j which includes the jar file name if it can be deduced plus the Java Package version the line of code comes from.

Here's an example...

org.apache.log4j.config.PropertySetterException: Hello
at org.apache.log4j.spi.ThrowableInformationTest.testStackTracePackageName(ThrowableInformationTest.java:306)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [1.5.0]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [1.5.0]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [1.5.0]
at java.lang.reflect.Method.invoke(Method.java:585) [1.5.0]
at junit.framework.TestCase.runTest(TestCase.java:154) [junit-3.8.1.jar]
at junit.framework.TestCase.runBare(TestCase.java:127) [junit-3.8.1.jar]
at junit.framework.TestResult$1.protect(TestResult.java:106) [junit-3.8.1.jar]
at junit.framework.TestResult.runProtected(TestResult.java:124) [junit-3.8.1.jar]
at junit.framework.TestResult.run(TestResult.java:109) [junit-3.8.1.jar]
at junit.framework.TestCase.run(TestCase.java:118) [junit-3.8.1.jar]
at junit.textui.TestRunner.doRun(TestRunner.java:116) [junit-3.8.1.jar]
at com.intellij.rt.execution.junit.IdeaTestRunner.doRun(IdeaTestRunner.java:94) [idea_rt.jar]
at junit.textui.TestRunner.doRun(TestRunner.java:109) [junit-3.8.1.jar]
at com.intellij.rt.execution.junit.IdeaTestRunner.startRunnerWithArgs(IdeaTestRunner.java:22) [idea_rt.jar]
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:118) [idea_rt.jar]
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:40) [idea_rt.jar]

So you can see what version of junit is being used (despite there being no manifest information) along with the JDK implementation version.
e.g.

at java.lang.reflect.Method.invoke(Method.java:585) [1.5.0]

means we found the package number but couldn't find the jar file but we found the implementation version of java.lang.reflect to be 1.5.0. Whereas

at junit.framework.TestCase.run(TestCase.java:118) [junit-3.8.1.jar]

means we found the jar file but the jar file has no manifest version information.

Am hoping one day this gets into some log4j release! You can find the patch here if you're interested.

am loving Beyond REST and the PIMP protocol

I really enjoyed reading Joshua Schachter's post beyond rest today - and particularly the comments. The basic idea is how to get a kinda publish/subscribe system to work on the web for processing real time updates to things like social sites at internet scale without introducing some really complex new protocol; but reusing the lovely RESTful web endpoints.

I posted a comment in the thread but figured it was quite big so figured I'd post it here too :)

I'm really liking the PIMP protocol and like Sam's strawman of using caching headers to implement it...

My suggestion is to think of this instead as another form of caching. All we really want is a header that tells the server that we are interested when a particular resource has been updated and how to tell us. The server can then either understand that header and acknowledge in the response that it will notify me. Here is my strawman:Request:
...
X-Cache-Callback:http://www.javarants.com/notify/joshua.schachter.org/atom.xml;SECRET
Response:
...
X-Cache-Callback: OK

Then if that resource is updated the service is expected to either HEAD the callback as a notification or POST the new contents of the resource, servers choice. You could later add semantics for merely updating the resource vs replacing it wholesale. I would also think about adding the ability for the server to specify a timeout after which you are free to poll again if you haven't heard anything on the assumption that sometimes the service may lose the state associated with your subscription.

Am thinking rather than returning OK the server returns the amount of time before the client has to re-issue the subscription to keep it alive. So the server can decide the maximum subscription time. Good PIMP servers (PIMPS :) might wanna make this quite long to reduce the polling overhead.

I also love the simplicity of the HEAD or POST to differentiate a notification of change to a notification-with-the-data.

I've long wanted a 'SUBSCRIBE' verb in HTTP for doing this kinda thing; but I think your cache-header approach is cleaner - as folks can either keep polling and/or subscribe for the update notification.

The nice thing too is that it allows easy migration to PIMP without introducing any overhead or new traffic; that clients continue to poll as normal - but they advertise themselves as being PIMP aware. Then eventually when one day the server becomes PIMP aware the clients receive their notifications (and then hopefully they scale back their polling :) - otherwise they can stick to polling.

Also webmasters can monitor their traffic looking for PIMP headers to know when it'll make sense to upgrade to PIMP. Not everyone is gonna need PIMP and it'll be a no brainer from looking at your logs to determine both when you've sufficient mass of PIMP enabled pollers along with knowing what the reduction in polling traffic upgrading to PIMP would save you.

I'm with Sam in the thinking of this as another form of caching. In implementing PIMP some folks might be able to create update notifications internally in their system when resources change to push out change messages into some kinda queue for posting to the callback URL. This would involve significant work for many sites though.

However it'll surely be pretty trivial to just install a PIMP-enabled caching web proxy inside your data centre in front of your servers - that does the usual cache thing, but also detects these extra PIMP cache headers and does a background poll of resources (respecting your existing cache & time to live headers) to detect changes both to update the cluster of front web caches (so non-PIMP pollers get more real time data) but also to drive the pushing of updates out to PIMP subscribers.

i.e. I can see this as a pretty easy upgrade to most web sites - folks just update their front end web proxies to a PIMP-enabled version and hey presto you can now support PIMP consumers. Am sure the web proxies could include an XMPP firehose too pretty easily for heavy hitters.

It should be pretty easy to hack the web proxies to do this I'd have thought? Even the problem thats been noted earlier in this thread - of trying to push updates to a URL endpoint might be slow, unresponsive or unavailable - the web proxies have to deal with already right in case a *local* server is borked.

Anyway - its a very interesting blog post and particularly the comments. Interesting stuff.

Running Nexus with launchd on OS X

I've finally got Nexus running for maven builds to help boost the speed downloading and checking repos etc.

This gave me the excuse to finally take a quick look at using launchd which is the preferred OS X way to run daemons and services. First look it appears cool and much nicer than rc/init.d/xinit.d et al.

Here's a really simple always-run nexus script, sonatype.nexus.plist. Just drop it into your ~/Library/LaunchAgents/ directory and you should be good to go.

From the command line you can then do this if you want to boot it up straight away - but it should restart on reboot (which I'll test out next time I've gotta reboot :)

launchctl start sonatype.nexus

I did try get clever and wait for port 8081 to be used before booting up nexus but couldn't get it to work properly so figured just starting it up on boot was easier :)

Update: BTW the script is available under the Apache License 2.0 if anyone wants to copy it or include it in any distro of Nexus

Neat eclipse templates for working with the Java or Spring DSLs for Camel

If you're an Eclipse user who works with Camel you might be interested in the Eclipse templates that Jon has just created - nice work Jon!

Another great Camel tutorial from Claus

Claus has done it again with part 4 of his excellent Camel tutorial. Check it out if you are interested in Enterprise Integration Patterns or Camel