Thursday, June 5, 2014

RHQ: monitoring JMX beans of a Java EE application deployed on Wildfly 8

You may already have read my post on monitoring JMX beans of a Java EE application deployed on EAP 6 ? And maybe wondered how to do the same with Wildfly 8?

Well it's possible now with a master branch build of RHQ (the feature will be released with RHQ 4.12). A couple of problems had to be addressed.

Firstly, we needed proper support for Wildfly monitoring. RHQ was able to discover Wildfly servers as such, but failed to monitor them as soon as they were added to inventory. This was due to a wrong product name check in AS7 / Wildfly8 plugin.

Secondly, the ApplicationMBeansDiscoveryComponent class needed  an update to take Wildfly management and port changes into account:
  • single port for management and remoting on a standalone server (HTTP upgrade)
  • single port for applications and remoting on a managed server in domain mode (HTTP upgrade again)
  • new JMX service transport ("http-remoting-jmx")
Also, the Wildfly version of jboss-client.jar (required to support "http-remoting-jmx" transport) has class loading issues with RHQ's AS7 / Wildfly8 plugin. So we had to update our EMS[1] library connection descriptor to isolate the connection classes loader.

The new feature has been tested with Widfly 8.1.0.CR2 in standalone and domain mode but since WildFly 8.1.0.Final was released recently, we'd be glad if you give it a try and come with feedback on the RHQ forum!

[1] RHQ uses the EMS library to simplify its JMX connectivity code.

Friday, May 16, 2014

A new version of rhq-agent-plugin-plugin is out

I just released a new version of the Maven plugin for building RHQ Agent plugins.

v0.6 is a small release which:
  • fixes a bug in the validation mojo (JAR without plugin descriptor used to pass validation)
  • lets you use the failOnError plugin configuration attribute with the RHQ CLI Command and RHQ CLI Script mojos (before RHQ 4.11 the CLI returned exit code 0 even after an error occured)
The bits are already available on JBoss public repositories and should be very soon on Maven Central.

Special thanks to Blandine Bourgeois, Victor Grousset and Eliesse Hadjem for their contributions (Marseille JUG Hackergarten power!)


Tuesday, March 18, 2014

RHQ: monitoring JMX beans of a Java EE application deployed on EAP 6

 Everything in this article applies to both EAP 6 and AS 7

On RHQ forum, we have been asked quite a few times how to monitor MBeans of a Java EE application deployed on EAP 6.

Users are often confused because on one hand, all the goodies from the JMX plugin need a JMX connection while, on the other hand, EAP 6 plugin communicates with managed servers over an HTTP management connection.

JMX connection to EAP 6

A JMX connector is accessible over the management-native port (9999 by default) on standalone servers and over the remoting port (4447 + server port offset) on managed servers (domain mode).

A custom protocol is used ("remoting-jmx") so the JMX service url will start with "service:jmx:remoting-jmx://" and EAP6_HOME/bin/client/jboss-client.jar has to be in the client code's classpath.

For authentication, standalone servers rely on the ManagementRealm, and Managed Servers on the ApplicationRealm.

If you correctly configured your server, you should be able to see your MBeans in JConsole at this point (see Using jconsole to connect to JMX on AS7).

Custom plugin requirements

So we want RHQ Agent to auto-discover our application MBeans and the corresponding resources to show up under the parent standalone server or the parent managed server.

For that, we need to create a custom plugin extending the EAP 6 plugin (see Plugin Injection Extension model). The plugin descriptor will contain a line like:

We also need component and discovery component classes from the JMX plugin, but  RHQ prevents plugin developers to use classes from more than one dependency. So there's no other choice here, the custom plugin will have to embed the JMX plugin JAR.

Eventually, we must create a component class (and its discovery counterpart) that will group our MBean resources and provide them with a JMX Connection.

AS7 JMX Util project

In RHQ's repository, the AS7 JMX Util project gives the root component classes for grouping MBeans resources: ApplicationMBeansDiscoveryComponent and ApplicationMBeansComponent. Here is how they work.

ApplicationMBeansDiscoveryComponent will be the discovery class of the custom "grouping" resource type. Plugins authors are required to provide a resource key and name and may provide a resource description and version by declaring plugin configuration properties of the following names:
  • ApplicationMBeansDiscoveryComponent.PluginConfigProps.NEW_RESOURCE_KEY
  • ApplicationMBeansDiscoveryComponent.PluginConfigProps.NEW_RESOURCE_NAME
  • ApplicationMBeansDiscoveryComponent.PluginConfigProps.NEW_RESOURCE_DESCRIPTION
  • ApplicationMBeansDiscoveryComponent.PluginConfigProps.NEW_RESOURCE_VERSION
Alternatively, they can create a subclass of ApplicationMBeansDiscoveryComponent and override one/some/all of the following methods:
  • getNewResourceKey(Configuration)
  • getNewResourceName(Configuration)
  • getNewResourceDescription(Configuration)
  • getNewResourceVersion(Configuration)
By default, application MBeans will be searched with an EMS query defined in the plugin descriptor by the ApplicationMBeansDiscoveryComponent.PluginConfigProps.BEANS_QUERY_STRING plugin-config property. It is also possible to define the query string in a subclass overriding the getBeansQueryString(Configuration) method.

Plugin developers can implement their custom MBeans lookup method in a subclass overriding the hasApplicationMBeans(Configuration, EmsConnection) method.

JMX host and port discovery

The JMX server host will be detected by looking at the top level server resource plugin configuration (StandaloneASComponent or HostControllerComponent). The JMX port detection mechanism depends on the parent resource:
  • on standalone servers, the discovery component will look at the management port defined in the parent StandaloneASComponent plugin configuration and add the value STANDALONE_REMOTING_PORT_OFFSET
  • on managed servers, the discovery component will use '4447' plus the port offset of the managed server.


JMX connectivity on EAP6 requires authentication and standalone and managed servers behaviors differ:
  • on standalone servers, the EAP 6 server will use the ManagementRealm, just as for HTTP management interface authentication. Consequently, the ApplicationMBeansDiscoveryComponent will pick up the credentials of the management user defined in the plugin configuration of the parent server resource (StandaloneASComponent)
  •  on managed servers, the EAP 6 server will use the ApplicationRealm. Consequently, there is (currently) no way for the discovery component to discover credentials automatically and it will use "rhqadmin:rhqadmin" by default.
When working with managed servers, plugin developers should subclass the discovery component and override the getCredentialsForManagedAS() method. For example, an implementation could lookup the credentials in a text file.

The custom agent plugin

The custom agent plugin must:
  • contain a plugin descriptor following the requirements described above
  • embed the JMX plugin JAR
  • embed the AS7 JMX Util project JAR
  • optionally, contain a subclass of ApplicationMBeansDiscoveryComponent 
This is very straightforward when the custom plugin is built with the Maven plugin for RHQ agent plugins: you need a POM file and a plugin descriptor (there is a sample project in RHQ repository). Let me show you what the POM file and the plugin descriptor look like:

That's it! Here are some screenshots of an example application in inventory:

"Myapp Services" grouping resource plugin configuration
"Myapp Services" grouping resource plugin configuration

Creating an "Hello Service" operation, on a managed server (domain mode)
Creating an "Hello Service" operation, on a managed server (domain mode)

Enjoy and let us now if you like it or need any help (as always, on RHQ forum or the users mailing list).

Monday, March 17, 2014

Announcing Simone project, a Java library for system monitoring

In English, pronounce sə-mōn'
Today, I pushed to Github the initial commit for Simone, a Java library for system monitoring, with no JNI code. Native invocation, when needed, is provided by JNR and friends (JNR Posix, … etc)

This work is experimental.

Currently, you can get information on CPUs, Memory and Processes on Linux systems. Next step is filesystems and networks statistics. Windows and Mac OS X are planned as target platforms.

Give it a try!

It's not yet available in any Maven repository so you’ll have to build it yourself.

Any issues? Questions? Feel free to report with GitHub issues.

EDIT: I forgot to point to the Simone project on GitHub

Wednesday, December 4, 2013

Announcing RHQ Agent Plugin Plugin

I'm happy to announce the release of RHQ Agent Plugin Plugin!

Plugin Plugin? Yes. Plugin Plugin: a Maven plugin to help you build RHQ Agent plugins.

It provides a handful of very convenient mojos for:
  • Packaging
  • Validation
  • Deployment to a local RHQ Server (dev-container)
  • Uploading to to a remote RHQ Server
  • RHQ CLI script or command execution
  • Standalone Plugin Container setup for integration tests

Dependent agent plugins are supported (many plugins depend on the parent JMX plugin, for example).

The RHQ Agent Plugin Plugin documentation is hosted on Github Pages and the bits are available on JBoss Releases and Maven Central repositories.

Need help to get started?

Thursday, October 10, 2013

Uploading large content for package backed resources or bundles with the RHQ CLI

Until recently, the only way to create a package backed resource, update its backing content or create a bundle version with the RHQ CLI was to fully load the file in memory as a byte array.

Consequently, as soon as you tried to send something of relatively large size, the CLI crashed with OutOfMemoryError. In practice, with the default CLI max heap size (128MB), that would be about 60 MB.

There is now a way to avoid that and send content or bundles of any size. The ContentManager has been extended to allow fragmented content uploads:

As you can imagine, you'll have to request first the creation of new temporary content handle. Then you'll be able to upload fragments sequentially. For ease of use, a utility method has been added to the ScriptUtil object (bound to 'scriptUtil' ):

So here's how you'll create a package backed resource:

The bundles.js sample script has also been updated in order to upload the bundle file in fragments when creating a bundle version:

These changes have been pushed to the master branch and will be released with RHQ 4.10.

That's it!

Tuesday, March 5, 2013

RHQ Net Services plugin enhancements

The RHQ Net Services plugin has just been updated with new features. It now defines three types of services:
  • HTTP service
  • Ping service
  • Port service
You can add any number of these services to a managed platform with the "Import" button:

RHQ netservices import
Import a Net Service on a managed platform
Note that the network connections will be made from the agent monitoring the platform you add the services to. So make sure this agent can connect to your HTTP or IP endpoints (is the remote network reachable? will you hit firewall rules?)

HTTP service

HTTP service monitors availability of an HTTP URL endpoint. By default the endpoint is reported up if a GET request to its URL gives any response. It can be configured to validate response code and response content (with a regular expression).

If you validate the response content, bear in mind that it will be entirely loaded in memory. So for efficiency and reliability don't use this option with large sized responses.

The HTTP service defines a "contentLength" metric. This metric is filled with the HTTP response content length header. When the response does not contain this header, the metric will have a default value of -1. You may want to disable this particular metric in this case.

Ping service

Ping service lets you know if a remote IP is reachable. The internal mechanism involved may vary upon JVM implementation but will typically be an ICMP echo request or a TCP request to port 7 (echo service).

Port service

Port service monitors availability of a remote port for TCP connection. This can be useful to check if a remote service you cannot manage with an RHQ agent is listening to your connections.

Want more?

Would you like to see other type of services in this plugin? Come and tell us in the forum.