Choosing an Application Framework for Microservices

08 August 2017

Tags: java spring-boot microservices

Earlier this year, I authored a blog post for New Relic where I discussed how my team decided to use Spring Boot to build microservices. Recently, the post has been picked up by a bunch of other outlets and has been making the rounds on the interwebs. Please give it a read and am interested in your thoughts on the subject!

Comments

The Peculiar Case of the Kafka ConsumerIterator

20 December 2016

Tags: java kafka

This post is written using Apache Kafka 0.8.2

Creating a Apache Kafka client is a pretty straight-forward and prescriptive endeavor. What is not straight-forward or even expected is the behavior of the Iterator that is used to poll a Apache Kafka topic/partition for messages. More on this in a moment. First, let’s look at the typical setup to consume data from a Apache Kafka stream (for the sake of keeping this post brief, I am going to skip the details around creating and configuring a ConsumerConnector):

final Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap =
    consumerConnector.createMessageStreams(Collections.singletonMap("topic",1));
final List<KafkaStream<byte[], byte[]>> streams = consumerMap.get("topic");
final ConsumerIterator<byte[],byte[]> = streams.get(0).iterator();‍‍‍‍‍‍‍‍

With the ConsumerIterator in hand, the next step is to poll the Iterator for incoming messages:

while(iterator.hasNext()) {
    MessageAndMetadata<byte[], byte[]> message = iterator.next();
    ...
}‍‍‍‍

‍ ‍This all seems pretty simple. Now, back to the issue with this code: the expectation is that this would check the Iterator for a message and if not present loop immediately and check again (standard Iterator behavior). However, this is not the case. The behavior of the ConsumerIterator is actually controlled by the consumer.timeout.ms configuration setting. This setting controls whether or not the Iterator “throw(s) a timeout exception to the consumer if no message is available for consumption after the specified interval”. By default, this value is set to -1, which means that the call to hasNext() will block indefinitely until a message is available on the topic/partition assigned to the consumer. The Java documentation for the Iterator interface does not specify whether or not the hasNext() method is allowed to block indefinitely, so its hard to say that the ConsumerIterator is violating the contract. However, this is certainly not the behavior anyone use to using the Iterator pattern in Java would expect, as collections typically don’t block until data is available in the data structure. If the consumer.timeout.ms configuration setting is set to a positive value, the consumption code would need to be modified to handle a ConsumerTimeoutException:

while(active) {
    try {
        if(iterator.hasNext()) {
            MessageAndMetadata<byte[],byte[]> message = iterator.next();
            ...
        }
    } catch(ConsumerTimeoutException e) {
        // Do nothing -- this means no data is available on the topic/partition
    }
}‍‍‍‍‍‍‍‍‍‍

‍ Now, the call to hasNext() will behave more like an Iterator retrieved from a collection, which is to say it will not block indefinitely. It is recommended that you do some testing to determine an acceptable timeout value to avoid looping too frequently, as this will cause an increase in CPU utilization by the loop. It is also worth noting that the Kafka documentation does not directly link the configuration setting and the ConsumerIterator and this issue would most likely go unnoticed in scenarios where data is consistently available to the client. In any case, this issue highlights the need to take a deeper look at any API or library you include in your application in order to ensure that you understand exactly how it works and what performance impacts it may have on the execution of your code.

Comments

Testing a custom Gradle plugin with Gradle Test Kit

29 March 2016

Tags: gradle groovy

With the release of Gradle 2.10, the Gradle Test Kit was included as an "incubating" feature to "[aid] in testing Gradle plugins and build logic generally." Prior to the creation of the Gradle Test Kit, it had been fairly cumbersome to test custom Gradle plugins. Tests often involved using the ProductBuilder to create a dummy instance of a Project and retrieving a declared Task and executing it manually. While this would test the task logic directly, it did not test the execution of the task as part of a normal Gradle execution. Furthermore, it would not exercise task-based caching, making it hard to verify that any configured inputs/outputs are being honored. This is where the Gradle Test Kit can help. It is focused on functional testing, which means that it emulates what a user will see when attempting to run tasks via the command line or Gradle wrapper. Being an "incubating" feature, however, some of the documentation is lacking, especially when it comes to testing a custom Gradle plugin within the project that contains the plugin definition and source. In this post, we will explore how to set up your custom plugin’s project to use the Gradle Test Kit.

Using the Gradle Test Kit

The first step is to include the Gradle Test Kit as a test scoped dependency in your project’s build.gradle file:

dependencies {
  ...

  testCompile gradleTestKit()
  testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'
}

This will pull in the Gradle Test Kit libraries for use during the test phase of your project.

Creating the Plugin Classpath

The next step, which is hard to determine from the documentation, is to make sure that custom plugin and its descriptor are on the classpath path when using the Gradle Test Kit. In its current form, there is no easy way to pass/build this classpath as part of a Spock Framework test at runtime. The trick is to follow what is outlined in section 43.2.1 of the Gradle Test Kit documentation, which outlines how to create a text file containing the classpath to be used by the Gradle Test Kit:

task createPluginClasspath {
    def outputDir = file("${buildDir}/resources/test")

    inputs.files sourceSets.test.runtimeClasspath
    outputs.dir outputDir

    doLast {
        outputDir.mkdirs()
        file("${outputDir}/plugin-classpath.txt").text = sourceSets.test.runtimeClasspath.join('\n',)
    }
}

In the example above, we use the runtime classpath of the test configuration to generate the classpath list to be passed to the Gradle Test Kit runner. The example in the Gradle Test Kit documentation uses the main configuration, which is fine if you don’t need to provide any additional libraries for testing. In my case, I needed to have some other custom plugins available for the functional test, but did not want those dependencies to be on my main compile or runtime classpath. If you don’t want to have to manually call this task each time you test your project, you can add the following your build.gradle script to tie its execution to the test task:

test.dependsOn(['createPluginClasspath'])

Now that we have a task to generate the plugin classpath text file, we need to use it as part of our test. In the example below, the contents of the plugin-classpath.txt file read, collected, converted into File objects and stored into a list:

class MyPluginFunctionalSpec extends Specification {

    @Rule
    TemporaryFolder testProjectDir = new TemporaryFolder()

    File buildFile

    File propertiesFile

    List pluginClasspath

    def setup() {
        buildFile = testProjectDir.newFile('build.gradle')
        propertiesFile = testProjectDir.newFile('gradle.properties')
        pluginClasspath = getClass().classLoader.findResource('plugin-classpath.txt').readLines().collect { new File(it) }
    }

    ...

The pluginClasspath list will be passed to the Gradle Test Kit runner via the withPluginClasspath method of the builder, which we will see in a bit.

Building the Functional Test

Now that we have our classpath sorted out, the next step is to build test(s) to execute your custom plugin and task(s):

def "test that when the custom plugin is applied to a project and the customTask is executed, the customTask completes successfully"() {
    setup:
        buildFile << '''
            plugins {
                id 'my-custom-plugin
            }

            dependencies {
                compile 'com.google.guava:guava:19.0'
                compile 'joda-time:joda-time:2.9.2'
                compile 'org.slf4j:slf4j-api:1.7.13'

                runtime 'org.slf4j:log4j-over-slf4j:1.7.13'

                testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'
            }

            repositories {
                mavenLocal()
                mavenCentral()
            }
        '''
    when:
        GradleRunner runner = GradleRunner.create()
            .withProjectDir(testProjectDir.getRoot())
            .withArguments('customTask', '--stacktrace', '--refresh-dependencies')
            .withPluginClasspath(pluginClasspath)
        BuildResult result =  runner.build()
    then:
        result.task(':customTask').getOutcome() == TaskOutcome.SUCCESS
}

In the example above, notice that we build a full build script, which includes the application of our custom Gradle plugin, and output it to buildFile created in the setup seen previously. This can be anything that you would do in a project’s build.gradle file. You could even store these files in src/test/resources and load and copy the contents of these files from the classpath and write it out to the file to be provided to the Gradle Test Kit runner. In the when block, we see the Gradle Test Kit in action. Here, we set the project directory to the TemporaryFolder that will contain the build.gradle file, the arguments to be passed to Gradle (e.g. the task(s) and switches), and the plugin classpath we generated in the setup. Without the plugin classpath, you will see errors related to Gradle being unable to locate any plugins that match your custom plugin’s ID. Finally, in the then block, we see that we test to make sure the status of the task execution is the one we expected. You can also inspect the output of the build by inspecting the output field of the BuildResult:

result.output.contains('some text') == true

This is also useful for debugging, as you can print out the contents of the result output to see the full output of the Gradle execution.

Test Failures and Xerces

Depending on what is on your plugin classpath, you may have tests fail due to issues related to the Xerces library. This is often due to multiple versions of Xerces being present on the classpath when the runner is executed and can be remedied by excluding Xerces from the generated classpath:

pluginClasspath = getClass().classLoader.findResource('plugin-classpath.txt').readLines().collect { new File(it) }.findAll { !it.name.contains('xercesImpl') }

Notice that we added a step to find all the classpath entries that do not contain the string xercesImpl to ensure that we do not end up with duplicate Xerces implementations on the classpath provided to the test kit runner.

Summary

The Gradle Test Kit provides an excellent way to functionally test your custom Gradle plugins. Because it uses actual build scripts, it is easy to build up a library of configurations that you want to continually test as changes are made to the custom plugin. Furthermore, the Gradle Test Kit drastically reduces the amount of test code that you need to write, allowing you to more efficiently test your plugin. All of these are great reasons to convert your plugin tests to use the Gradle Test Kit or to write tests for the first time if you don’t currently have test coverage for your code.

Comments

Using Spock/CGLIB with Java 8 and Gradle

18 August 2015

Tags: gradle java groovy

Recently, I attempted to convert a project that uses CGLIB and Spock Framework to execute unit tests to use Java 8 source and target versions. When I went to run the unit tests, I was met with a nasty java.lang.IllegalArgumentException. Some quick internet searching showed that there is an existing issue with the currently released version of CGLIB and the version of the ASM library it depends on. Despite pleas from the community, an updated version of CGLIB with Java 8 compatibility has not yet been released. However, there are commits on the project’s trunk that add Java 8 compatibility. Luckily, a quick look at the Spock Framework source code showed me how to get around this issue. The trick is to use JitPack.io as a Maven repository in your build script to build a particular commit hash of the project and expose it as a dependency for your project. After some digging through the source, I determined that commit 52e118aca4 is the proper one to use. To use this version of CGLIB in your build.gradle file, you simply need to add the following:

repositories {
    // ...
    maven { url "https://jitpack.io" }
}

dependencies {
    testCompile 'com.github.cglib:cglib:52e118aca4'
    testCompile 'org.spockframework:spock-core:0.7-groovy-2.0'
}

You can generate this information using JitPack.io itself by going to the following URL: https://jitpack.io/#cglib/cglib/52e118aca4. Now, my project will get JitPack.io to build the custom release of CGLIB without me needing to build and push it into a repository!

Comments


Older posts are available in the archive.