lysergicjava

oh, technology

Menu Close

Category: tech (page 1 of 2)

Feeding TestNG data providers (stupid TestNG tricks volume 1)

TestNG is great.  It’s one of my favorite tools.  Beneath the simple facade lurks a creature of flexibility and power.  That being said, every tool has its limitations, and when TestNG doesn’t give me what I want out of the box, which is rarely, I go poking around looking for ways to game the system.  Here’s one.

Data providers are pretty sweet.  They can feed a test method with a wide variety of data to run against, and the framework even reports the arguments the provider passed in, making forensics easier.  There are limitations in their design which can make them clunky to use in certain situations, though.  Chief among these is that the data providers themselves don’t have an easy interface to accept arguments.

Why, you may ask, would a data provider need to accept arguments?  It’s a fair question, seeing as how the data provider’s responsibility is to dole out arguments.  Turns out there are good reasons.  For instance: I had created a data provider that reads through a file and sends each line as an argument to a test method.  Fairly straightforward stuff; but my first implementation had the provider reading the same file every time.  The file name was hardcoded in the provider.  What I wanted was a way to have a single provider read any arbitrary file, and what’s more, to have the test method itself tell the provider where its data could be found.  Code reuse, yah?

Problem is, data providers can’t take arbitrary inputs.  They can, however, accept a reference to the current test method.  So, with this chink in the armor, I created a new annotation, DataProviderArguments, which I could attach to a test method:

package com.netflix.systemtests.api.commons.dataproviders;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

/**
* Annotation for feeding arguments to methods conforming to the
* "@DataProvider" annotation type.
* @author jharen
*/
@Retention(RetentionPolicy.RUNTIME)
public @interface DataProviderArguments
{
  /**
  * String array of key-value pairs fed to a dynamic data provider.
  * Should be in the form of key=value, e.g., <br />
  * args={"foo=bar", "biz=baz"}
  */
String[] value();
}

Next, I wrote a helper class to unravel the test method and extract the arguments:

package com.netflix.systemtests.api.commons.dataproviders;

import java.lang.reflect.Method;
import java.util.HashMap;
import java.util.Map;

public class DataProviderUtils
{
	protected static Map<String, String> resolveDataProviderArguments(Method testMethod) throws Exception
	{
		if (testMethod == null)
			throw new IllegalArgumentException("Test Method context cannot be null.");

		DataProviderArguments args = testMethod.getAnnotation(DataProviderArguments.class);
		if (args == null)
			throw new IllegalArgumentException("Test Method context has no DataProviderArguments annotation.");
		if (args.value() == null || args.value().length == 0)
			throw new IllegalArgumentException("Test Method context has a malformed DataProviderArguments annotation.");
		Map<String, String> arguments = new HashMap<String, String>();
		for (int i = 0; i < args.value().length; i++)
		{
			String[] parts = args.value()[i].split("=");
			arguments.put(parts[0], parts[1]);
		}
		return arguments;
	}
}

And with that, my FileDataProvider can read any given file:

package com.netflix.systemtests.api.commons.dataproviders;

import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

import org.apache.commons.io.IOUtils;
import org.testng.annotations.DataProvider;

public class FileDataProvider
{
	@DataProvider(name="getDataFromFile")
	public static Iterator<Object[]> getDataFromFile(Method testMethod) throws Exception
	{
		Map<String, String> arguments = DataProviderUtils.resolveDataProviderArguments(testMethod);
		List<String> lines = FileDataProvider.getRawLinesFromFile(arguments.get("filePath"));
		List<Object[]> data = new ArrayList<Object[]>();
		for (String line : lines)
		{
			data.add(new Object[]{line});
		}
		return data.iterator();
	}

	public static List<String> getRawLinesFromFile(Method testMethod) throws Exception
	{
		Map<String, String> arguments = DataProviderUtils.resolveDataProviderArguments(testMethod);
		return FileDataProvider.getRawLinesFromFile(arguments.get("filePath"));
	}

	@SuppressWarnings("unchecked")
	public static List<String> getRawLinesFromFile(String filePath) throws Exception
	{
		InputStream is = new FileInputStream(new File(filePath));
		List<String> lines = IOUtils.readLines(is, "UTF-8");
		is.close();
		return lines;
	}
}

And now, my test methods can specify what file to fetch their data from:

@Test(dataProviderClass=com.netflix.systemtests.api.commons.dataproviders.TestAccountDataProvider.class, dataProvider="getTestAccountsFromFile")
@DataProviderArguments("filePath=src/main/resources/input-files/testAccounts.txt")
public void fetchInstantQueueAsJSON(String email, String accountID) throws Exception
{
  // blah blah blah testy test goes here
}

For my next trick, I think I’ll work on getting TestNG TestFactories to play the same game.  Then it’ll get interesting.  Till then, keep your bars green and your code clean.  Happy testing!

JPath 1.0 released

Ask anyone who’s had to work with JSON in Java: consuming, querying, and validating JSON objects in Java is a tedious chore. Either you’re stuck writing structured beans mapped to JSON responses, which is time-consuming and brittle, or you need to resort to circumlocutious chains of get() calls with typecasting (or foreknowledge of types) at every step. In particular, there is no equivalent (or even approximation) of XML’s XPath — until now.  JPath allows for concise, flexible navigation and validation of JSON data.  Check it out over at BLC.

Mounting shared folders with Win7 / Ubuntu Virtualbox pair

The classic instruction set for mounting a shared folder between a windows host and a linux guest, found here, works as advertised except for one small detail.  If your host is a windows7 install, you must define the shared folder in Virtualbox BEFORE starting the guest OS.  If, like me, you define the shared folder while the guest OS is running, the shared folder definition will be placed under the “transient folders” section of the definitions, not the “machine folders” section which is where it needs to be.  The trouble starts when you try to mount the shared folder.  You’ll get the error:

“/sbin/mount.vboxsf: mounting failed with the error: No such device”

Shut down the guest OS, add the shared folder to the virtual machine’s definition, and then you’re good to go.

Wireshark on MacOS X happy fun time

Wireshark is killer, except when it doesn’t play well with the underlying OS.  Like how OS X finds it hilarious to hide your network interfaces from your applications.  Go ahead, fire up Wireshark on the mac and watch as it can’t find a single interface.  I’ll wait.

Yeah, pain, isn’t it?

Well, you can fix that:

sudo chmod 664 /dev/b*

And you’re back in action.

Fixing Apple Laptop Network Slowness Problems

I’ve experienced this on two different laptops now: the wireless connections start to suck.  Out of nowhere the network just slows to a crawl.  Plug into ethernet and you’re jake.  All other wireless devices on the network are humming away just fine with no loss of bandwidth.  It’s just the Apple laptops.  It’s infuriating, but there’s a very simple (albeit weird) fix for it.

Just go to network preferences and pull down the location menu. From there select edit locations. Create a new location. Doesn’t matter what you call it, it’s all arbitrary. Then delete whatever location you were using, even if it is “Automatic”. Switch to the new location and set it up like normal.

That’s it.

God only knows why this works, but it does.

Making Maven builds platform-independent

This isn’t much publicized, but encoding issues can bite you in the butt, especially when switching platforms.  There’s an easy way to eliminate 95% of all encoding hassles,  though — just add a single property:


<properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

to your pom, or parent pom.  Voila!  You're set for life.

Google grows a pair

According to the latest Google Public Policy blog post, the big G is “reconsidering” what doing business with the big C means.  Apparently, someone (cough: the Chinese government) has been attempting to break Google’s security and steal information about human rights activists. Google’s of the opinion that enough is enough, and I couldn’t agree more.

Everyone, lend your support to Google in this endeavor.

Original post: Official Google Blog: A new approach to China.

Netflix client initial release

Haven’t been active on the blogging front lately, as I took a new job, moved, and all that.  Also, I spent some spare time building a client library for the Netflix API.  You can take a look here.

Closing the book on closures

I keep getting questions about closures, mostly from people who know me and my YAGNI attitude.  “I don’t get what’s so great about them.  What can you do with closures that you couldn’t do otherwise?”

Answer: Nothing.

Well, that’s only partly an answer.  It’s true, closures aren’t magic.  There’s nothing you can do with them that you couldn’t do otherwise, but there is something great about them.  For example, in Groovy, you could do something like this:

def sql = Sql.newInstance("jdbc:mysql://server/database", "sa", "pword", "Driver")
sql.eachRow("select foo, bar from foobars") { row ->
  def foo = row["foo"]
  def bar = row["bar"]
  print "Foo = $foo and bar = $bar"
}

Here, Groovy’s Sql class has a method called eachRow that accepts two parameters: a query, and a closure to execute for each row in the result.  Functionally, this is the same as:

// assume you've got your db connection setup
ResultSet rs = conn.execute("select foo, bar from foobars")
while rs.hasNext()
{
  def foo = rs.getString("foo")
  def bar = rs.getString("bar")
  print "Foo = $foo and bar = $bar"
}
rs.close()
conn.close()

…and so you may be forgiven for thinking closures are much ado about nothing.  After all, in both cases, all you’ve done is open a connection, execute a query, iterate over the results, and do something with each row.  Not a big deal, right?

Well, while functionally the two code samples are equivalent, the closure example reveals something special.  See the last two lines of the non-closure example, the calls to rs.close() and conn.close()?  You don’t need those when calling Sql.eachRow because it does it for you.  The designers of Groovy, in essence, recognized that when you’re dealing with databases it’s real common to open a connection, execute a query, iterate over the results, and do something with each row.  Oh yeah, and clean up the resources afterward.  Now, with repetitive things it’s a good idea to encapsulate the invariants (opening, iterating, and closing) and abstract what varies (the queries and what you do with the results).  The query’s already abstracted as a SQL string, so let’s abstract the code that processes the results and we’re good to go.

There’s something else.  What we’re doing when we pass a closure to the eachRow() method is metaprogramming — partially specifying, at runtime, the behavior of another object.  But we’re doing it responsibly; we’re not just clobbering or overriding some behavior, we’re fulfilling a contract.  Of course, without closures, you could accomplish the same thing by using, say, the command pattern.  (More or less this is how the Groovy runtime does it behind the scenes, anyway).  But the closure paradigm is cleaner and more convenient.

Which gets to the heart of it: using closures properly results in cleaner, more elegant code, and that’s all.  But isn’t that what you want, anyway?

Hacking Eclipse plugin configurations

Reading Neal Ford’s generally excellent book The Productive Programmer, I experimented with his multiple plugin configuration hack. In short, creating multiple plugin configurations in a single eclipse install allows for a team to keep their plugin configurations in version control, so everyone has exactly the same configurations as anyone else on the team. No more “works on my machine but not on yours” weirdness. You can even manage them on a project-by-project basis, which is good.

IT Stamp of Approval

How many times have you heard this?

But there are two aspects of working with multiple plugin configurations that are strange: specifically, creating and deleting them. In order to create an additional plugin configuration, you have to:

  1. Create a folder to hold the configuration. It must be named “eclipse” and it must not be in Eclipse’s directory structure.
  2. Within your “eclipse” folder, you have to make an empty file called .eclipseextension, and two empty folders, features and plugins.

As Ford points out, Eclipse (inexplicably) won’t do this for you. You have to do this by hand. It’s not hard, just strange. At least this way you have more control over where the folders and files are located. (As I have multiple Eclipse installs, I used nesting to keep track of everything. I created a top-level folder called “eclipse-configurations”… under that, I made another folder for each of my named installs, and under each of those, I placed the “eclipse” folder as mentioned above. So, the versioned configuration for my “xquery” install of Eclipse is at /Users/haren/eclipse-configurations/xquery/eclipse.)

From there, you can go to Help -> Software Updates -> Manage Configuration and add your configuration location(s). Then it’s a simple matter of installing your plugins to the config locations desired. You can then enable and disable multiple plugins as a group, switch between versions, etc. It’s very handy.

But I’d mentioned that there were two strange things about the process. Creating additional configuration locations was one, deleting them was the second. Just as Eclipse gives you no love in creating them, it makes it even harder to get rid of them.

Let’s say, for example, that you’ve added your new location as an extension not to the top-level list, but as an extension to an extension. (Yes, you can do this.) But let’s also say that’s not what you wanted. Well, you can disable your extension-within-an-extension, but you can’t get Eclipse to ignore its existence entirely. If you then try to add it to the top level, Eclipse won’t let you, complaining that you’ve already added it elsewhere. Arg.

Well, there’s a way around that, too (but Ford doesn’t mention it). Under ${ECLIPSE_HOME}/configuration/org.eclipse.update there’s a file called platform.xml. Up at the top there are “site” nodes, and one of those will be your offender. Delete the bad guy and restart Eclipse. Now you can place your configuration elsewhere. (Or, you can just change the path in the node).

Anyway, as noted, there’s a lot to gain by using multiple plugin configuarations, once you get around Eclipse’s strange reluctance to make it intuitive. Happy hacking!