Want to show your appreciation? Please to my charity.

Saturday, March 23, 2013

Integrate Java and GraphicsMagick – gm4java

Introduction

Now we added interactive or batch mode support to GraphicsMagick (GM hereafter), we need to enable the Java side of the integration to complete the implementation of the proposed solution. Hence the gm4java library is born.
In this post, we’ll briefly discuss how gm4java is implemented, then put more focus on its features and API.
Update (3/30): gm4java is available from central maven repository.

Implementation

The implementation of gm4java that we are going to discuss sounds to be complicated but the good news is you don’t have to deal with that because gm4java encapsulates the complexity and does the heavy lifting for you. If you are not interested in the detail, you can skip right to “Feature and API” section. But I believe having some understanding of what’s under the hood always makes me a better driver.
Source code of gm4java can be found at http://code.google.com/p/kennethxublogsource/source/browse/trunk/gm4java/ (update 4/15: moved to https://github.com/sharneng/gm4java)

Interacting with GM Process

gm4java uses Java ProcessBuilder to spawn GM process in interactive mode. Code below illustrate how the GM process is started
   1:  process = new ProcessBuilder().command(command).redirectErrorStream(true).start();
   2:  outputStream = process.getOutputStream();
   3:  inputStream = process.getInputStream();

Note that a pair of the input and output stream are obtained from the process. gm4java uses them to communicate with the GM process, by sending commands to, and receiving feedback from it.
gm4java uses below GM batch options to start GM batch process.
        gm batch -escape windows -feedback on -pass OK -fail NG -prompt off

  • gm4java always use Windows style escape for the compatibility across the platforms
  • prompt is turned off because it is just noise in the machine communication.
  • feedback is turned on because that is how gm4java come to know
    • If GM had completed the execution of the command.
    • The result of the command, whether it was failed or succeeded.
  • gm4java choose OK/NG for pass/fail feedback text, as it is unlikely that the output of any command will produce such text alone in one line.
You now understood that gm4java relies on the proper batch option to operate correctly. Hence, you should never use “set” command directly in gm4java.

GM Process Pooling

Using one single GM batch process in a highly concurrent environment is obviously not enough. Managing multiple GM batch processes is a difficulty task to say the least. gm4java solves this problem by maintaining a pool of GM batch processes for you. It internally uses Apache Commons Pool but hides the complex detail from you so you don’t need to know the Commons Pool in order to use gm4java’s GM process pooling service.

Code Quality

gm4java has nearly 100% code coverage excluding simple getter and setters.

Features and API

The API of gm4java was elaborately designed to be simply. The public interface is very well documented by Javadoc. I’ll cover the basics of the API but I refer you to the API documentation (javadoc) for further reading.
The use of gm4java is very much like using JDBC connection and connection pooling, which most of Java developers are very familiar with.
The primary interface of gm4java is GMConnection.

GMConnection Interface

Just like each JDBC Connection object represent a physical connection to database server, each GMConnection instance represent an interactive session with a GM process running in interactive mode until the connection is closed by invoking its close() method.
   1:  public interface GMConnection {
   2:      String execute(String command, String... arguments) throws GMException, GMServiceException;
   3:      String execute(List<String> command) throws GMException, GMServiceException;
   4:      void close() throws GMServiceException;
   5:  }

GMConnection has two overloaded execute methods, those are the methods you would use to execute GM commands. Each command passed to the execute methods is send to the interactive GM process for execution. The output of the command is then return as a String if it was executed successfully. Otherwise a GMException is thrown and the exception message contains the output of the GM command.
The two execute methods are essentially identical, having two of them is to give you the convenience of passing the command in one way or another.
In simply uses case, you can use varargs version for simplicity. e.g.
    String result = connection.execute("identify", "a.jpg");

But if you need to construct the command conditionally and dynamically, you’ll find the List version is more convenient.
You can use execute methods to run any GM command supported in interactive mode except “set” command. Currently gm4java doesn’t prevent you from doing it but the result is undetermined, mostly failure of all subsequent commands but can also hung. In future, gm4java will try to prevent you from being able to send “set” command.
You must call close() method after you are done with GMConnection, this is again similar to database connection, failure to do so can cause connection leaking. The close() method can effectively end the associated GM process, or just returns the GM process to the pool depends on the implementation of the GMConnection.
Below is the typical usage pattern to ensure connection is closed.
   1:  final GMConnection connection = ...
   2:  try {
   3:      // use connection to run one or many GM commands
   4:  } finally {
   5:      connection.close();
   6:  }

All three methods may throw GMServiceException. It doesn’t like GMException, which is only relevant to the specific GM command being executed, GMServiceException from these methods is usually a permanent condition. It indicates a communication error between gm4java and the interactive GM process, for example, you get this exception if the GM process crashed.
Lastly, please be warned that like JDBC Connection, GMConnection is NOT thread safe!
So far all sounds simple and easy, but GMConnection is an interface, where can we get a real instance of it? That’s is GMService.

GMService Interface

Continue the JDBC analog, GMService is like DataSource. GMService defines three methods, but the important one is the getConnection() method,
   1:  public interface GMService {
   2:      String execute(String command, String... arguments) throws GMException, GMServiceException;
   3:      String execute(List<String> command) throws GMException, GMServiceException;
   4:      GMConnection getConnection() throws GMServiceException;
   5:  }

The getConnection() method returns a GMConnection object associated to either a newly started GM interactive process, or one retrieved from a pool. Whether former or later depends on the actual implementations that we’ll cover in the following sections.
The two execute methods are convenient methods to execute one GM command. Both internally calls the getConnection() method to obtain a connection, execute the given GM command and immediately close the connection. Please see the Java API document for more detail on this.
All methods in GMService are guaranteed to be thread safe.
Although, GMService methods also throw GMServiceException, but it is not necessary, and usually is not, a permanent condition. This is because every time, it deals with a difference instance of GMConnection, hence a different GM process.
GMService is an interface, gm4java provides two concrete implementations of it.

SimpleGMService

SimpleGMService, telling by its name, is a simple implementation of GMService, Its getConnection method always starts a new GM interactive process and return an instance of GMConnection that is associated with that GM process. Closing of that GMConnection effectively stops the associated GM process. Because of this nature, applications use SimpleGMService to create GMConnection should make max use of the connection before closing it. Below is a typical usage pattern.
   1:  GMService service = new SimpleGMService();
   2:   
   3:  public void bar() {
   4:      GMConnection connection = service.getConnection();
   5:      try {
   6:          // use connection to run a lot of GM commands 
   7:      } finally {
   8:          connection.close();
   9:      }
  10:  }

The two execute methods in the SimpleGMService are provided for the purpose of completeness and should not be used in general. The implementation starts a new GM interactive process, run the given command and stop the GM process. That is actually slower than simply running the GM in the old single command mode.
SimpleGMService also has a property called gmPath. You’ll need to set this to full path of GM executable if it is not already in the command search path.

PooledGMService

PooledGMService is the real star in gm4java. Analog to a JDBC connection pool, PooledGMService is able to manage a pool of GM interactive process and distribute the GM commands across them. Hence it is capable of delivering high performance and scalability in a heavily concurrent environment.
Since it internally uses Apache Commons Pool, it can support all pooling features brought in by Commons Pool which is a highly configurable object pool implementation. PooledGMService must be constructed with an instance of GMConnectionPoolConfig object, which contains a list of properties to configure the pool.
The getConnection() method in PooledGMService returns an instance of GMConnection that is associated with a pooled GM interactive process. The close() methods of the returned GMConnection effectively returns the GM interactive process back to the pool for reuse.
For the applications that run many concurrent threads and each thread is just to run one GM command, the PooledGMService is especially helpful. Its two execute() methods become very useful now. Not only they simplify the code to be written, but also optimize internally to perform better. e.g., code below
   1:  final GMConnection connection = service.getConnection();
   2:  try {
   3:      return connection.execute(command, arguments);
   4:  } finally {
   5:      connection.close();
   6:  }

can be simplified to
    return service.execute(command, arguments);

and the later is more efficient.

im4java Integration

So far, gm4java gives a new way to communicating with GM process to get the work done efficiently. It requires you to know the GM command very well and pass each command and its parameters to the execute methods. For people who are not familiar with native GM commands, there can be a steep learning curve.
Thankfully, im4java did a great job of providing a Java friendly interface to construct the GM commands. So gm4java doesn’t need reinvent the wheel. We’ll let you use your familiar im4java interface to construct operations, then give it to gm4java to executed it efficiently.
GMBatchCommand is the bridge between im4java and gm4java. The usage of it is best illustrated by a few sample programs.
Execute a convert command
   1:  GMService service = ...;
   2:   
   3:  public void foo() {
   4:      GMBatchCommand command = new GMBatchCommand(service, "convert");
   5:   
   6:      // create the operation, add images and operators/options
   7:      IMOperation op = new IMOperation();
   8:      op.addImage("input.jpg");
   9:      op.resize(800, 600);
  10:      op.addImage("output.jpg");
  11:   
  12:      // execute the operation
  13:      command.run(op);
  14:  }

Execute the identify command
   1:  GMService service = ...;
   2:   
   3:  public List<String> foo() {
   4:      GMBatchCommand command = new GMBatchCommand(service, "identify");
   5:   
   6:      IMOperation op = new IMOperation();
   7:      op.ping();
   8:      final String format = "%m\n%W\n%H\n%g\n%z";
   9:      op.format(format);
  10:      op.addImage();
  11:      ArrayListOutputConsumer output = new ArrayListOutputConsumer();
  12:      command.setOutputConsumer(output);
  13:   
  14:      command.run(op, SOURCE_IMAGE);
  15:   
  16:      ArrayList<String> cmdOutput = output.getOutput();
  17:      return cmdOutput;
  18:  }

Please note that there are limitations of GMBatchCommand, at this time it doesn’t support im4java asynchronous mode and doesn’t support BufferedImage as output.
We do recognize the integration between im4java and gm4java is still weak. But we also believe there exists strong synergy between im4java and gm4java. Together, we can provide next level of integration between Java and GM.

Future

In next post of this series, I’ll present the new test result of the same web application that we have tested before. This time it uses gm4java.

Other Posts in the “Integrate Java and GraphicsMagick” Series

  1. Conception
  2. im4java Performance
  3. Interactive or Batch Mode
  4. gm4java
  5. gm4java Performance

21 comments:

Mark Waschkowski said...

This looks neat!

"Please note that there are limitations of GMBatchCommand, at this time it doesn’t support im4java asynchronous mode and doesn’t support BufferedImage."

I don't quite understand this - can you expand on what you mean wrt BufferedImage. I'm assuming that means this library doesn't work with any of the older awt classes, correct? If so (please confirm!), what are the limitations in terms of the actions that can be taken with GraphicsMagick (ie. what ones need to be avoided that depends on the awt classes)?

Thank you!

Mark

Mark Waschkowski said...

PS. before I could use the GM library, I was getting class not found errors around slf:

Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory

so I installed slf and I was able to run some test code.

Kenneth Xu said...

@Mark, thanks for the feedback.

The limitation means that you can only ask GM to operate on the files. You cannot steam your BufferedImage to GM, nor stream GM output back as BufferedImage.

Do you us maven, I believe there is a maven dependency setup so if you use maven, it will get the slf4j for you. Otherwise, you'll have to get it yourself. I guess I should have mentioned.

J.J said...

Do we get any progress to merge GM4Java into IM4Java?

Kenneth Xu said...

@J.J, no progress. But you can use IM4Java to construct the command line and execute them with GMBatchCommand to execute them on GM4Java services.

Keith Tsui said...

Trying to convert from im4java to use gm4java. Already got graphicsmagick installed but the question now is how do I use input and output streams since my images are in memory and not on physical disk. im4java commands had getInputProvider and setOutputConsumer. Is there an equivalent or some kind of workaround to do this?

Also, the im4java integration section uses IMOperation. Is that assuming that the user has im4java in their maven dependencies as well? GMBatchCommand also doesn't seem to accept an operation as input.

Kenneth Xu said...

@Keith, If you are using im4java, then you can take a look at the code examples in the "im4java Integration" section of this post. Yes, you need to have im4java in maven dependency and GMBatchCommand does accept IMOperation or GMOperation as input. As to the image in memory, although im4java make you feel like you are sending this directly from memory to GM, but under the hood, it saves your in-memory image into a temp file then pass it to GM. The workaround when using gm4java is to create the temp image file yourself. I found this is better because I can have control of where the temp files are located. I found a small RAMDISK can give great performance for this. It comes free with most of Linux distro and there are free options available for Windows.

Keith Tsui said...

I guess one of my biggest questions are that what is okay to use in im4java and what isn't? I have this current setup to return from an inputstream but the commands seem to be hanging on command.run(op).

IMOperation op = new IMOperation();
final String format = "%w\n%h\n%m";
op.format(format);
op.addImage("-");

GMBatchCommand command = new GMBatchCommand(gmService, "identify");
// set up command
Pipe pipeIn = new Pipe(new ByteArrayInputStream(image), null);
command.setInputProvider(pipeIn);
ArrayListOutputConsumer output = new ArrayListOutputConsumer();
command.setOutputConsumer(output);

command.run(op);

Kenneth Xu said...

@Keith, InputProvider cannot be supported because the stdin/stdout are used to communicate between GM and Java. The workaround to writing your the InputStream to a temp file on some RAMDISK, then pass that file to GM.

Mike Chiarappa said...

Hello. I Have downloaded source by GitHub; for compile has been required two patches:
-1) to avoid a warning in [pom.xml] I have specified the version for artifactId [cobertura-maven-plugin]: at line 94 added version [2.0.3] (the latest available.
-2) for remove a Test error in [GMOperation.java] in method rotate() modified the line 230 in the following:
args.add(String.format((java.util.Locale) null, "%.1f%s", degrees, annotation.asAnnotation()));
This last has been required because in Italy the separator is comma and not dot then (null) Locale avoid conversion ;o))

Mike Chiarappa

Kenneth Xu said...

@Mike, thanks mike for correcting this. If you can send in a pull request, I'm more than happy to merge your changes. BTW, gm4java is on central maven repo. So you can get the binary there if that is what you are interested in.

Anonymous said...

@Kenneth, It's unfortunate that streaming is not supported. This is a show-stopper in using this library for my company. I'm not sure that I buy the "communicate between GM and Java" reason because im4java supports streaming (I have a working implementation) and it, too, must arrange communication between the native process and Java. Does anyone know if streaming is planned for the future, or is this feature off the table altogether?

Deekshant Belwal said...

Hello, I am getting "Broken pipe" error using gm convert using GM4Java. Anyone encountered this error before?

Kenneth Xu said...

@Deekshant It's difficult to help without seeing your code. I would recommend you to post the code and questions to stackoverflow.com, it will be much easier to handle it there. Make sure to tag it with gm4java so that I'll get an alert after you post the question.

Ilya Kavalerov said...

This is a great library! I've been testing the PooledGMService as it is demonstrated here with scala, and it's great. However, I noticed that it does not perform similarly to batch mode within the command line interface for gm. When I run a gm batch file from the command line with any number of images, it launches 1 gm process. However, if I run "execute" with 1 image on a shared PooledGMService across 4 threads (4 images total), I see that 4 separate gm processes are started. I believe that this has performance impacts (a test with 100 images, my scala code against the gm cli with a batch file, takes about the same time, but my scala code uses 4x as much CPU).

My question is: how do use gm4java so that a single gm process is working on several images, just like the cli batch mode? I've tried a few attempts (some desperately silly) with no luck here: https://gist.github.com/ilyakava/1486672d367b4b1654ce
My scala code, which is modeled after the code in this post, can be found here if you are curious:
https://github.com/ilyakava/singlet/blob/master/src/main/scala/remote/RemoteMaster.scala#L40

Thanks, and all the best!
Ilya

Kenneth Xu said...

Hi Ilya,

Glad that you find gm4java useful. If your goal is to use only one GM process, then the SimpleGMService is the friend. Also, why would you run multiple threads in Java if all you want is to use a single GM process?

100 image is a too small sample size to test performance. If you goal is the make best use of your multi-CPU server to convert images, you need to test with large amount of images (at least few thousands) and tweak the configuration to control how many concurrent GM process you'd like to use. If you have only 8 CPUs, don't start more than 7 GM processes. If you are testing on 2-CPU desktop, don't run more than 2 GM processes. I also answered your question in gist.

BTW, if all you need is to create thumbnails, scale command is much faster than convert.

HTH
Ken

Ilya Kavalerov said...

@Kenneth, wow, yes, 3x improvement after switching from the resize command to the scale command (don't have to read and check the size of the image). Thanks for the tips, and for hopping on my SO question:
http://stackoverflow.com/questions/23846193/multiple-image-opperations-in-a-single-process-with-gm4java

I'll continue the conversation there juuuuust in case anyone else jumps on.
Thanks again,
Ilya

Ankit said...

how to set environment variable(I want to set env variable TMPDIR etc to a separate faster disk) while using gm4java in batch mode.

Kenneth Xu said...

@Ankit this is controlled by im4java.Not 100% but if my memory serves me, it uses the Java temp folder. So you can try to change the temp folder of JVM. I tend to not use im4java's feature of sending BufferedImage to GM, which uses a temp file, but rather create the temp file myself and pass the file path to GM. Thay way you have full control of the location and name of the temp file.

Arijit said...

Hi,
I am urgently looking for a high performance image conversion requirement.
I have 200x3 tif files in a folder which is to to be combined in multi-page tif files like
1_1.tif,1_2.tif,1_3.tif-->1_m.tif
2_1.tif,2_2.tif,2_3.tif-->2_m.tif
..........
200_1.tif,200_2.tif,200_3.tif-->200_m.tif
Can i have a command line solution like "gm benchmark mogrify -format tif *.jepg".
what would be gm4java alternative ?
Currently my convert opration through JAI is taking ~27s,can it be reduced to ~2-3s ?
I am testing on WIN 7,Intel core i3 CPU 550 @3.2Hz (2 core).
Please do help ..

Unknown said...

I just use below code to append some text to an image, but get an error "org.gm4java.engine.GMException: convert: Request did not return an image."

public static boolean convertDrawTextToImage(String inputImage, String outputImage, String text, String text_color, int text_size, int x_point, int y_point) {
StringBuilder sbr = new StringBuilder().append(ActionEnum.CONVERT.getAction()).append(" ").append(OptionEnum.POINTSIZE.getOption()).append(" ").append(text_size).append(" ").append(OptionEnum.FILL.getOption())
.append(" ").append(text_color).append(" ").append(OptionEnum.DRAW.getOption()).append(" \"text %d,%d %s\" ").append(inputImage).append(" ").append(outputImage);
String commond = String.format(sbr.toString(), x_point, y_point, text);
System.err.println(commond);
// convert -pointsize 50 -fill #FF6600 -draw "text 75,50 Hopehope..." source.jpg target.jpg: this can be execute in commond, but here does not work well
return doExecute(commond, new ArrayList());
}

private static boolean doExecute(String commond, List commonds) {
assert null != commonds;
String result = "";
try {
// System.err.println(commonds.stream().collect(Collectors.joining(" ")));
if (logger.isInfoEnabled()) {
// logger.info("Do excute commonds is: [{}]", commonds.stream().collect(Collectors.joining(" ")));
}
if (StringUtils.isNotBlank(commond))
result = getGMService().execute(commond, commonds.toArray(new String[commonds.size()]));
else
result = getGMService().execute(commonds);

} catch (GMException |GMServiceException | IOException e) {
e.printStackTrace();
if (logger.isInfoEnabled()){
logger.info("Result: {}, Exception: {}", result, e.toString());
}
return false;
}
return true;
}

please give me some help, thanks very much

Post a Comment