Using Input Pipelines In Your Android Game

Real-time Android games have a few threads to worry about. You can't block up the main UI thread because it's needed by the OS. If you block it, even for a second, your app will be deemed unresponsive and users will get the dreaded "Force Close or Wait" dialog. If your game doesn't do anything in a logic or main thread then you should be in the clear. If you do use a main thread for your game and run it as fast as it can go, as is the case with 2D games that draw to the canvas or 3D games that simply have a lot of game logic to process, you may want to think about how you're handling your input.

I had problems with the original Light Racer's input. I was using the example from Lunar Lander originally which simply synchronized the main loop and the input entry point. For some reason, even though in every Java app I've ever developed before that has the same code, the game would randomly not give the synchronized block to the UI thread that's waiting for it. The main loop would synchronize on the same mutex and release it over and over, even sleeping for a few MS to allow for other threads to get in there, but still the UI thread wouldn't do it. It was very strange and frustrating. After thinking hard about the problem, I decided to switch to input pipelines and all of my problems were solved.

How does an input pipeline work?

For our example, I'm going to use motion events since they can be intense.

Normally, the OS has an input event ready to go so it calls Activity.onMotionEvent() with the event. In a game activity that has a game thread running, you usually don't want to just interrupt the thread because there's no telling where it is in the processing and if you change the input on it mid-logic, you could really cause some problems. So, instead, it works like this:

doLoop() {
synchronized(mutex) {
process everything
}
sleep(2)
}

And the input comes in like this
doMotionEvent() {
synchronized(mutex) {
process motion event
}
}

In theory, this works fine. The problem I've seen on Android is that it doesn't work fine. If the main loop is really CPU intensive, the motion even sometimes gets stuck and is never able to get into the synchronized block! I don't know why that is. In a normal Java VM, the moment that the main loop is out of the synched block, the next thread that is waiting to get into the block is allowed to go.

An Input Pipeline is a simple a queue of input events that don't block the UI thread at all. Here's kind of how it looks:

The UI Thread calls:
doMotionEvent(MotionEvent e) {
InputEvent ie = convertToInputEvent(e);
gameThread.feedInput(ie);
}

We convert the MotionEvent into our own event and then feed it to the game thread.

On the game thread side, it needs to be able to handle the feed and then process the event.

feedInput(InputEvent ie) {
synchronized(inputQueueMutex) {
addToInputQueue(ie);
}
}

There's still a synchronized block but it's much more specific. This mutex is only used for 2 methods: feedInput and processInput. Guess what calls processInput? The main loop, and since it's for such a small fragment of time, feedInput is almost never blocked. This works much better than synchronizing the entire game loop which, for a really computationally intensive game, can and does get held for nearly 100% of the time.

Ready for the code? Here it is:

The InputObject:

public class InputObject {
public static final byte EVENT_TYPE_KEY = 1;
public static final byte EVENT_TYPE_TOUCH = 2;
public static final int ACTION_KEY_DOWN = 1;
public static final int ACTION_KEY_UP = 2;
public static final int ACTION_TOUCH_DOWN = 3;
public static final int ACTION_TOUCH_MOVE = 4;
public static final int ACTION_TOUCH_UP = 5;

public ArrayBlockingQueue<InputObject> pool;
public byte eventType;
public long time;
public int action;
public int keyCode;
public int x;
public int y;

public InputObject(ArrayBlockingQueue<InputObject> pool) {
this.pool = pool;
}

public void useEvent(KeyEvent event) {
eventType = EVENT_TYPE_KEY;
int a = event.getAction();
switch (a) {
case KeyEvent.ACTION_DOWN:
action = ACTION_KEY_DOWN;
break;
case KeyEvent.ACTION_UP:
action = ACTION_KEY_UP;
break;
default:
action = 0;
}
time = event.getEventTime();
keyCode = event.getKeyCode();
}

public void useEvent(MotionEvent event) {
eventType = EVENT_TYPE_TOUCH;
int a = event.getAction();
switch (a) {
case MotionEvent.ACTION_DOWN:
action = ACTION_TOUCH_DOWN;
break;
case MotionEvent.ACTION_MOVE:
action = ACTION_TOUCH_MOVE;
break;
case MotionEvent.ACTION_UP:
action = ACTION_TOUCH_UP;
break;
default:
action = 0;
}
time = event.getEventTime();
x = (int) event.getX();
y = (int) event.getY();
}

public void useEventHistory(MotionEvent event, int historyItem) {
eventType = EVENT_TYPE_TOUCH;
action = ACTION_TOUCH_MOVE;
time = event.getHistoricalEventTime(historyItem);
x = (int) event.getHistoricalX(historyItem);
y = (int) event.getHistoricalY(historyItem);
}

public void returnToPool() {
pool.add(this);
}
}

In the game activity:

private ArrayBlockingQueue<InputObject> inputObjectPool;

protected void onCreate(final Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
//gameView = new GameView(this);
//setContentView(gameView);
createInputObjectPool();
}

private void createInputObjectPool() {
inputObjectPool = new ArrayBlockingQueue<InputObject>(INPUT_QUEUE_SIZE);
for (int i = 0; i < INPUT_QUEUE_SIZE; i++) {
inputObjectPool.add(new InputObject(inputObjectPool));
}
}

@Override
public boolean onTouchEvent(MotionEvent event) {
// we only care about down actions in this game.
try {
// history first
int hist = event.getHistorySize();
if (hist > 0) {
// add from oldest to newest
for (int i = 0; i < hist; i++) {
InputObject input = inputObjectPool.take();
input.useEventHistory(event, i);
gameThread.feedInput(input);
}
}
// current last
InputObject input = inputObjectPool.take();
input.useEvent(event);
gameThread.feedInput(input);
} catch (InterruptedException e) {
}
// don't allow more than 60 motion events per second
try {
Thread.sleep(16);
} catch (InterruptedException e) {
}
return true;
}

In the game thread:

private ArrayBlockingQueue<InputObject> inputQueue = new ArrayBlockingQueue<InputObject>(INPUT_QUEUE_SIZE);
private Object inputQueueMutex = new Object();

public void feedInput(InputObject input) {
synchronized(inputQueueMutex) {
try {
inputQueue.put(input);
} catch (InterruptedException e) {
Log.e(TAG, e.getMessage(), e);
}
}
}

private void processInput() {
synchronized(inputQueueMutex) {
ArrayBlockingQueue<InputObject> inputQueue = this.inputQueue;
while (!inputQueue.isEmpty()) {
try {
InputObject input = inputQueue.take();
if (input.eventType == InputObject.EVENT_TYPE_KEY) {
processKeyEvent(input);
} else if (input.eventType == InputObject.EVENT_TYPE_TOUCH) {
processMotionEvent(input);
}
input.returnToPool();
} catch (InterruptedException e) {
Log.e(TAG, e.getMessage(), e);
}
}
}
}

Then in your main loop, just call processInput().

This is what I use for all of my real-time games and it works very, very well. If you can't get it to work, spend some time looking at the concept and how the pools are used to reuse objects.

I hope this solves the mysterious Wait or Force Close problems for you. Even for games that I didn't have that problem on, I noticed that the input was handled better because it didn't skip over a few ticks waiting to get the synchronized block.

Enjoy!

43 Comments

Post a comment here or discuss this and other topics in the forums

Thanks

Great help. This truly is beautiful!

Thanks a billion for posting

Thanks a billion for posting this. I had this exact problem and your solution is elegant and clear. Kudos.

You just need to override

You just need to override onKeyDown and onKeyUp in a manner like onTouchEvent to support keyboard input, then the two process methods are game-specific. Implement them and have them process the touch and keyboard in a way that makes sense for your game.

processMotionEvent

hi... i used ur code in my application... everything works fine, bt
processKeyEvent and processMotionEvent are not working. It says that u need to create two methods of those names...

Any idea why this is happening?

Input Pipeline Follow Up

I have been using the input pipeline and it has been working great, until now. Lol, it's not the input pipeline fault. It's mine. What I'm trying to do is simply exit the game from a simple touch of the screen.
The only threads running at the time would be the UI thread and the Game Thread. I want the user to be able to touch a certain part of the screen and exit the game. I have been trying to do this by setting isRunning to false, when the screen is touched, then using activity.finish(). This worked until I add the input pipeline, I now get the following error message.

FATAL EXCEPTION Thread-8
java.lnag.NullPointerException
at com.testgame.GameThread.processMotionEvent(GameThread.java.322)
at com.testgame.GameThread.processInput(GameThread.java.294)
at com.testgame.GameThread.run(GameThread.java.208)
Force finishing activity com.testgame/.GameActivity

What do I need to do? and Also, would love a good reference on threads and mutex and all that goes with that.

Thanks again,

John

yes

yes

Thanks for the compliments!

Thanks for the compliments! For key events, you can do the same thing. Just put the keycode into the inputobject and set a type as a key event and the action as up or down.

The point is to spend as

The point is to spend as little time blocking as possible. Segmenting the locking to different discrete functions can help significantly with that.

Getting the Keyboard Input

First, thanks for all the great articles. It has really help. I trying to use your input pipline for a Space Invaders type game. The OnTouch for motion events using the input pipeline is working great, thanks. The problem is I'm not sure what the best way is to get keyEvents in the pipeline... like if the letter 'P' is pushed to pause the game or the D_PAD used to move or fire weapons. I see how the Motion events are handle in the Activity but I do not see how the KeyEvents are handled, if you would post a code example it would appreciated. I see on the thread end how they are suppose to get processed.

Thanks,

John

Clarificaiton

I'm a little confused. Is the point of this to synchronize the main activity and the input events separately?
I'm working on a game and sometimes the device just stops taking inputs, I think it may be the same problem you solve here. However, I don't see how logging the inputs and cycling through the log will avoid the issue, since the inputs are never processed in the first place? Am I misinterpreting the concept?

It depends on what you're

It depends on what you're doing. If you want discrete control, you'll want to track the ID and also add a little code to tell if the second pointer just came down or went up, as those events aren't sent explicitly.

Sorry, I had a brain fart

Sorry, I had a brain fart there. I meant the MotionEvent's pointer ID, not process ID. I was wondering if it was necessary to explicitly test for the pointer ID in case the user's got two or three fingers on the screen, but I've got the code running and it looks like that isn't necessary.

Thanks again for all the help.

You want to use your own

You want to use your own intermediary object because you can't queue an Android event - those are pooled and the values may change while in queue :)

I'm not sure what you mean about the multitouch gestures and process IDs. Care to elaborate?

Cheers - I'm glad you've found the articles useful. I hope to find some time to write more in the upcoming months.

Great article, have a couple of questions

First of all, thank you very much for posting these articles up. This web site is easily one of the best on the web for learning Android development.

I had a couple of questions. First, it looks like Android touch events are mapped to the custom InputObject class before being added/removed from the queue, passed around, etc. Is that for reasons of speed/efficiency (processing a lightweight class without all of the overhead of the API's event class)?

Second, I am guessing that if you are not interested in multi-touch gestures that you can simply eliminate any event with a process ID not equal to 0?

Thanks again for all of these great articles.

dose onTouchevent() in ui

dose onTouchevent() in ui thread?

Sleep for 32ms at the end of

Sleep for 32ms at the end of onTouchEvent() for Android 1.5 or else you will get a ridiculous number of events thrown at you.

many thx ur reply. i use

many thx ur reply. i use android 1.5 not on the real device.

What device/android version

What device/android version are you testing on?

pls help me

this is my code,pls help me check it , is any problem ?
the image fall down , but when i touch the screen or move on screen it's trun slowly. why? and how i do?
thx.

public class UIthread extends Activity {
/** Called when the activity is first created. */
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);

setContentView(new MyView(this));
}

class MyView extends SurfaceView implements SurfaceHolder.Callback , Runnable
{
SurfaceHolder holder;
Bitmap bitmap;
int i=0;
public MyView(Context context)
{
super(context);
setFocusable(false);
setClickable(false);
setFocusableInTouchMode(false);
holder=getHolder();
holder.addCallback(this);
bitmap=BitmapFactory.decodeResource(context.getResources(), R.drawable.icon);
}

@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width,
int height) {
// TODO Auto-generated method stub

}

@Override
public void surfaceCreated(SurfaceHolder holder) {
Thread t=new Thread(this);
t.start();
}

@Override
public void surfaceDestroyed(SurfaceHolder holder) {
// TODO Auto-generated method stub

}

@Override
public void run()
{
while(true)
{
Canvas canvas=holder.lockCanvas();
canvas.drawColor(Color.BLACK);
canvas.drawBitmap(bitmap, 100,i,null);
holder.unlockCanvasAndPost(canvas);
try {
Thread.sleep(10);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
i+=5;
if(i>480) i=-bitmap.getHeight();
}
}

}
}

I use a queue size of around

I use a queue size of around 30 and it never has to wait.

Wow, thanks!

Ei! Really thanks for this article. The onTouche Event problems were driving me crazy, they dropped the FPS of my game to the 50% or so. One only doubt: What should it be the optimal value for "INPUT_QUEUE_SIZE"?

Thanks for sharing

Thanks for your posts Robert ! keep them coming :)

I thought So :-)

I have had the thought that this could be the explanation, great that this is the case, makes the code brilliant - good work!

Thanks for the compliments!

Thanks for the compliments! Glad this is all useful.

It's because any event history is always a move. Ups and downs always come as their own events, history is specifically for move events.

Great Code this, got a question though...

Yep as said this is great, I'm using it in a game where touch actions where lost due to processing of physics. I have one question though, why do you deem all qued up touch events to be move events?

As seen here when you catch un in the touch event:

public void useEventHistory(MotionEvent event, int historyItem) {
eventType = EVENT_TYPE_TOUCH;
action = ACTION_TOUCH_MOVE;
.
.

BTW - keep up the good work, excellent!

/Schwartz

You can put it in the view if

You can put it in the view if you want.

Your main loop would go in the run() method of your thread. You call processInput() from there.

some questions about the code

I am a little bit confused about the following areas:

1) Is it possible to put the code for the activity in a Custom View class? My onTouchEvent is implemented in the View object as opposed to the Activity object

2) Also your tutorial mentions calling processInput in the main loop, what do you mean by that, in onCreate() on the Activity class or in the run() of the Thread class?

Again, thanks for all your help.

It will block but if you're

It will block but if you're smart you set it to a number high enough so that it would never need to block in practice. I've never had it block set to 20. That's 20 input events to process in the highest tick period, which, during lag, would be 100ms. There are never more than 20 input events in 100ms, so you can rule that out.

Android has touch processing issues, as shown but the numerous posts about the issue on the Android developer's group. Those account for most of the problems people have with lag.

My guess would be that the

My guess would be that the delay is caused by using an ArrayBlockingQueue. Once the queue is full, the UI thread is blocked until an InputObject has been processed and removed.

Fantastic job!

Hello Robert,
Just to say that I found your articles very useful, and that inspiring me to begin my own game.
I appreciate very much your generosity sharing your experience with the world.
Thank you very much!
Jose