I’m working with Dave Humphrey my professor at Seneca College with a reference test tool he began for Processing.js. He started working on this tool because many of our tests require image-image comparisons. That is, our final rendered image should match identically to the same sketch produced by Processing. And doing that manually would be painstakingly slow.
The tool is comprised of a few parts, one of which takes a Processing sketch, renders it, and dumps out the raw pixel values of the canvas into a web page. This data can then be saved to a text file and added to the batch list of tests to run. Since he already had the script dumping out the result from 2D sketches, he asked me to develop the same thing for 3D sketches.
We are using WebGL to do all the 3D rendering for Pjs. We already ported over some 3D functions from Processing such as points, lines, box and sphere. My job was to take the following sketch and get the values from the framebuffer.
size(100,100,P3D);
background(33,66,99);
So given the above code, I needed to produce something such as this as our reference ‘image’:
//[100,100]33,66,99,255,33,66,99,255 .....
size(100,100,P3D);
background(33,66,99);
The first two values in size are the canvas dimensions and are followed by the series of values which would need to be extracted from the framebuffer (which also includes the alpha component). I understood what I needed to do and got to work.
Since we are using WebGL, I knew we had to use readPixels. I went to the WebGL spec which had the declaration of the function.
// spec:
WebGLArray readPixels(GLint x, GLint y,
GLsizei width, GLsizei height,
GLenum format, GLenum type)
raises(DOMException);
// example:
gl.readPixels(0, 0, 100, 100, gl.RGB, gl.UNSIGNED_BYTE);
The fist four arguments are straightforward. The format defines what data you want returned and the order. It can either be RGB, RGBA or ALPHA. The type argument specifies the type of the value returned. You can request things like UNSIGNED_SHORT_5_6_5 which defines per-component bit lengths, but the simplest is just passing in UNSIGNED_BYTE.
After a bit more reading I got down to hacking some code. I began working on this using Webkit, Safari’s nightly rendering engine. I wanted to create a simple test case, but I got stuck trying to do anything with the result of the call. Safari kept throwing an exception.
var buff = gl.readPixels(0, 0, 100, 100,
gl.RGB, gl.UNSIGNED_BYTE);
// Something with buff...
// Result of expression 'buff' [undefined] is
// not an object.
I struggled with this for some time. I kept thinking I needed to feed the return value of readPixels into a newly allocated WebGLUnsignedBufferArray. I couldn’t understand why buff was undefined. But eventually I tried requesting RGBA as the type and it actually returned something!
gl.readPixels(0, 0, 100, 100, gl.RGBA, gl.UNSIGNED_BYTE);
So I learned it wasn’t actually my fault, something was wrong with Webkit. Unfortunately, I was still under the impression I had to allocate a special WebGL buffer. The specification states “The specific subclass of WebGLArray returned depends on the passed type.” So if you pass in UNSIGNED_BYTE, you’ll get a WebGLUnsignedByteArray. But eventually I found out I could just use a regular JavaScript variable to hold the return type. No special allocation necessary.
Once I figured those two things out, I was able to make some progress. I assigned the return value from readPixels to a variable and queried the type. It was a [object WebGLUnsignedByteArray] as expected. I assume the subscript operator is defined for this object since I was able to get the pixel values using []. The object does have a this defined (which likely does the exact same thing):
getter GLubyte get(in unsigned long index);
So I was able to iterate over the elements and my code progressed to this:
var buff = gl.readPixels(0, 0, width, height,
gl.RGBA, gl.UNSIGNED_BYTE );
var pixels = [];
for(var i = 0; i < buff.length; i++){
pixels[i] = buff[i];
}
Finished! Or so I thought. When trying to run the same code in Minefield my complete output was:
//[100,100]
size(100,100,P3D);
background(33,66,99);
Where were my pixels? It didn’t take me long to figure my for loop wasn’t running. After playing with it for a while, I just resorted to querying the type and found it to be a [object Object]. Not exactly a WebGLUnsignedByteArray is it? Just a regular JavaScript object. To get the sweet goodies (property names) I wrote a simple for-in loop:
for(var i in buff) {
alert(i);
}
This gave me the properties width, height and data. So a simple assignment was all I needed to get the data for this case!
pixels = buff['data'];
DONE!
Since I knew Chromium’s XHR bug breaks our Pjs code and Opera doesn’t yet support WebGL, I only had two browsers to consider, hence the extremely basic regex. But if you’re using readPixels, feel free to take whatever you need from my code and adapt it to your needs. Be it support for Chromium, browsers for mobile devices or whatever.
var agent = navigator.userAgent;
var isSafari = agent.match(/safari/i);
var context = canvas.getContext("experimental-webgl");
// Safari returns undefined if format RGB is requested
var buff = gl.readPixels(0, 0, width, height,
gl.RGBA, gl.UNSIGNED_BYTE );
var pixels = [];
if(isSafari) {
for(var i = 0; i < buff.length; i++){
pixels[i] = buff[i];
}
}
// Minefield
else if(!isSafari) {
pixels = buff['data'];
}
I hope the return value of readPixels is standardized before the developers at Mozilla, Apple, Google, Microsoft, etc. release their complete implementations of their WebGL-enabled browsers, but I have a feeling they probably will.
Filed under:
Open Source,
Processing.js,
WebGL 
Planet Feed (Atom)