Please note: Leopard Only! I've been using QC exclusively on Leopard, and have used Leopard-only patches liberally. I doubt that any of the files below will work at all on Tiger. Also, some may run slowly on older hardware, or not at all if you're on some shitty intel GPU. Sorry!
16bit colour doesn't work on Nvidia!
This is pretty important if you want to use 16bit colour modes: they don't work on Nvidia cards. At all. For the record, only 8bit colour will work on intel GPUs as they only support 24bit colour depth.
You can use 16/32 bit colour in QC as well as 8bit (and to be clear I mean 'per channel' colour, i.e. 8bit RGBA = 32bit, 32bit float RGBA = 128bit total). Colour depth is an option in the settings of the Render in Image and Accumulator patches, and any 'upstream' core image based patches from a render in image or accumulator will inherit that colour depth. I.e. if you generate an image with a CI filter, and do a load of processing, then render inside a Render in Image, if you set the renderer to 32bit colour the whole chain will then get rendered in 32bit.
The problem though is with 16bit - 8 or 32bit will both work fine on ATi and Nvidia. With 16bit, there are 2 modes: 16bit integer (RGBA16), or 16bit half-float (RGBAh). ATI supports both in recent chips, but Nvidia only supports the half-float mode. It seems QC specifies the 16bit integer mode, so on Nvidia hardware you'll get a black screen and a ton of console messages about failing to create a suitable context.
I don't think there's a 'simple' workaround for this - if you really need 16bit (which I do! 32bit is too slow and 8bit lacks enough precision) then you're stuck with ATI. It might also be possible to create a plugin that sets an RGBAh context, takes an input image in that pixel format and outputs it again in a RGBA or RGBA16 - that would make the core image processing work in 16bit float so it would work on any hardware (QC supports it internally). I'll work on that soon and put the plugin here if it works out.
Vertex samplers don't work... at all
If you're doing GLSL, you might want to use vertex samplers (sampling from a texture inside the vertex shader, so you can do displacement based on texture data etc). Well, it doesn't work - if you try, it'll just crash QC.
I spent a LONG time investigating this, and it's caused by colorspace conversion. Basically, if your texture has colorspace information it will crash QC. The only patches that don't output colorspaced textures are plasma and fire unfortunately - either of these will work with a vertex sampler, but if you process them at all they'll get colorspaced and crash again. You can see colorspace info with the image debugging patch, or by enabling extended tooltips (in the hidden prefs - hold alt while opening the prefs to get them).
There's no way to remove colorspace info inside QC unfortunately, but if you're wanting to use vertex samplers email me, I've written a plugin to handle it.
The kind of thing you use vertex shaders for:
Premultiply your alpha!
Not so much of a tutorial as a word of caution. This applies to Core Image filters mostly, but it's worth reading if you're doing GLSL too.
"Correct" alpha blending should usually be done by pre-multiplying the colours by the alpha, so that colours are correct when transparent. Premultiplying can also help speed up GLSL shaders, as you don't need to worry about transparency so much in the rest of the shader if it's premultiplied at the start.
Anyway, the main reason for bringing this up, is that Core Image expects premultiplied alpha, and a lot of stuff in QC will render it automatically. If you don't know that, and you start writing filters with transparency handling, you'll get a headache trying to figure out why your colours are all wrong (I found this out the hard way ;). There are two commands for handling it in CI: premultiply() and unpremultiply(). They're the first thing to check if you're not getting the expected results from anything with transparency.
Apple's explaination of it: http://developer.apple.com/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_concepts/chapter_2_section_8.html
I've made a quick example file showing the difference (and the problem!) that premultiplication makes.
A gradient is rendered over each side of the picture - on the left, it's just rendered over the top. When alpha is zero, it fades correctly and becomes completely transparent, as expected. On the right, the alpha is applied to the pixels in a Core Image filter. At alpha 0, it becomes 'transparent' but retains it's colour, which is wrong. At alpha 0.5 (like the screenshot), the gradient is semi-transparent, but the unpremultiplied version has way too much colour. The colours should fade towards black as alpha descreases, which is what the premultiply does. Tick the 'fix' button to apply premultiplication, and it fixes itself - see the CI filter to see how it works.
1988
This is probably the best place to start. It's the complete .qtz file + assets for the '1988' part of my 8808 music disk. All the files are in the zip, and the .qtz file is nicely organised and fully commented so you can see how everything works.
Below are a few extracts from my blog on mac.scene.org with some useful info. I've pasted them unedited (I'll do some tidying later), so you'll have to excuse the irrelevant bits.
Sunday, February 10, 2008
I've been playing with text effects during the last week, among other things, and built a custom font generator (well, kind of) and a scrolling mechanism.
It's easy to create text in quartz composer: just use a string with image patch, type in the text, tell it which font to use, done. The problem is that it's fixed - you can only use the built in fonts, and the whole thing gets rendered to one big image so it's not possible to do say a sine scroll. Here's how I got around it:
Font generator
It's easy to process a string image with the standard image processing tools, to add colour, depth, shadows etc. So, I decided to generate an image for each letter. You need to be able to get the image for a certain letter easily, so it's best to store the images in a structure (basically an array). It'd make the most sense to use an associative array, with the character as key and the image as data, but there's a catch - QC supports it, but javascript doesn't seem to support associative arrays! The lack of a 'make structure' or 'merge structures' patch is a bit of a shortcoming.
There's a make structure patch available on kineme.net that would do the job, but I don't want custom patches here as it would mean anyone wanting to view my stuff needs them installing.
The solution I came up with was to create a structure of character images only, but to order them by ascii code. It's then trivial to select the correct image by ascii code with a little javascript.
The font generator is available here (it's commented so it's clear how it works)
Scrolling
This is a bit easier - just use the string components patch to divide the string into characters, and feed the resulting structure into an iterator. The iterator just renders each letter to an image, which is rendered as a sprite. Use LFOs, counters or whatever else to animate the sprites.
A scrolling demo is available here
Thursday, January 31, 2008
So, I've started work on a demo, and I'm using quartz composer as a base. It's MUCH improved in leopard, I recommend taking a look if you haven't already. The whole app is much more usable and useful, and it's now possible (or at least much easier) to write custom patches, which is pretty essential for serious demo work. It's part of the dev tools, so install those off the DVD if you want to play with it. If you just want to play the files, they'll play in safari or quicktime (you can embed them in web pages too). Anything that needs custom patches won't work like that though, you need to create an application, include the patch as a resource and compile.
Anyway, I'll make some notes on what i've discovered here, it'll be good for reference. First, a few notes on bugs in QC:
Be very careful with the javascript patch. It actually does "live coding" - even while you're typing, if the parser can parse the script, it's running. Great for experimenting and seeing what you're doing, but bad if you try a while(..) loop or anything like that, because it's just too easy to write an infinite loop and freeze the whole app before you've finished typing. Stop the playback before trying anything like that, and it's fine.
There's some problem with atan in the math expression patch. Not tried it yet, but I hear it's there.
There's also a problem with connecting a moving image output to the 2d input of a glsl patch (not tried this either but I think it causes a kernel panic). I guess using a 2d image instead of a CI image would work.
Check out kineme.net for lots of custom patches, including some GL stuff, texture generation, midi, etc. It's also a good spot to get help, as there are a few active developers. There's also an xcode template for making custom patches.
A couple of compositions I've done so far (these might be useful for learning how it works more than anything else, they need leopard, no custom patches):
Infinite (but ugly) bobs. [edit - replaced with a better version :) ] This has a moving sprite, accumulators, multiplexing, timelines, and a nice structure. If it doesn't run smoothly, make the window smaller and restart it. It's ugly because I didn't bother to make a proper texture ;)
Motion blur without using the accumulator. This actually abuses a bug to get a motion blur effect, so it might stop working with a future update. You can do the same effect with the accumulator, but it wouldn't be as easy or fast.
Site and all contents ©2008 Chris Wood (except where stated).