Shadow/highlight (shortened to S/H) is, to put it roughly, a feature that lets you do 50% white and 50% black translucency to some limited extent. It's a pretty powerful feature, but it can be a bit hard to figure out how to make good use of it which has resulted in this feature going underused.
So here's an explanation of both how it works and some tips on how to make good use of it so you don't have to waste time figuring it out yourself.
Shadow/highlight is enabled by the same register that controls the screen resolution, so go check how to change screen resolution. Once enabled it'll "just work".
Simple example (change resolution as needed):
; The "_SH" part is the important one here SetGfxMode GFXMODE_320x224_SH
How it works
Shadow/highlight works per pixel. There are two things that affect the color of the pixel: first is whether a sprite is displayed, some colors have special meaning.
- Palette 3, color 14: pixel is brighter, sprite becomes transparent
- Palette 3, color 15: pixel is darker, sprite becomes transparent
- Otherwise: no change
Then all layers are put together (sprites, plane A, plane B) and their priority flag is used to check if pixel becomes darker or not (note: this includes transparent pixels in plane A and B, but not transparent pixels in sprites)
- If sprite color 14 (any palette): no change
- If any layer is high priority: no change
- Otherwise: pixel is darker
If both rules above end up being "pixel is darker" (i.e. sprite palette 3 color 15 on top of everything being low priority), then the pixels is still just 50% darker, not made even darker.
On top of that: brighter + darker gives a normal pixel, not some ugly muddled color (in other words: highlight from sprite undoes shadow from layers).
The fact that layer priority influences the outcome of S/H tends to make it hard to figure out a way to easily exploit the feature. The following setup would make for a good starting point, with the idea being that you build your effects on top of this:
- Plane A (foreground) full of high priority tiles
- Plane B (background) full of low priority tiles
- Sprites change priority as they move depending on whether they should show on top or below the foreground. You're probably going to need a "priority map" for this, indicating whether each chunk of map is "low" or "high".
Now for some ideas on what kind of effects you could use it for:
- Obviously, it can be used to make translucent sprites.
- You can use it so indoor passages are darker than outdoor areas, by using high priority tiles (even blank ones) outdoors then low priority tiles indoors. If you need high priority non-tile aligned shapes indoors, you can fill in the silhouette with sprites.
- It can be used to make shining light rays in an otherwise dark room, by filling the room with low priority tiles, then using transparent high priority tiles where the light should shine. This trick is used by Castlevania Bloodlines and Ranger-X.
- You could use it to make polarized glass, by making most of foreground high priority except where the glass is (which would be low priority transparent tiles). This trick can also be used to make translucent water (Bari Arm does this). Vectorman goes further and mixes this effect with the above one.
- One quirk is that color 14 of sprites never goes darker (even when everything is low priority). Instead of making it black (to work around the quirk), you could make it the color of some light that would remain bright even under darkness.