The Magic Matte Fixer

Surprise, another making of TAGAP 4 blog post, huh?

Here's something I remembered I promised to cover but all honesty just forgot. Whilst I was cleaning my hard drives and moving stuff over to the new dev computer I obviously came across the scripts and remembered I promised to share these – better late than never!

Introduction to the Matte Problem

A 32 bit image with transparency loaded into Photoshop (RGBA, RA, GA, BA).

Every 2D image – sprite or texture – in modern graphics is either 24 bits or 32 bits per pixel. What this means is that there are either three or four channels of colour per pixel; red, green, blue and – in case of 32 bits – alpha. Alpha is used in hardware rendering for transparency, where 0 is completely invisible and 255 is completely solid.

The actual colours with transparency a separate alpha channel (RGB, R, G, B, A).

With me so far? Good. The problem with working on 32 bit graphics for game rendering is the matte. Matte is the contents of RGB channels when 'alpha' isn't fully solid. This doesn't pose much of a problem in other forms of graphic design, but in games it is a huge deal.

Almost every graphics editing software handles with matte in their own way – some messing it more than others. In an ideal world all you'd have to do is hit 'save as' and the RGB would remain untouched whilst the transparency would be saved to alpha. But that isn't the case. At all.

Here is an example of what happens to the above image when saved using the default options Photoshop (CS6) gives you.

Channels using quick mask (RGB, R, G, B, A).

Channels using 'remove white matte' option (RGB, R, G, B, A).

Not only do the default tools given to you screw up the contents of RGB of semi-transparent pixels, they leave matte as single solid colour. The visible RGB being screwed is obviously bad, but when it comes to scaled 32 bit graphics, so is the solid colour matte (especially white). When the image is scaled, the colour of the matte bleeds slightly into the visible area, leaving an outline for the sprite.

Scaled sprite in TAGAP Engine, saved using quick mask. Notice the messed up matte.

This is garbage; it's wholly inaccurate and looks like a piss-poorly transposed green screen.

What to do, then?

The 13 Step Solution

Now, there is a way to fix this. It takes only 13 steps to do it! Obviously this should be just a 'save as' function, but since current-day Adobe can't program their way out of a wet paper bag unless it involves microtransactions and a cloud, we have this:

1. Create a duplicate of the current history state of the image.

2. Merge all visible layers.

3. Select the transparency of the new merged layer.

4. Add a mask to the layer. It is created from the selection of the previous step.

5. Switch to channel view of the image and choose to view just the layer mask.

6. 'Select all' and copy the layer mask.

7. Back out to layer view and delete the layer mask.

8. Solidify the merged layer. By this I mean fill the matte. For the fastest (and most often best) results, I recommend using Flaming Pearl's Solidify plug-ins (specifically Solidify B). It is available as freeware on their site, as part of the 'Freebies' bundle.

   What the plug-in does is fill the matte. If pixel has transparency larger than 0, the RGB is used. If the pixel is transparent, RGB is filled with the nearest non-transparent colour. This creates a smooth matte and completely eliminates the visible edge when scaling the sprite.

   Alternatively you can do this by hand – or just create a solid colour layer. Under the merged one as background. The first takes tons of time and the second can give you undesired pixels near the sprite edges, but they are still options.

9. Flatten the image.

10. Add an alpha mask to the flattened image. Make sure there is no selection before you do this.

11. Swap to channel view and choose to view the alpha mask only. It should be all black.

12. Make sure you don't have a selection active and paste in the previously copied layer mask.

13. NOW you can finally save the image.
     

The optimal matte (RGB, R, G, B, A).

Scaled sprite in TAGAP Engine, with the correct matte.

This creates the best results, but it is bloody time consuming when you have to do it for every damn sprite. For example, Pablo's TAGAP 4 entity alone is 42 separate sprite frames (not counting the masks). That means there would be whopping 546 steps to just save them all! If you'd do that, you could feel yourself age as you go through the steps over and over again.

There has to be a better way!

The Better Way

So me being a nerd made a script to automate it. Despite its modern failings, one good thing I can say about Photoshop is that they allow scripting for project automation. That said, I've developed a sneaking suspicion this is these days used as an excuse for not being arsed to do any actual QoL features.

What this script does is that it goes through the first 12 steps from above (adding one extra one just for compatibility). It stops short of the 'save as' so you can verify the matte looks good before you save it.

Download

Running the script

There are three different ways to run the script:

1. Browse for the file: File→ Scripts→ Browse

   And select the script file.

2. Add the file to the script list:

   Copy the script to the Photoshop's script folder: <photoshop install directory>\Presets\Scripts

   The script becomes available in the script list: File→ Scripts→ Penguin DT CS6 Solid Matte

   If you use CC (my condolences), note that by every major update the installation folder changes, forcing you to copy the scripts and plug-ins to the new folder.

3. Keyboard shortcut:

   First do 'Option 2' to get the script added to the script list. Then:

   • Go to: Edit→ Keyboard shortcuts.
   • Select 'Application menus'.
   • From, navigate to: File→ Scripts→ Penguin DT CS6 Solid Matte.
   • Add a shortcut for it.

I recommend option 3. I myself had this script bound to F1. Instead of doing the '13 steps', I just punched F1 and then Ctrl + S. That's it.

Closing words

I hope you find this script file useful. Though I wrote it for TAGAP, I'd image it works splendid for all kinds of 2D art asset generation where 32 bit images are involved. It was specifically done for TGA files, but it could work on other formats too (I haven't tested it).

No need to credit me for using it or anything – just have fun. Feel free to share and edit it further.