Hi!
Got a bunch of sprites I'd like to tinker with palette swapping and was wondering if anyone knew of any good tools for that sort of thing?
In the past, I've just sort of done things by hand in the Gimp with a combination of 'select by color' followed by 'flood fill', but have enough images that I'd kind of like something a bit quicker this time.
The idea is simple, take an image with Palette A and create a new image with Palette A', where there is some way to manually specificy a mapping from each color in Palette A to a color in Palette A'.
Thinking specifically of something that allows you to see and set each color mapping individually, and let's you see an A and B version of the image, so you can easily see the mapping and how it affects the image.
Something like a panel showing:
Color A -> Color A'
Color B -> Color B'
Color C -> Color C'
....
plus a panel with: Image -> Image'
Did some googling and came up empty. I don't know if this is something people normally just do within their favorite editor, or if maybe this entire approach is wrong, but I figured I should ask on here before I start down the long and winding path of rolling my own tool for the job.
@William.Thompsonj: If you've a minute to explain it and the rest of the thread can bear some programmer talk for a bit, I'd love to hear more about what you meant by 'C# does a lot of boxing of variables and crazy nonsense.' Have had a few run ins with C#'s memory management systems myself...
@withthelove
Certainly! Things I ran into that caused serious performance issues:
- Default object comparison functions Equals() and GetHashCode()
- Default behavior of HashSet and Dictionary data types and their underlying need for IEquatable and/or IComparable interfaces to behave nicely with user defined data types
- Lack of (or incorrect) Unity issues with == and != operators
- LINQ performs terrible with large datasets without significant effort
I'll try to explain each point in plain programmer language so please don't be upset if you're not a programmer.
Point 1: Default object comparison functions Equals() and GetHashCode()
C# behaves differently with objects (reference type) than with structs (value type). If C# compares two objects (reference type) it checks to see if they refer to the same object (very fast). Simple reference check, nothing more.
Value types (structs) are different. C# uses reflection to find member variables and uses all of them to compute a hash based (think structInstance.GetHashCode()) to compare equality. If your struct stores 1 or 1000 member values they are all computed to get the GetHashcode() before they are compared (each member has struct.member.GetHashCode() called on it). Reflection is not super expensive but it is used every time it compares two structs to determine all the member variables. If you have (in my case) tens of thousands of structs to compare that gets very expensive very quick. If your struct uses an IEquatable interface then it implements the proper functions needed to compare two of the same structs. Example:
var a = new Color32(32, 64, 128, 255);
var b = new Color32(32, 64, 128, 254);
if(a == b)
{
// do something...
}
The Color32 struct in Unity is defined in UnityEngine namespace (RGBA struct with some methods). Without the IEquatable interface implemented, C# doesn't natively know how to compare two Color32 structs easily. The C# runtime boxes each struct instance (makes them objects), uses reflection to figure out all member variables, and then computes the hash values of each struct member variable to compute the final overall hash for the struct instance. This can get pretty expensive if your struct stores other structs because it causes recursion.
By default, GetHashCode() does some significant/expensive stuff to generates a semi-unique integer. If you override/define your own GetHashCode() it can be very efficient. Do this. Also, override and define your own Equals() function for every class/struct; implement the IEquatable interface so it plays nice with things like HashSet and Dictionary.
Point 2: Behavior of HashSet and Dictionary
HashSet only allows unique objects/structs; it uses GetHashCode() to figure out what's unique. That said, it (really desperately) needs the above things in Point 1 to run efficiently. A dictionary datatype needs the IComparable interface implemented because it uses those things to produce the highly optimized (nearly) O(1) search behavior. Hash sets and dictionaries are highly optimized in nearly every language so that's why I'm using them in C#. To get the best performance out of a dictionary it needs to store things that implement the IComparable interface because it internally sorts keys based on <, ==, and > operators to get the highest yield performance. HashSets use the GetHashCode() function on everything they store to ensure it gets unique items.
Point 3: Unity issues with == and != operators
Unity is notorious for lack of (correct) support when it comes to comparison of structs and objects (below is just one example, there are many).
https://blogs.unity3d.com/2014/05/16/custom-operator-should-we-keep-it/
Unity is designed to be fast for small collections of game objects, not for making tools. It's not designed for created enterprise applications or robust tools that deal with large amounts of data. It does a lot of things that are very safe but also very inefficient to ensure nothing breaks. I had to learn about many of these features (like the onGUI() function) and learn how to work around them. It's been a learning curve for sure. These are Unity specific problems.
Point 4: Microsoft .Net LINQ
Oh where to begin. We could start here:
https://stackoverflow.com/questions/1576679/reason-not-to-use-linq
https://softwareengineering.stackexchange.com/questions/160368/why-is-th...
https://medium.com/@raphaelyoshiga/linq-performance-dangers-6e9757607884
https://www.reddit.com/r/csharp/comments/6h2mbj/what_is_the_ugliest_linq...
https://dataflowe.wordpress.com/2017/01/24/the-disappointment-of-queryin...
Or we can talk about it briefly in relation to what I learned during this project. Microsoft LINQ is a fantastic tool for small datasets and it gives you so much freedom at the expense of memory and CPU cycles. First I designed with a lot of LINQ functions because hey, it's fast, it's easy, and it's already there. Then, I realized it created a lot of weird bottlenecks I didn't need so I got rid of it everywhere. It took a lot of work to remove it but my stuff runs much faster and uses much less memory now. Originally I was barely able to load a 512x512 PNG image without hitting performance and memory issues, now I'm trying to optimise a 10 megapixel image translation without issue (without LINQ). In my brief experience with LINQ it's great for things you know are small and won't grow. I will never use LINQ for anything larger than say a peanut or a baseball (figuratively speaking of course). If I do it'll be because there's no other option (there always is) or because it's the best tool for the job (it never is).
LINQ? What are you using database stuff for?
Just curious.
Totally know what you are talking about with Equals() and IComparable.
You make a good point about 'GetHashCode()' returning only 'semi unique' integers. I'm guessing whatever hashing algo they use is pretty good and not likely to generate many collisions, but all the same, if you really want to be sure your Equals() results are returning true equality, then best to write your own comparer. So just another reason to implement your own Equals() tests, etc.
Interestingly, I find myself almost never using structs in C#, which is odd because I have a lot of c/c++ in my past where structs are definitely a way of life. I guess I read early on that passing structs into functions was expensive and have just stuck with classes ever since. I do sometimes miss the ease of copying structs but once you get in the habit of writing Clone() and Copy() routines for your classes it's not a big deal.
Is there anything special your using stucts for? or something you find useful about them vs. classes? Curious to know if I'm missing out! :)
@withthelove
I was using functions like Distinct() and ToList() without realizing what they did in the background. Also tried tuning it until I did unit tests and realized how much memory and cpu they cost.
https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concept...
My default thing is to make stuff into classes too but Unity uses structs to describe color (both UnityEngine.Color and UnityEngine.Color32). I didn't need to copy that convention but kinda did without thinking much. I assumed structures didn't have the same overhead as objects and just went with it. It would seem in C# that's a dangerous assumption.
https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/choos...
@withthelove
Almost forgot, I had to use extension methods to define Equals() for the Unity Color32 struct because it was failing through to using default object Equals(). That was another significant issue that had to be discovered. Unity gives an array of Color32 structures that describe image data and expects a similar array when setting multiple pixels at once.
https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes...
I wrote an Aseprite extension for colour remapping.
Posted about it here: https://community.aseprite.org/t/extension-recolouriser/12011
Red warrior needs caffeine badly.
I'm back to this and updated https://github.com/basxto/loadgpl. I now support subpalettes and shared colors:
"./loadgpl.py src/tileset/zxtileset_cblack.png src/palette/dmg_basic.gpl tileset/zxtileset_cblack_dmg.png -u3 -s2"
That would load the palette into the third subpalette and share the first two colors of all palettes. Default colors per subpalette is 4.
It would load the firts two colors of the given palette into the first two colors of the PNG palette and then the other two into 7th and 8th (3rd and 4th of the third subpalette). Every subpalette has two unique colors in this case.
The origin of shared colors is NES, which shares the first color of all palettes, but also the first sprite color being transparent on NES and Game Boy. Doing it like this instead of duplicating the color should improve PNG compression a bit, since especially sprites have sometimes complete rows filled with transparent.
I'm doing a lot with palettes right now since I try to create a tileset that works with black and white (Arduboy), four color grayscale (Game Boy), two colors per 8x8 (ZX Spectrum) and four colors per 8x8 (NES, MSX). That makes palettes even more complicated since now I'll also have duplicate subpalettes.
Next will probably be a tool to modify indexed PNGs:
Maybe even more simple changes like tile rearrangement, sprite frame duplication and stuff like that. Afaik there isn't really any tool that can do such stuff. Indexed images are quite niche and you'd usually do such stuff with imagemagick, but imagemagick always destroys palettes.
Edit:
A first version can check the subpalettes with specified tile dimensions, colors per subpalette and amount of shared colors between subpalettes.
It wil print stuff like "Tile 1x5 has 25 pixel(s) with colors from wrong subpalette(s)!"
https://codeberg.org/basxto/palgic
Edit2:
Now it can also check if a metatile does not use too many colors. (RGB not palette indices)
NES can only have palettes per 16x16 metatile and not for every 8x8 tile ... unless a special chip is used. So far I ignored that restriction. I'm not sure if this could have any general use, since I mostly need it because my palette is more for switching systems. Maybe for checking that colors for metatiles don't get too out of hand.
Pages