Hell, I don't community enough. I hang out on the IRC, but there's just something inconvenient about going from blog to blog, which has, without my noticing, been made much less inconvenient lately.

Anyway, hi.

I was delving back through the forgotten pages of my late childhood, thanks in huge part to the Harmony of a Hunter album I had playing while I went about my usual coding routine of poking at the piece of code that doesn't work, recompiling it occasionally to see if it wants to just start working randomly. It reminded me of the explosive excitement erupting through me as I waited for the new Metroid installment, Metroid Prime 3: Corruption, for which they had bothered to compose a really awesome new intro tune which did just as good a job of capturing the Metroid feel as the original. I remember watching the spiderball boost jumping intro, and thinking of how I'd do that in Game Maker, because at the time, GM was the shit. I must have had some incredible imagination back then, because now, when I try to think of ways to do that in Game Maker, I no longer possess a neuron which does not immediately return the string, "YOU CAN'T LOL; GIVE UP."

It's amazing how many memories a little tune can have attached. Music is actually a pretty integral piece of my life, despite my not having any musical talent and almost no musical appreciation. That brings me to my next point: we need a new sequencing format. I've been meaning to mention this for a while.

I'm sure a lot of people around here (or who used to be around here) have played or even created MOD/XM/S3M/IT sequences; they're the (generally) chiptooney arrangements that store all their samples and such in the audio file, then sequence them in real time as they play. Back in the day, all video game music worked that way, whether the sequencing program shipped with the console or as part of the game (or indeed, as part of the sound file).

If you know a thing or two about sequenced audio, you'd know that XM, S3M, MOD, IT… all these sequence formats consist of <INSTRUMENTS/SAMPLES><PATTERNS><TRACKS>, where samples are notes of an instrument stored in one way or another, which are then arranged into patterns, or small melodies, which are then played in tracks. If you know three or four things, you might also know that this isn't how console music works. NSF, GSF, PSF, VGM… all these sound formats instead are laid out as follows: <PLAYER PROGRAM>.

So yeah, there isn't much to video game music formats; the entirety of the format is a specification for a little ARM program that generates the song as it is run. Good luck converting that to XM. An AI could do it; a patient human could do it. In some cases, a program can do it, as, historically speaking, there is a disjunct between artists and programmers which has led to the need for artists to have a simple, understandable way of sequencing notes that doesn't involve ARM assembly. What I'm saying is that many of these sound files use the same player format as a base, creating a sort of sub-format which is theoretically simple to convert to a regular sequence file. However, this isn't something I'd recommend doing; someone informed me that it has been attempted, but he couldn't name a program that did it (or he named one that I was unable to find and have since forgotten about).

The point is, these formats are not trivial to convert, and impossible to convert in the general case. There are a bazillion different sequence formats, and the players that exist for them aren't very portable and have all sorts of nasty licensing problems. Wouldn't it be great if we had a sound format that was backward-compatible with all of them?

It really isn't that hard to accomplish, when you consider the implications of supporting conversion from a sound format that is a damn ARM program. The solution? Support basic scripting, and compile it to byte code. Like Java byte code, only hopefully not anything like Java byte code, because it gives programmers the heebies.

Maybe you're not convinced. Consider the original Super Mario Bros music. The samples were a square wave multiplied by a sine wave. With time, samples became more complicated, eventually becoming small wave files, but the concept was the same, and the point stands: You can make extremely memorable music using a wave function that fits in a kilobyte and uses nothing but sin() and some loops.

Now, a player program alone is not sufficient. There are obvious problems with it, most of which I've already mentioned. Even with a good GUI to generate sample programs for you, a program for each wav file seems like overkill. And it is. The bytecode would need to be emulated, or at best, JIT-compiled, but either way, you have overhead from safety bounds checking. So we'd want to support the basic framework of samples, patterns, and tracks, too.

So here's the idea: Two sample types: file, and function. Sample functions can be played for any length, and are probed as any stream. Sample files are just included and transformed by note pitch + effect, where effect can be a built-in preset, as are available in XM/S3M/IT, or—you guessed it—another program! So you can have a program whose only purpose is to transform input given source and destination pitch(es), for the purpose of creating more complicated, ear-pleasing audio effects.

Now, sprinkle on an optional compositing program and, look at that: you have an entire audio pipeline. So you have samples which can be files or programs, which can be modulated to the correct pitch(es) by the default sequencing program or your own program, and then composited into your song by the default compositing program (multiply by volume, sum as floats, cast to bytes), or your own program. And of course, all pattern generation can be run in parallel.

The actual programming language would be imperative and support such functions as sin(), tan(), oggdecode(), flacdecode(), and random()/random_start()/random_next().

Sounds cool, but what have we accomplished?

You can convert to this (Turing complete!) audio format from *ANY* audio format.

I.e.: You can convert to this (Turing complete!) audio format from *ANY* audio format.

To clarify: You can convert to this (Turing complete!) audio format from *ANY* audio format.

To convert from MIDI:

1. Grab all relevant samples from the sound font, adding them as file-based samples to the new file.

2. Convert each instrument track to a pattern + track

3. ???????

4. MICROSOFT LAWSUIT

To convert from XM:

1. Figure out which way to hold the XM specification

2. Attempt to read and understand the XM specification

3. Fail; repeat from (1).

4. ???????

5. Copy over samples as file samples.

6. Copy over patterns.

7. Translate the effect indices (see step 2)

8. PROFIT

To convert from NSF:

1. Binary translation. Yes, you.

2. Place translated binary in a new function sample.

3. Create single pattern containing single note of infinite length

4. Create track containing that pattern

To convert from any other existing format:

1. Decode to WAV

2. Encode to OGG

3. oggdecode() in function sample using original sample as raw data

4. Single sample playing decoded ogg

5. Single track with single pattern playing that sample

As an interesting side-effect, you could make this continue forever: http://www.wimp.com/pisounds/

The more you know ♪

TL;DR: Hi.