Introduction (for downloads, see the bottom of this page)
The name Open Source Music was chosen to encourage composers to not
only publish their music, but also the raw tracks so that people who
are interested can make their own mixes, if they want.
Tracker modules such as .xm, .it and .s3m provide a relatively well
documented way to store multi-track music files. There are very few
well-documented formats that support multi-track music, and that are
at the same time capable of combining pattern-based music with
recorded audio.
Problem
There is basically a single drawback in the .xm, .it and .s3m
formats: they do not store the data very efficiently. (The IT
format does have some sample compression built in, but quite primitive
compared to many recent compression algorithms). A multi track
file can easily grow to tens of megabytes in size, because of the
sheer size of the audio. This currently makes .xm, .it and .s3m
rather unsuitable for transporting multi-track files. Even a
multi-track mp3 file would be fairly heavy; however if we can combine
the benefits of mp3 with the benefits of module-stored music, we
might achieve an overall better solution for transporting multi-
track files.
Existing solutions
It would be possible to use a compression technology such as OGG
or MP3 to compress the audio of these modules. A format that does
this already exists: MO3 modules, by Ian Luck (www.un4seen.com).
I was pleasantly surprised that the idea was already implemented
and even used by one of my favorite mod composers (Skaven from
Future Crew). However at the current time, the author does not seem
to have intentions of opening up the source code of the format, and
as he told me, it is fairly complicated. Time for a better solution ;)
The OSM Format:
OSM should be portable, easy to understand and easy to implement
because like many programmers I don't have a lot of time on my
hands. Also, I want to do this without getting into much detail
about the module formats.
Features
The list of features, shamelessly stolen from Ian's MO3 format
description:
MP3/OGG compression
Lossless compression
Structure compression
Sample specific tuning
Text removal
Decompressable
Playable
Implementation strategy
Suppose we rip the samples from a it/xm/s3m module to file and
replace them by silence in the module. We then have a silent module
that is a very good candidate for traditional compressors. We now
compress the samples by some lossy or lossless algorithm, and write
a little text file containing info about the original size, position
and type of each sample in the silenced module.
When we compress the whole lot into a .tgz, we have a compressed
module that would allow (except for the text removal, big deal)
all features mentioned above. In addition:
- Easy to support compressong more module formats as long as samples
can be accurately pinpointed in the original files. The rest of each
module is preserved, so no module specific special features are
lost in the compression process, nor do we need to know about them.
- Easy to decompress to rebuild 'original' modules:
Possibly a rudimentary decompressor can be built with about 10 lines of
shell script and 8 lines of C code.
Someone that has tar or winzip, mp3 decoder, a module editor and half
a brain is probably able to reconstruct a module sounding very close
to the original without ever needing a dedicated module decompressor.
- It would be possible to enhance the format with post processing info
(mixing, equalization) for more serious studio work.
Building blocks
- To pinpoint samples in the modules:
I'll leave this task to a very slightly modified DUMB library
which supports S3M, IT and XM. Function it_read_sample_header
(itread.c) is called with sample offset for each sample that
is loaded. Thanks go to Ben Davis for this excellent library,
for pointing me where to look and for support and tips.
- libtar for archiving. For overall compression, I can probably choose
between libtar, zlib and the bzip2 library, credits and thanks go to
their respective authors.
- steal code from DUMB for IT sample decompression (as it was stolen by
Ben ;)) Simple delta decompression, I'll write the delta compressor
for file restauration myself. Promise.
- use existing mp3/flac/ogg libraries for compression, once again credit
where credit is due;
- Wrap the whole into a piece of bug-ridden code
- Inform people about the compressed format to gain support. Thanks already
to Ben for showing intentions to add OSM support to his library.
- Fame and glory.
Project Scope
The scope of this project depends on the stage it is in.
Stage 1
- First target is having a compressor/decompressor for xm/s3m/it archives.
In this stage, no additional information will be added to archives
other than the strictly needed to recover the "original" modules (the
recovered modules may sound very close to the original although not
exactly equal due to lossy compression. Frequently recompressing and
decompressing is not recommended, as it isn't in MP3 files.
Stage 2
- Allowing additional information, such as mixer info that is not contained
in the module files.
Stage 3
- Migrating modules to a text-based, fully open module format.
Thanks go to Ben Davis for the DUMB library, which takes care of the most complex part of locating the samples inside the module file. This library was slightly adapted to allow it to output the sample location info.
Requirements:
- perl and tar available in the search path
- depending on (de)compression types used, the following utilities available in the search path: flac, oggenc, oggdec, lame, mpg321
- This application is only tested on linux so far but should run fine without modifications on most platforms supporting perl and gcc.
Downloads:
Beta release: osmzip-0.2.0beta.tgz
This is a beta release. The current feature set is:
- Compresses and decompresses .xm modules and .s3m modules
- .ogg, .mp3 and lossless .flac compression
- testmode
Last development snapshot: download here.
The development snapshot differs from the pre-release in the following ways:
- None. The current development snapshot *is* 0.2.0beta.
Known bugs:
- Lossless .flac compression isn't completely lossless yet.
- Program assumes all requirements are available, if some of them are not met the decompressed modules might be silent or sound wrong
Coming soon:
- Support for .mod files
- Support for .it files
|