Developer diary: plans and progress reports.

Bump Sunday 1st July 2012

Boxer 1.3.1 is out now at a browser near you! Release notes are here: I’m afraid this update comprises boring old bugfixes and minor tweaks, with no new features to delight and astound.

1.3.1 is mainly intended as an end-of-life update for users still stuck on Leopard; bye guys! Boxer 1.4 will be along later this month, with a brand-new shader-based renderer and retina display support, ready for OS X 10.8 – and you’ll need Snow Leopard or newer to run it. (Yes, for really reals this time.)

This is why we can't have nice things

It's been a long time since I last talked about putting Boxer in the Mac App Store. The landscape has changed significantly since then, and Apple’s current policies mean that Boxer’s probably not coming to the App Store – not until the development climate changes, for the worse or for the better.

The main thing is that as of the start of June, all apps submitted to the App Store must now run in a restrictive sandbox environment that curtails the kinds of things an app can do and removes access to certain OS features altogether.

If you follow other Mac developers, you’ve likely read a lot of rants about the ramifications of the sandbox and the kind of things it prevents apps from doing. Here are all the ways (that I know of) that the App Store would affect Boxer:

App Store apps can only access files that are within the app’s own sandboxed filesystem (which is hidden from the user) or to which the user has implicitly granted them access: this by either opening files directly with the application, or by choosing them from a file-picker, or by drag-dropping them into the application.

This restriction would mean:

App Store apps can no longer open other apps directly, or talk to them via Applescript. This restriction would mean:

App Store apps cannot intercept and handle system-level key events. This restriction would mean:

App Store apps are not permitted to create or download other apps. This restriction would thwart my future plans to make gameboxes into first-class applications of their own.

This proviso is clearly intended to keep out alternative app stores like Steam and Bodega.

App Store apps are not permitted to download or support additional executable code, i.e. plugins. I have no plans to offer traditional plugins for Boxer, but it's debatable as to what actually constitutes a plugin: e.g. whether MT-32 ROMs or additional OpenGL shaders would be counted as plugins by Apple’s fickle review staff.

A case can be made that the Mac App Store terms of use are incompatible with the Grand Ole GPL, which opens Boxer up to being pulled from the App Store on spurious legal grounds. This threat has already been realised on the iOS App Store, where some GPL-licensed apps have been unceremoniously pulled after DMCA complaints from parties with axes to grind.

Compared to some apps, Boxer gets off lightly, and none of these restrictions on their own are dealbreakers for running DOS games: but they would put a dent in Boxer’s ease-of-use, and curtail future development angles, without offering me anything in return besides the increased exposure. Given that I don’t make any money off Boxer, the exposure would bring me no benefit (beyond basking in the glory of more happy users) and it would come with a much bigger support burden attached. Apple's mercurial policy changes would also leave Boxer at the risk of losing further features just to stay in the store once it’s there, or of being pulled from the store altogether at the whims of other people.

Closing remarks

It’s very late in the lifespan of an operating system to attempt to lock it down like this, and I’ve heard it likened to putting the genie back into the bottle. Some apps are simply unable to function within the sandbox, and so there has been a miniature exodus of app developers from the App Store since the new rules went into effect. Some high-quality apps, ironically including apps that Apple themselves have promoted, are now laying fallow in the App Store without the opportunity to submit new updates, and many other popular apps will now never be able to join the App Store. It’s possible that Apple will expand the sandbox API at a later point to let some of them back in, but I don’t have my hopes up.

Fortunately Apple seem to be in no hurry to remove your Mac’s ability to run well-behaved software from outside the App Store; they’ve made moves in OS X 10.8 to codify a distinction between unapproved apps that are made by any old rabble, and apps by trusted developers that just could not be distributed in the App Store for whatever reason. This indicates they recognise the ecosystem outside their App Store is still vital to the platform; let’s hope they keep thinking that way.

So there we go! For now Boxer will stay out of the App Store: until Apple expands the range of things possible in the App Store sandbox, or until Boxer evolves to a point where those restrictions are no longer relevant, or until Apple force my hand by preventing 3rd-party apps from running on OS X at all.

Ding! Friday 1st June 2012

Boxer levels up again, to 1.3! This version sports the volume controls I told you about last time: along with screenshotting and fast-forwarding, and a heaping helping of bugfixes. Full release notes are here, in HTML even this time.

I tried to make this release as boring as possible, but a couple of interesting improvements still snuck into it while I wasn't looking:

Smarter drive persistence

One of those why-didn’t-Boxer-always-do-this features: Gameboxes now remember all the drives you’ve added and queue them up next time you play, even drives that aren’t physically part of the gamebox.

Previously, Boxer would only ‘remember’ drives that were part of the gamebox you were playing. This meant that to get an outside drive to be mounted all the time, you either had to muck about with DOSBox mount commands or let Boxer physically import the drive into the gamebox. Which, you know, sucked.

So now: fire up a gamebox, add a new drive, and it’ll just be there every time you play the gamebox. You can rename or move the drive’s source folder or image, and the gamebox will still find it each time. Drag the drive out of the Drives panel and the gamebox will forget about it again. That’s that.

This finally makes it super-easy to share OSX documents with a DOS app, or share a bunch of utilities among several apps. It also frees you from having to do any drive-importing of your own: until and unless you want to share your gamebox with others.

I’m making this sound like a big fucking bright idea but seriously, why didn’t it always do this?

Copy-paste for all

Boxer 1.3 now boasts the ability to paste text from OS X into any DOS program (as well as the DOS prompt, which was supported already in Boxer 1.2.2).

Pasting still has some limitations: only ASCII characters can be pasted at the moment, so accents get lost in translation. And of course, you can’t copy text back from DOS (at least, not yet.)

But it's still awesome, dammit, so I'm going to bore you about how it works:

OK, that’s all! Go back to what you were doing. (Downloading another awesome Boxer update, that is.)

Seeing how the sausage is made Saturday 3rd March 2012

“Volume control” is a single bullet point in the feature list for the upcoming Boxer 1.2.3, but I thought it would be interesting (read “validating”) to give a behind-the-scenes tour of what actually goes into a simple feature like this.

Deciding what (and what not) to do

  1. First things first is deciding the scope of the feature. I chose early on to make the volume control apply application-wide to every game, rather than tracking it for each individual game, so that its behaviour would be consistent and predictable. The use case I targeted was one I faced daily: muting Boxer to listen to other music or videos in the background.

  2. I chose not to target a more sophisticated use case: tweaking the volume of a specific game, such as one with annoying beeper-speaker effects or really loud MIDI music. I felt this was too complex for a single volume control.

    Instead I made plans to add a separate mixer panel later, which would let you tweak the volume of each individual “channel”: PC speaker, digital audio, MIDI and CD audio and so on. The volumes you’ve set for individual channels would be remembered for each game, and scaled by the application’s master volume. But that will have to wait for a future release.

Creating what the user sees and does

Now I'm ready to start on the physical volume control UI that the user manipulates.

  1. I decided to add a slider to the DOS window's status bar, so it would be obvious and immediately accessible. I made the speaker icons at either end of the slider minimize/maximize the volume when clicked, like they do in iTunes. (And because the volume control is sharing space with the “Click to lock the mouse” help text, I also had to make sure the latter is hidden at small window sizes so they won’t overlap.)

  1. But wait: OS X’s standard slider widget was designed for pale backgrounds, and looks dumb against the darker status bar area. We need a custom slider widget that’s restyled to fit. Good thing I had some custom widget code already for the Inspector panel sliders; unfortunately it needed a lot of rework before it was suitable.

  1. Wait, what if the user is in fullscreen or has the status bar hidden? They won’t be able to access the control. So I added a “Sound” menu containing its own slider, which can be accessed from anywhere. The menu also lets me expose keyboard shortcuts with which the user can nudge the volume up or down.

  1. Because the user can nudge the volume with those shortcuts while no volume slider is visible, we need some way to give them feedback that something has happened. Time for a new notification bezel! (Though we should only show the bezel if the volume slider isn't already visible, otherwise it’s redundant: so check for that too.)

  2. I wanted the bezel to show a speaker that changes according to the current volume, so I snapped screenshots of OS X's menubar volume icon in each of its states and recreated them as vector versions in Photoshop.

Making it all work

So now the UI for our volume control is pretty much ready, but it doesn't actually do anything yet. Time to hit the emulation layer.

  1. First, we need to control the volume of the DOSBox mixer: this component mixes together all of the audio produced by the emulated PC speaker, Sound Blaster etc. This means injecting some more code into DOSBox so that Boxer can dictate the master volume.

  2. But General MIDI output doesn’t pass through the DOSBox mixer: it goes straight to OS X’s CoreAudio MIDI synth. Time to dust off my CoreAudio documentation and figure out how to manipulate the volume on audio units.

  3. If we’re sending MIDI music to a real MIDI device, like an MT-32 plugged into your Mac, this uses a different system again: we need to send actual MIDI commands to the device to change its own master volume. Time to dust off the General MIDI spec. (Of course MT-32s don’t respond to General MIDI volume commands, so we need to send a different message to those instead: time to dust off the MT-32 spec too!)

  4. Because MIDI is a low-bandwidth realtime medium, we need to take care not to spam a real MIDI device with tiny volume changes that would crowd out other MIDI messages. This means coalescing volume updates and sending them at certain intervals, once the device isn’t otherwise busy.

  5. Another wrinkle: many MT-32-compatible DOS games send their own MIDI commands to set the master volume. If we let those go through to the external MIDI device they’ll override our own volume: so we need to detect and intercept those commands, and scale them by our own volume before sending them on their way.

  6. And one final snag: Audio tracks from physical CDs are played back through yet another system, a decrepit and awful SDL API, whose playback volume is beyond the reach of Boxer altogether. I decided this was a bridge too far for this release, since Boxer usually rips CD audio anyway to play back in a manner it can control.

    Instead, I swore a dark and solemn oath to rewrite the DOSBox audio architecture for Boxer 1.3: to remove the lingering dependencies on SDL and to unify the disparate outputs under my own roof.

So that’s what goes into a simple application volume control. This is something of an extreme case: some simple features really are trivial to implement, and most don’t have to cover quite so many bases. But they do usually involve a lot of unexpected edge-cases and layers of complexity hidden beneath the surface.

Second verse, same as the first Sunday 12th February 2012

Boxer 1.2.2 is out, hot on the heels of the previous release to fix a couple of serious crasher bugs. The full release notes are here, in RSS form as usual.

Because I’m pathologically incapable of doing a simple bugfix update, I also slipped in a new feature: a simulated numeric keypad to help out players with Macbooks or Apple Wireless Keyboards. In a nutshell:

While Fn is held down the 789, UIO and JKL keys act as the 9 numpad keys, the “M” and “,” keys act as 0 and period, the “.” key acts as Enter and the 6 key acts as Num Lock. The column of keys to the right of these acts as /, *, + and - on the numpad.

If you don’t want to hold down Fn all the time, you can also toggle the numpad behaviour on and off with Cmd+U or Emulation ▸ Simulate Numeric Keypad.

I’ve also (re)introduced paste-to-DOS, which had been around in pre-1.0 versions but was cut for being too buggy. You can now paste single or multiple lines of text at the DOS prompt, letting you easily enter command sequences and complicated paths.

In a future release I'll expand this feature to let you paste text into any DOS application - I can see that coming in really handy for text editors and parser-driven games.

Finally, after some constructive pushback in the comments for the 1.2.1 release, I’ve also reverted Boxer’s default location for the games folder back to [your home folder]/DOS Games/. I appreciate better now how many users regard the Documents folder as sacrosanct—unfortunately, OS X is pretty short on good places to put the-kind-of-objects-that-game-ROMs-are. (This will probably be revisited again when Boxer goes through app store submission, which will be Real Soon Now I Promise.)

Happy Mayan Death Year 2012 Sunday 29th January 2012

Salutations chaps and chapettes! Boxer 1.2.1 has been kicked into the street, shivering and bloody. You can grab the new release here, and as usual the full release notes are here (RSS). This release is still 10.5- and PowerPC-compatible, as it contains Serious Bugfixes. The next feature release won’t be however.

The big things for this version are a flicker-free renderer, GOG game-import fixes, and the removal of the first-run games folder choice.

Back to basics

Back when I tore out DOSBox’s old renderer and wrote my own, I was inordinately proud of an Apple-specific rendering optimisation I put in: one which mapped the emulator’s framebuffer directly to an OpenGL texture, saving on potentially expensive writes when new frames are ready.

So proud I was of it, that I blinded myself to three things:

  1. It significantly increased frame-tearing, as Boxer was frequently rendering the previous frame to the display while the emulation was writing over it with the next frame;

  2. Many Mac GPU chipsets didn’t handle the trick very well, causing stuttering or weird rendering artifacts in fullscreen mode;

  3. The actual performance improvement was negligible because these are 20-year-old games for god’s sake, not Wipeout HD.

So in 1.2.1 I swallowed my pride and rewrote the frame updates in a more traditional way: to my chagrin, it turned out very similar to how DOSBox does it. With tweaks to sync up rendering with the display’s vertical refresh rate, Boxer now renders completely free of frame-tearing. If you play games with lot of scrolling (like platformers or pinball) you’ll notice the difference straight away.

The moral of this story: sometimes the smart solution isn’t very smart.

War of attrition

This update also fixes a serious (and stupid) bug in 1.2 whereby Boxer would crash at the end of importing a GOG game that had BIN+CUE disc images, resulting in a broken game import. If you’ve encountered this, try reimporting games now.

However, Boxer still won’t import certain newer GOG releases correctly: Rayman Forever being exhibit A. Fixing these will require fairly major rewrites to Boxer’s import system, which I hope to do for the next update if I don’t slit my wrists first.

Goodbye choices

I finally pulled my thumb out of my ass and removed the initial first-run window that asks where to keep your DOS games. This now defaults to Documents/DOS Games – though the folder can still be changed later from the Preferences window, or just moved somewhere else via Finder. This change has no impact on existing game folders, it only affects people who are running Boxer for the first time.

I’ve been planning this for a while, because I felt humiliated having to ask the user for a commitment before they’ve tried out the app, and because this is a key part of the shift to a ‘post-filesystem’ game browser model. That shift won’t prevent you from accessing your games from the filesystem or organizing your games how you wish — the goal is just to eliminate the need for everybody to care about where things are located.

Anyway! I should be getting on with the game browser in the next major update, though there may be another minor release before then.

Crossing the floor

Since November I’ve been settling into my new career as an iOS games developer at TicBits! Crossing over into games and iOS development alike have been major long-term goals of mine, and I’m able to work with a great bunch of people, so this has been the perfect opportunity (and one that Boxer helped me get, incidentally). My first game will be out shortly, so look forward to borderline-inappropriate cross-promotion in the coming weeks.

Game development as a day job has meant learning to balance my creative juices between that and Boxer, which is why it’s taken 3 months for me to get a straightforward bugfix release out the door. I think I’ve finally got the hang of this though (the key: weekends) so there should be more substantial updates from here on in 2012.

Incidentally, 1.2.1 is the version I’ll finally be submitting to the Mac App Store: I’ll keep you posted on how that process goes.

Design by 40watt.