Flash Player: A Declining Asset?

4 YEARS AGO – “A DECLINING ASSET”

I’m working at a technology startup and today I am talking to one of the founders. He looks at me and says, “Our main product is a declining asset.

This is the product that generates 90% of our revenue and pays both of our paychecks. It’s the one that made our company a success, put us on the map.

Uh oh.

NOVEMBER 12, 2011 – ADOBE’S BIG IDEA

If you watched the Digital Media section of Adobe’s recent analyst meeting, you know that Adobe is putting a lot of focus on HTML5. Their recent announcement regarding dropping mobile web browser support for Flash Player caused a lot of turmoil, too, along with a shift in direction for the Flex SDK, their enterprise app framework.

If you look at the marketplace and the technologies at play, it seems that Adobe has realized that Flash’s position in the marketplace is eroding, that the erosion probably can’t be stopped, and they need to treat Flash as a declining asset. Just to review, here are some reasons that Flash’s position is eroding:

  • The many 3rd party mobile, native, and web-targeted development tools like Corona, Moai, Unity and others.
  • Non-Adobe Flash runtimes like Scaleform, Iggy. Companies like The Behemoth have their own Flash-compatible runtimes, too.
  • And of course the big one – HTML5. It can handle more and more enterprise apps, animation/multimedia content, and 3D. Browser vendors are in competition but increasingly targeting Flash-like capabilities.

Long term, HTML5 and other non-Flash technologies are unlikely to go away. Adobe may as well be proactive about owning the space rather than fight an unwinnable battle to keep everyone on Flash.

One more point to consider: Flash is made up of three big pieces. You have the tools, like Flash Builder and Flash Pro. You have the runtime, like the web plugin, the standalone player binaries, and AIR for desktop and mobile. And finally, you have the platform itself – the file formats, AVM specification, compilers, and APIs that define the behavior of Flash content.

They are all independent to a greater or lesser degree. The only part that probably wouldn’t migrate to HTML5 is the actual runtime (but see Gordon). And Adobe has been rumbling about compiling AS3 to JS/HTML5 and supporting C via Alchemy 2.

LABELS AND COMMUNITIES

Now, the funny thing about that conversation from four years ago is that, because of the mental label of “declining asset” we assigned, (at least) two interesting things happened. First, the company got acquired and tried to diversify into a couple of new markets. Second, I along with a few other guys left the company and went on to start a new one.

But the “declining” product continued to make more money than ever before. And in fact, it lives on today, despite the original company getting liquidated by its owner when the diversification strategy didn’t work out. So what does it mean, exactly, to be a declining asset?

I think “declining asset” is a label you put on something to help you make decisions. In Adobe’s case, the decision they made was to move their long term focus toward HTML5 and away from Flash Player.

There are some important things to keep in mind with the communities that develop around technologies and products. First, realize that the conversation is often dominated by the vocal minority – so what is said most often and loudest often doesn’t reflect on the actual needs of your user base. Second, realize that the people who post on your forums are emotionally invested in the product, have it as part of their identity, and they will be deeply unsettled by any signs that support is fading. Finally, realize that users often have a limited perspective. Community members are not tracking major market trends, they are looking at how they can meet their immediate needs (like getting contract work or finishing a specific project).

In other words, the community tends to act like a mob.

And I saw no better example of this than when I was on a group video chat last week and saw Flash professionals practically weeping, calling out Adobe representatives, demanding, threatening to break up, over these announcements. It was more like seeing your drunk friend vent over his ex-girlfriend than it was watching a group of well-respected developers discuss their future. Everything is in a turmoil, it’s the end of the world, everyone is screwed, etc.

REPORTS OF FLASH’S DEATH

Ok, but that isn’t actually the end of “Flash” as a whole. Probably. Even though it really sounds like it. Let me explain.

Adobe has a ton of outs from this situation that let them preserve their and your investments. The most obvious out is replacing Flash Player with HTML5. You export from Flash Pro or Flash Builder and it runs directly on HTML5. In fact, they have been inching towards this in different forms for a while now (the conversion tool on Labs, Edge, Muse, etc.).

Even if they drop AS3 and go with JS, their tools can still be useful. If Flash Pro can still create banner ads and interactive experiences for a large audience, who cares what the output runs on? Life will continue relatively unchanged for a lot of Adobe customers.

There’s also a more subtle out:

HTML5 has its weaknesses. Lots of them. But public opinion supports it. Maybe it’s just a Betamax vs. VHS difference. Or maybe HTML5 is doomed due to the conflicting goals of vendors and the difficulty of the implementation task.

Maybe HTML5 ends up being great for less demanding uses – like basic enterprise apps, ads, motion graphics, etc. – but can’t get it together for highly demanding and integrated stuff like games. Adobe can keep Flash around and focus specifically on the game use case – which, by the way, is also highly beneficial for non-game apps, since they tend to use subsets of game functionality – and get as much value from it as possible for as long as possible.

Between the games angle and inertia, Flash could remain relevant for years. It could even end up totally dominating that space for a long time to come, even as HTML5 takes over the bottom of the market, due to being able to be more focused and agile.

CONCLUSIONS

Let me add two caveats. First caveat: At some point you can only expect so much out of a platform – you can’t get a guarantee that it will remain relevant for ten years. Even proven, still-relevant technologies like C have had their death announced many times. At some point you just have to say, “well, N years more relevance is good enough and I’ll re-evaluate in a year.”

Second caveat: Maybe Adobe screws the pooch and that’s that. Maybe they cut too many resources from Flash. Maybe they don’t build good stuff on HTML5. Maybe they ruin everything. So don’t bet the farm. Make sure you learn a few different technologies well. It will make you a better developer, even if you still just do Flash work day to day. And you’ll sleep easier knowing that if worst comes to worst you have an out. I’ve never seen a successful programmer regret having learned a new language or paradigm.

I don’t think Adobe is making bad decisions, just difficult ones.

Bottom line: Flash is a declining asset, but declining assets aren’t dead or even out of the fight. Everyone needs to look at technologies on their merits and see if it’s a good fit for your needs. There are a lot of places where Flash will continue to be a good fit for a while to come – and the places where it is ambiguous deserve careful consideration regardless of Adobe’s stated plans.

(Thanks for reading! If you liked this article, please consider voting it up on HackerNews or Reddit)

Tips for Flash Developers Looking at Hardware 3D

Adobe has announced 3D support in Flash. A huge step forward for the Flash development community! Tremendous things will be possible. If you haven’t seen it already, here’s a great video of what’s possible:

The demo at Adobe MAX 2010 was impressive, and it was a huge treat to see my friends at Mythos Lab and Alternativa get featured on the big screen during the keynote. 🙂

But let’s get back to reality. This is nothing that hasn’t been out on other platforms for years. Why is it worth talking about? Why does it matter when your PC can do way more?

It’s the audience. When Flash 3D drops, you will be able to create 3d content and bring it to all those billion people who have Flash Player installed. And I am sure that with Adobe’s focus on mobile, they will look towards mobile support, too.

If the model for AAA games like God Of War and Mass Effect is big budget Hollywood movies, then the Flash space is television. There are lots of short-form products (5-10 minutes avg playtime) produced by tiny and often independent teams (like much of what’s on Kongregate or Newgrounds). But there are also longer form products, updated weekly, that have huge distribution. In the TV space they call them series; online they’re called social games. The numbers are similar, too. Farmville gets 56M monthly players currently. How many people watch NCIS or another hit TV show in a month?

TV is probably still bigger for now, but social games monetize better per user. 🙂

So – 3d in Flash will have a big impact. And a lot of developers will be dragged kicking and screaming into the brave new world. I wanted to share a few lessons that I learned in my previous life as a 3d middleware developer at GarageGames. Some are big, some are small, but hopefully all are things that will save you, a Flash developer looking at this new hardware 3D, time and pain. They took me a while to learn.

Framerate and Performance

Don’t use frame rate as a measure of performance. It’s useless. All you really want is to target a fixed framerate of 30Hz, 60Hz, or 120Hz. Here’s how that goes down, by the way: your project will run at 120Hz until the artists get their hands on it. Then you will decide that 60Hz is a great target – looks good, feels good. But when it comes down to the deadline, you’ll hit some bumps and settle for 30Hz. Lucky is the project that can ship with 60Hz, and the only guys I’ve ever seen go out with 120Hz are doing VR, where you have to have it to avoid nausea.

What about variable framerate? It’s bad. Don’t go there. Humans are very good at compensating for fixed delays, and very good at detecting variable delays. It’s much better to cap to a fixed low framerate, because users will automatically compensate and cease to notice it. (This is how our nervous system works; your arms have a fixed delay for nerve signals to propagate, muscles to react, etc. So the capability is built into the brain at a very low level.) However, if it varies, the user will notice every little change in performance and be frustrated.

Back to the performance measurement issue, frame rates are not linearly comparable. What you want is milliseconds per frame (mspf). How come? Because Hz is not linear. It is harder to get from 90Hz from 60Hz than it is to get from 30Hz to 60Hz. How come? Looks like the same diff, right? Let’s put the same problem in mspf. Going from 30Hz to 60Hz means going from 32mspf to 16mspf. But going from 60Hz to 90Hz means going from 16mspf to 11mspf. In the first case I am cutting 16ms off my frame time, which is a big chunk, but I still end up with a relatively generous 16mspf. In the second case, I have to squeeze 5 mspf from an already meager frame budget.

Using mspf also makes it easy to discuss performance gains. If I have a task that takes 4mspf (say applying a full screen blur), and I can optimize it down to 3mspf, I know exactly how much of a win I’ll get no matter how the rest of the application is performing. Compare that to saying that the blur runs at 250Hz then I have optimized it to 333Hz. And when I am budgeting performance I can easily divvy up my available frametime to specific tasks. We might have 8mspf for physics, 4mspf for render calls, and 4mspf for the GPU to finish.

Bottom line: use milliseconds per frame not hertz for performance measurement and budgeting.

GPU Readbacks

(This advice and the advice in the following sections is general; it applies to OGL/D3D and console HW APIs, and I doubt Molehill can deviate from the fundamental hardware reality. Of course, Molehill’s software renderer probably has different performance characteristics – slower, but more permissive.)

Never ever read back from a GPU resource. Ever. Don’t. Stop it. Ok – you can maybe do this ONCE per frame, if you are careful and build your renderer around it.

Why is readback so incredibly bad? Because it forces synchronization (as do a few other things in the graphics API). Then, while the GPU is stopped, you do a slow memory read back to the CPU, which is also stopped and waiting for the data.

Normally, the GPU will run ahead of the CPU – you fire off some draw commands and it goes and does its magic. But imagine you issue these commands:

  1. Clear framebuffer.
  2. Draw triangles.
  3. Read back from framebuffer.

Normally, you’d never see the cost of #2 on the CPU – it would immediately return and you’d go on your way. Say it takes 1ms to do that draw. The call returns in a few microseconds and you keep doing. But when you issue command 3, the CPU has to wait for the GPU to finish command 2. Then, because framebuffers are often stored in proprietary formats either for performance or quality reasons, the GPU has to prepare it for readback. Then it can finally stream the requested data back to the CPU, and subsequently continue on with its rendering. All this cost shows up as you waiting a long time on #3.

Readbacks are tempting to the Flash developer because copying stuff out of the Flash vector renderer is basically free. But on hardware, it gets you a triple whammy – you stop the world, you make the GPU do a bunch of extra work to prepare data for readback, and you have a data copy operation. Don’t do it.

Let the GPU Run Free

Ideally, you want to issue the minimum number of calls to the 3d API that will let the GPU do maximum useful work without interruption. GPUs are like bullet trains. They go really, really fast as long as they don’t have to stop or make right turns. Sometimes it is even better to do somewhat wasteful things because the gain from continuous operation is so big. GPUs are on track to have thousands of parallel processing cores. Just let ’em run and you’ll be amazed at what they can do!

Compare this to the Flash vector renderer which is best when it is given minimal work to do. It’s a great renderer but it doesn’t have the benefit of powerful dedicated hardware that has been tuned for decades to run it fast.

If you have a big batch of geometry, it’s often best to simply draw it. You will want to figure out what the threshold is at which culling is beneficial, but it can often be on the order of tens of thousands of triangles.

Vertex shading is practically free when you’re looking at rendering 1080p at 60Hz. You’re going to be hitting 124M pixels every frame at MINIMUM – if you have any overdraw it can easily hit 300M or higher. If you have a million triangles in your scene, you’re only going to be doing 60M triangles in that same time frame. So take advantage of it when you can.

Don’t let the GPU run too far ahead, though – it introduces control latency. GPU drivers can often buffer several frames ahead, and if you aren’t careful you can add 50-100ms of latency between a user hitting a button and something happening. Introducing a small readback at end of frame can help force synchronization. (Molehill might deal with this for you, too.)

Performance Trade-offs & Maintaining Your Image

On the GPU, you have huge opportunities to trade space for time for quality. Lookup textures can replace costly math functions (or make them tweakable by artists). You can sacrifice precision (either using half precision or just being a little bit wrong) for performance, too, to get big wins. The great thing about graphics programming is that it only needs to LOOK right, not BE right. So pile on the hacks – if it fits your game’s look, it’s a good solution.

That’s enough on performance. I could write a whole book on optimizing 3d game performance. In fact, I have; it’s called Video Game Optimization. It covers the whole problem from scheduling for performance to measuring to identifying bottlenecks to fixing them. It even has a chapter on Flash optimization!

Stand on the Shoulders of Giants

Graphics are sexy and well studied. The fount of graphics research is SIGGRAPH (any further back than that and you’re tracking individual researchers). Literally every rendering problem you will encounter when working with Molehill has a solution that was published at Siggraph thirty years ago, played with at SGI twenty years ago, considered at 3DFX 15 years ago, and brought into the mainstream and shipped on a console or PC game 5-10 years ago. Around then it was also republished as a practical implementation in a Gems book, and also shipped as a card demo by ATI, nVidia, or Intel. If it’s really good, Carmack or Abrash prototyped it when they were working on Quake 1, Epic has it in Unreal, Crytek has some great demo videos of it, or Valve has published slides on it.

Am I claiming that there’s no fundamental research left in 3d graphics, or no expressivity? Of course not. But you owe it to yourself to be familiar with existing work in the field before you go reinventing the wheel, and the field is aggressively researched. So I would not expect to stumble on any huge break through right away, just like if you pick up oil painting you’re probably not going to reinvent the field…

I think the interesting part of graphics is finding the best fit for your specific problem and enabling artists. It’s a craft. Look at what Pixar does. Their research has always been in support of their story and their specific technology needs. Out of that they have ended up contributing a lot of great stuff, and innovating in a lot of ways, but they did it with a strong understanding of existing technology.

I encourage you to do the same. It is what I have done in my own projects and it has worked out well. When I have pursued tech for tech’s sake the results have always been less effective – good for a demo but ultimately ineffective in enabling my team to succeed.

It’s the Assets, Dummy

How do you make a good-looking game? Really, it’s not the technology. Shaders, complex rendering techniques, and so forth are all fine and they can enable good results. But great art is what carries a project.

Look at that video at the top of the article (it’s OK; I’ll wait). There are very few complex rendering techniques at work. Mostly, it is great art (and a great lightmapper) with just a few simple shaders and effects.

Look at Mass Effect. They use shaders, sure, but the real source of the game’s look is the tireless work of many talented artists.

Or look at one of our older projects, Grunts: Skirmish:

All the runtime is doing is compositing unrotated, unscaled bitmaps. But the great work of our artist, Tim Aste, makes it a memorable and interesting visual.

To that end it’s vital that you make it easy for artists to work with your technology. There are two lessons I have learned in this are:

  1. Make it easy to see. Have an automatically updated live build somewhere and let the artists have the URL. That way they can check art in and see how it looks in the game, or tweak things as needed, without involving a programmer.
  2. Make it easy to process art. Make sure that the inputs to your art pipeline are high-quality uncompressed files. Then process them into whatever format and quality level you require. Optimize the final format for very fast download and unpacking.
  3. Your art needs and formats will change. Be able, at any time, to reprocess all of your source assets into their final form. Get everyone used to the system early on and you will have the agility you need late in the project to succeed. There’s nothing worse than asking your artists to hand-process dozens or hundreds of assets due to some technical change! (It’s not just unpleasant, it also introduces mistakes, since they are people, not machines.)

Art pipelines and processing is a subject worth of a whole bookshelf. Go out and learn as you need it. Not every project needs an all-encompassing solution, but… well, read the next section. 🙂

Trends & Closing Thoughts

3D changes the landscape of application development. Flash has tracked the console space’s evolution, on a much shorter timeframe. Looking at how consoles were affected by the introduction of 3D is very instructive:

  • Art becomes more complex. Creating an animated character in the SNES days was the work of a single person for a few days. Creating a character for Mass Effect is months of work for a team.
  • Asset size increases. A full SNES game is 16MB or so. 40 hours of gameplay, tons of content, music, visuals, cutscenes, etc. A typical XBox 360 title is 6gb, often compressed, and now 2 disc games are coming out. An asset footprint of 15gb is not unrealistic. It will be interesting to see how this plays out in the online space.
  • Budgets and complexity swell. I bet you could make a complete, high quality clone of A Link to the Past for two month’s operating budget of Mass Effect or Halo. Teams will get bigger, projects will become more complex, and technical requirements will increase.
  • Gameplay remains relevant. The hit titles in the console space are those with high technical quality – but they are also fun. Mass Effect is highly immersive. Halo has highly polished gameplay. God of War is a remarkable experience in an industry full of remarkable experiences. And indie titles that are genuinely fun and engaging – like Minecraft – are able to succeed without large marketing or art budgets.

Right now, Flash is at the same place the PC space was in 1998-1999. We can do compelling 2D or limited 3D content and teams are getting larger but not yet huge. Molehill will bump us forward to around 2001-2002, perhaps a bit further. It’s an exciting time to be a Flash developer.

The kinds of experiences we will be able to build are going to undergo a quantum leap, as will the demands on us as developers. We are in a unique position to leverage the lessons of the past to build the games of the future. What do you think the major issues will be? How can we build great content on this new feature set?

The Flash Display List And Games

This is why I always run a raster pipeline. The DisplayObject API is a trick played on the gullible and the trusting… — @bengarney

Everyone has their hot buttons. I work on libraries and architecture for a living, meaning that I am personally invested in those kinds of issues, meaning that I get excited about things like platform feature sets in ways that maybe most developers don’t.

Here are my assumptions: most Flash developers want to put art on the screen and move it around at a mostly decent framerate. Most of these developers are building interactive experiences and don’t have heavy coding backgrounds (although this is changing). Their priorities are influenced by this – they want ease of use, fast iterations, and don’t necessarily care about high performance (although this is changing, too).

Meanwhile, I inhabit an alternative universe. The last game I contributed to, Social City, had high thousands of frames of animation and hundreds of animated objects on screen at any given moment. In the whole scene there might be low thousands of visible objects. Because the world was isometric all these objects had to be drawn in the right order and most of them updated every frame.

In our initial technology tests, just having that many objects in the display list would make mouse interactions take up most of the CPU. Rendering was also extremely slow. Between these two issues we were entering the seconds-per-frame zone – and we hadn’t even added the gameplay or complex animation yet!

I am a big believer in finding the fast paths and sticking to them. Computers are fast. Really fast. My current low-end MacBook Pro, on a single core, has something like 1200 instructions per pixel to burn if I am trying to render 640×480 @ 24Hz. In the Social City case, all we wanted to do was copy opaque pixel data – which should be something like 5-10 ops per pixel, probably much less. So really, even taking into account that buildings often overlap, I should be able to push around a hundred frames a second if all I do is render.

All this is primarily to point out how awful a one second frame is. It means I am running a thousand times slower than I should be.

You might think that the most important thing in a game is how fast things can run. After all we always want to push the limit. So maximum speed is vital, right? That’s not necessarily true – we’re only looking at part of the picture. The human eye responds to rate of motion. It is really good at spotting speedups or slowdowns on the order of tens of milliseconds. It’s actually better to run at a lower, consistent framerate than it is to have a higher average framerate with occasional dips, because dips draw attention to performance or lack thereof. That’s why movies run at 24Hz, even though the TV in my living room can update at 120Hz, and why console games often target a fixed 30Hz or 60Hz.

A big part of building a rendering system for a game is identifying how to deliver a consistently good experience for the player, even when they have a fully built out complex level with tons of bad guys, interactive objects, and/or special effects.

Before and After the Flash Display List

How does all this relate to the Flash display list? Well, it is very powerful. But it is also unpredictable. It’s like the dark side of the Force. It’s a quick path to great power. Only later, do you realize that you’re wearing restricting black clothing, hanging out at weird parties with old men, and experiencing hard-to-diagnose performance issues.

Or more technically: the enemy of a system with consistent performance is black box code. Of course, I use the display list for lots of things, just like any Flash developer. It’s a useful, flexible, and powerful tool.

But when I have specific, performance oriented needs, I ditch it in favor of something I can control completely. That’s why I always use a raster pipeline – drawing all my own pixels – for my game rendering. Using BitmapData.copyPixels has a consistent, reliable cost, whereas the cost of drawing a DisplayObject hierarchy, either on stage or via draw(), is highly variable and difficult to predict in terms of both memory and CPU consumption. This is also why I avoid using Flash IDE created assets; it’s too hard to rely on the performance characteristics of art produced from it. If I have to, I always render everything to a BitmapData first and draw it later on my own terms.

(As an aside, Mobile Flash platforms lack CPU, so taking advantage of the GPU is important. In this case, I would still use bitmaps but let the GPU manage compositing when I can, for instance by using flash.display.Bitmap. I would also carefully measure the cost of off-screen Bitmaps to see if I need to manually add/remove stuff as it becomes visible. Some early tests also suggest that you can get pretty far on BitmapData on mobile if you are careful, too.)

I could write more on raster based rendering, but if you are interested in nitty gritty, you might enjoy looking over the Rendering2D library from PBE; it’s the basis of the rendering in Social City and other titles that I have worked on, and demonstrates a framework for efficient hybrid raster/vector rendering.

PushButton Labs Profiler

Do you want your Flash content to perform well?

Have you struggled with the tools built into Flash Builder?

Well, if all you want is a basic functioning profiler, let me introduce the PBLabs Profiler! It shows you where your time is going:

It’s available under the GPL with source and binaries on GitHub. You can also post on the PBLabs Profiler forums. There is more information on the profiler in the forums and on the github site.

PushButton Engine 101 Talk Video Now Available

I gave a talk on the PushButton Engine at the Flash Gaming Summit a few weeks ago, titled “PushButton Engine 101.” It is available online, thanks to Adobe’s kind support in sponsoring FGS and recording the sessions. You can view the PushButton Engine 101 talk here. It’s about 40 minutes long. People seemed to like it!

If you have comments/questions/feedback, I highly recommend visiting the PushButton Engine forums thread about it.

The Flash Gaming Summit was a blast. I got to spend time with the guys from 8bit Rocket, with Dan Cook, and, of course, the crews from Adobe and Mochi. For the second conference, Adam Saltsman eluded me, but I had a great conversation with Nicolas Cannasse about building companies. Lunch was spent with the PBE crew in attendance – JD Conley, Dion Whitehead, and a couple of others. I had a million other awesome conversations, too, which is why I love going to conferences. 🙂

Other than the very early start, I thought FGS went a lot better than last year. (Perhaps because they had me on the advisory board – conference organizers take note! ;)) They had a solid tech track in addition to their business track. Maybe next year it can be two days – or just start a little later.

3D in Flash 10 & Git

picture-1

I spent a little time with Flash 10’s 3d features recently. Since Flash 10.1 is imminent and FP10 has been at 90%+ penetration for a while now, it’s probably safe to start looking at using FP10 stuff in my projects. 🙂

I also used this as an opportunity to try out git. It was easy to get git installed on OSX (I used the command line version, installed from git-osx-installer) and put my code up on Github. You can browse my test code at http://github.com/bengarney/garney-experiments/tree/master/exploringFlash3D/.

My main concern was the transformation pipeline – I think there might be some benefits to using 3d positions for the rendering pipeline in PBE. So I wanted to do a brief survey of the built-in 3d capabilities, then look more closely at the transformation routines.

My first test was making a DisplayObject rotate in 3d (minimal source code). It runs well, and if you turn on redraw region display, you can see that it’s properly calculating the parts of the screen that need to be modified.

This was easy to write, but it revealed the primary flaws with the built-in Flash 3d capabilities. First, look closely – the edges of the shape were sharp, but the interior detail is aliased. This is because whenever the 3D rendering path is engaged, it turns on cacheAsBitmap. This is fine for simple scenarios (say, taking a single element in a UI and giving it a nice transition) but not for more complex situations.

Which brings us to the second and bigger flaw. I added a thousand simple particle sprites at different 3d positions (source code). This runs extremely slowly because of an issue described by Keith Peters involving nested 3d transforms. Nested 3d objects cause excessive bitmap caching, dramatically reducing performance. You might end up with bitmap-caching in action on every 3d object and every DisplayObject containing 3d objects.

In addition, because it’s cached, moving objects further/closer from the camera results in an upsampled/downsampled image. So you tend to get mediocre visual results if your objects move much.

My next step was to stop using the 3d capabilities of DisplayObjects, and just position them in x/y based on their 3D position (source code, notice it is two files now). This gave a massive performance gain. At low quality, 1000 particles runs at 1440×700 at acceptable FPS. Most of the overhead is in the Flash renderer, not the code to update DisplayObject positions, but it still takes a while to do all the transformation, and it causes a lot of pressure on the garbage collector from all the 1000 temporary Vector3D instances that are created every frame. (600kb/second or so – not insignificant.)

Next I figured it would be helpful to make my camera move around (sample code).

This required that I understand the coordinate space all this operated in. What are the coordinate spaces? According to the official docs, screen XY maps to world XY. So forward is Z+, up is Y-, right is X+. Once I figured that out, I had to prepare a worldMatrix with the transform of the camera, then append the projectionMatrix. The PerspectiveProjection class always seems to assume screen coordinate (0,0) is the center of the projection so you will have to manually offset. Maybe I was not using the projection right, since the docs imply otherwise.

There were two other details to sort out. First, I had to reject objects behind the camera, and second, I had to scale objects correctly so they appeared to have perspective. The solution revolved around the same information – the pre-projection Z. By hiding all objects with Z < 0 and scaling by focalLength / preZ, I was able to get it to behave properly.

Next up is Matrix3D.transformVector… which is slow. Transforming 1000 vectors eats 3ms in release build! This is really slow in absolute terms (Ralph Hauwert has a good example of the same functionality running much much faster). I didn’t really want to introduce Alchemy for this project. But we can use AS3 code that avoids the allocations, saving us GC overhead and getting us an incremental improvement in performance.

Andre Michelle has some interesting thoughts on the problem of temporary objects related to transformations (see http://blog.andre-michelle.com/2008/too-late-very-simple-but-striking-feature-request-for-flash10-3d/). I did notice that Utils3D.projectVectors had some options for avoiding allocations, but it did not seem to run significantly faster (even in release build). (sample code for using projectVectors)

In the end, I settled on my own implementation of transformVectors, as it seemed to give the best balance between performance and ease of us. There’s a final version of the sample app where you can toggle between transformVector and the AS3 version by commenting out line 105/106 up on github, so you can test it for yourself. The transform function took some effort to get right, so here it is to save you the pain of implementing it yourself. It transform i by m and stores it in o.

        final public function transformVec(m:Matrix3D, i:Vector3D, o:Vector3D):void
        {
            const x:Number = i.x, y:Number = i.y, z:Number = i.z;
            const d:Vector. = m.rawData;

            o.x = x * d[0] + y * d[4] + z * d[8] + d[12];
            o.y = x * d[1] + y * d[5] + z * d[9] + d[13];
            o.z = x * d[2] + y * d[6] + z * d[10] + d[14];
            o.w = x * d[3] + y * d[7] + z * d[11] + d[15];
        }

Time for some conclusions. I think that the 3D capabilities built into DisplayObject are OK, but very focused on light-weight graphic design use. Building a significant 3D application requires you write your own rendering code built on Flash’s 2D capabilities (either DisplayObjects or drawTriangles and friends). The 3d math classes are ok, but immature. Some things are very handy (like the prepend/append versions of all the methods on Matrix3D), but the tendency for Flash APIs to implicitly allocate temporary objects limits the usefulness of some the most central API calls. In addition, important assumptions like the order of the values in Matrix3D.rawData were not documented, leading to frustrating trial and error. I am excited to see Flash’s 3d capabilities mature.

Flash on iPhone: My Experience

Hello, world!

I wanted to post some real world information on Adobe’s latest big announcement, Adobe Flash Applications for iPhone. I participated in the pre-release beta along with several other very talented Flash developers. We got to be some of the first outsiders to work with the iPhone technology. To try out the tech, we wrote and shipped Trading Stuff in Outer Space in just 8 days.

By the way, if this is Flash on iPhone stuff is new to you, you should definitely read Aditya’s “Developing for the Apple iPhone using Flash” article on adobe.com. In addition to being an all around great guy, Aditya was also a huge help in working through the very early beta issues we encountered. Ted Patrick posted some cool iPhone example apps with source. (Not only does Ted make a good jedi, he also helped me debug some rendering issues in Trading Stuff at the last minute. Thanks, Ted!) And Mike Chambers, who brought me into the program, has posted more explanation with a lot of useful links for Flash/iPhone work.

It’s worth stating emphatically that Adobe has a large, talented team working hard on iPhone. I couldn’t hope to properly thank all the Adobe folk who explained, demonstrated, debugged, and implemented through long sleepless nights to build this tech and fix bugs so we could ship Trading Stuff. Guys, you rock!

A brief history of Ben – I spent five years working on 3D and 2D C++ game engine technology before I moved to Flash, so I have a little different background than most Flash developers.

Ok – thanks and context out of the way. Let’s get to the meat. What’s it like to develop for the iPhone with Flash?

First, the toolchain is fantastic. All you do to get on the iPhone is write a Flash app however you like, and then (more or less) click the “compile IPA” button. You get an IPA, you deploy it on your device, and bob’s your uncle. That’s the easy part, and it has been that easy since day one. (For comparison, I’ve seen builds with pre-release and final XBox 360 SDKs, with homebrew PSP and Nintendo DS, with PS3 and Wii – they are all a lot harder to start building on.) Sometimes takes a while for all the optimization to happen, but you don’t need that unless you are packaging a release for distribution.

Deploying to the iPhone is the easy part. The hard part is getting your content to run silky smooth and in an iPhone-friendly way. What do I mean by that? Performance? Sure, you have to be careful with performance, but you also need to make sure you respect the touch interface’s limitations, that you obey Apple’s guidelines, that your art fits the device, and so forth. We spent as much time on that stuff as we did on performance in Trading Stuff.

The main things to watch out for is rendering – you must use hardware acceleration, which acts a lot like cacheAsBitmap – and memory allocation – the iPhone has very slow memory, so the GC running is the kiss of death. I could list a bunch of specifics but at this stage in the game, they would be out of date by the next SDK update. The best thing to do in this and all other optimization cases is to measure, find the biggest bottleneck, change it, and measure again to see if you got a win. Repeat until you have adequate performance.

Louis Gerbarg posted a great analysis of the iPhone tech based on Trading Stuff. Since I wrote Trading Stuff, I figured I should comment on it. 🙂

On the LLVM piece, Adobe has put in major effort to make this function properly. And it really does work. Even on day one of the prerelease, the compiler worked pretty well. Occasionally you’d hit an optimizer error – but it’s worth noting that in LLVM most optimization operations are separate from the backend. IMHO, Adobe was very smart to use LLVM as opposed to rolling their own or repurposing GCC, due to code maintainability and extensibility issues. You will notice that they have used LLVM before in Alchemy, so there’s prior experience here. And though Louis cites Apple not using LLVM for iPhone compilation as a weakness, Apple is using LLVM for their desktop compilation, which shows that LLVM is going to be a good choice long term.

For the resource bundling, that’s entirely my fault – I chose to embed all my resources in my SWF and when the SWF was converted, it was converted to a binary with those resources in it. I could have put them alongside the SWF and they would have been bundled as part of the IPA (in fact, this is the case for Default.png, which is treated as a normal file even though it has special behavior thanks to OS X). I’m not so sure about this linker behavior Louis is referring to; checking the Mach-O segment reference doesn’t reveal any insights beyond that yes, it is probably bad to keep the compressed version of assets in RAM. But Trading Stuff isn’t paging anyway, making it a non-issue.

For the rendering performance, the build of Trading Stuff that is on the App Store is not using hardware acceleration at all. And the ARM11 is a crappy chip for software rendering. I was showing a build at Adobe MAX that used hardware rendering, and it runs smoothly, like a real game should. I am waiting for a few bugs in Adobe’s SDK to be fixed, at which point I will release a new version on the store. The performance difference is night and day. I’ll post about that when I get the update out.

Let me put a caveat here: the iPhone is something like a tenth of the machine your desktop PC is. So if you want to do incredible 3d graphics and hand-optimize your assembly, maybe you should look into using XCode and Objective C, and build your content and code explicitly for the iPhone. However, if you know Flash (or have a project in it to port) and want to put something out quickly, this is a fantastic path, and worth a look. And it’s only going to improve from this very early version.

Time for some closing thoughts. Adobe has shown that they value the platform and want to bring their content to it by making some serious technology investments. I hope that Apple will respect that, and work with Adobe to make sure Flash works well on the iPhone.