dispatchEvent()™

While checking out Grant Skinner’s new tweening engine, gTween, I was bothered by one small phrase…

gTween is a small (4.5kb), fast (1500 instances, 0.5s duration, ~25fps), instance based tweening class, with a huge number of options and capabilities.

The definition of ‘fast’ in terms of Flash Player performance is somewhat of a mystery. We’re looking for high frame rate i guess? Lots of things on stage? Total time of operations? But frame rate and number of instances don’t really tell the whole story. There are a number of factors that make the Flash Player performance a very difficult thing to measure.

• Flash Player performance varies based on the speed of the viewer’s computer.

That’s nothing new. All apps deal with this. However, Flash Player has these added complications.

• Flash Player performance varies based on what version of the player is being used.
• Flash Player performance varies based on the browser in which it is embedded.
• The browsers’ Flash Player runs at a different speed as desktop versions (browsers seem to have a speed cap around 50 or 60 fps while stand alone versions do not)
• Loading times for external assets must sometimes be taken into account.
• Framerates can vary based on the set framerate of the Flash app. Rumored ‘magic framerates’ may affect this as well.
• Flash Player can sometimes hang, crash, or self-destruct if too many processes are going on at once.
• Flash Player 10′s support for video hardware should complicate things further (although it will probably make our lives easier in the long run).

The results of our latest benchmark

As far as I know, there is no single standard for testing performance of Flash applications (unless you count the Flex Profiler which we probably should). I’ve definitely seen benchmark tests for comparing specific operations, like tweening and 3d rendering, but they are usually very specific to the task at hand and the results are difficult to compare because of all the variables.

Any good scientific experiment will always measure a control group. That is, a test of a relatively variable free environment used as a baseline measure. In the case of Flash Player performance, this would be some fixed set of functions that may produce different results on different platforms but should reflect the overall performance of each platform. For example, a suite of math and visual computations similar to the ones found here. This would allowing us to describe test operations in terms of a multiplier of the baseline. In other words if the baseline took 623ms with an average framerate of 45fps and a Tweener test took 2430ms @ 37fps you might say that the results were 3.9*base time @ 82% frame rate. Ideally, this number would stay more or less the same regardless of which computer or player it was running on.

I had considered what might happen if there was a public forum for posting swf files. There, a developer could post a file and gather averaged data from the other users of the site. The price of this data would be that the developer would participate by viewing, say, 5 other projects thus insuring that there will be plenty of eyes on the work that’s posted.

But in my experience, the average Flash developer has a very different setup from the intended audience. So another possible solution is to create a testing environment based on the most popular machine spec. As this post at Draw Logic points out, we don’t really know that much about the hardware specs for our audiences, although, we could probably assume that the average user has Flash Player 9.0 and is using a single-core, Windows-based machine and Internet Explorer.

Caleb Johnston of this blog notes:

I think that the machine-standard is a bad idea because the reality is that performance is relative. That’s how Flash works… its an important thing that everybody neglects. I think that people should be more aware that their program may run faster/slower on different machines.

In the end, the best practices for benchmarking in Flash Player remain elusive. What do you think about this? My questions to you, the reader:

1. Broadly, what is the best way to benchmark in Flash? What techniques have you used successfully?
2. What are the factors we’re interested in when measuring success? Framerate? Total time of operation? Something else? What are the best practices for measuring these factors?
3. How can we overcome some of the pitfalls pointed out in this article?

I’m looking forward to reading your comments!