Spark AR and the limits of tools

Spark AR is an AR effects tool created by Facebook to create Facebook/Instagram filters. This is an issue when the first idea that you have for something to create with an AR toolkit is not one of those things.

I wanted to talk about Spark AR, or in particular my experience experimenting with it, as it is a very interesting program with a lot of cool applications, and it made me realize a few things about these tools created for certain tasks. Spark AR is meant to create AR filters for photography for a body attached display, a handheld display, usually on your phone. Although can be hooked up to a spacial display via a webcam setup as well.

As I am not an avid social media user, and in fact don't take many photos for reasons other than creating a quick digital image of something to reference later. My immediate thought upon seeing the examples was not to look at the face tracking and decoration, but rather the image recognition and AR placement.

I wanted to create something that you could point at a ttrpg character sheet (In this case I decided to try the [Mothership](/blog/2020/03/16/inspiration-first-contact/) character sheet) and create a random character with all the numbers filled in, in AR.

The first step of the experiment is where I started to hit barriers in the Spark AR toolkit. I wanted to get a random number, and display it on the screen.

Getting the random number was easy enough, there was even a random number range "animator" patch (The method that Spark AR uses for graphical coding) that I was able to get from the creators. The issues came up when I attempted to display that number in a text box. The text boxes only accepted "Text" type variables, while the random number was, well, a number. After some searching I discovered that there was no way to convert them with patches, although I have a decent amount of knowledge in coding, so managed to find a way to create my own script that enabled that conversion.

A test of the random number creator.

The next step was implementing that random number generation into an image placement AR project, so that it could place the image on the character sheet. I came across an issue when I was not able to get the program to recognize the sheet.
I believe the pattern on the image may have been too complex, and so it was having difficulty with any sort of change in angle. Although that is just a guess as I didn't figure out how to fix it.

The second part of this project was almost a week later, when I decided I would set my sights on a different target, specifically making a small game. This game would focus on the strengths of the program, specifically it's facial recognition, even more specifically, blinking. After a little playing around with getting it to recognize multiple faces, then a bit of just working on the logic and altering my script to take multiple inputs and convert them, I created the game successfully.

To start the game you tap on the screen, this starts a 10 second timer, then both players blink as fast as they can, the one who blinks the most wins.

After I finished, I realized I had hit limits of what the Spark AR toolkit was designed to handle almost instantly. As I alluded to earlier, I am not part of the target audience for this toolkit, or more specifically the target audience for the types of things that this toolkit is designed to create. So my thoughts on what I wished to use it for, where not what it was designed for, and thus I hit the limits of what it could do.

In the same way that a hammer is designed for dealing with nails, Spark AR is designed for dealing with aesthetic AR image/video filters. In the same way that a hammer is not so great at dealing with other fasteners, such as screws, Spark AR is not so great at dealing with other uses for AR, such as games or functional tools.

Anyway, it's a very nice program for what it is meant for, just not one I can see myself using for projects that I want to do.

References:
1. Spark AR: https://sparkar.facebook.com/ar-studio/