Documentation

Concept:

We chose the Desktop Skeuomorph theme, with the idea of it being a skeuomorph for a mouse pointer; the Hwacha represents the physical manifestation of a whole bunch of arrow clicks. 

To be truthful though, that was mostly a jumping off point for us to explore an AR concept. Xander's been in Korea the past few months to do research for his thesis next year (based around Korean history), and so has been constantly going to history museums. In museums in Korea (and obviously not just Korea, but it's basically all the history museums here) there are these intricately set up physical dioramas depicting all sorts of subjects. Some of them are so vivid and expressive, but obviously by nature of the medium they're inherently static, and so it's had us thinking what kinds of things we could do with AR to bring that kind of thing for life.

There are also these wooden model kits (historical houses, ships, and this medieval weapon we ended up choosing) in basically all the museums, and so it seemed like a natural thing to test the concept with. I know it's not so accessible in America (and I'm sorry for that; I hope the video is still demonstrative), but this kind of kit is practically ubiquitous in Korean museum stores, and I can just imagine how cool and engaging I'd have found it as an inquisitive child to be able to "bring it to life," so to speak.

Especially if it had historical background, which brings me to the final point of the concept; we're very interested in how games and creative media can be used as methods of teaching. We knew from the start that, in addition to being able to make it "shoot" with AR, we wanted to integrate historical background in some way.

Documentation:

In the end we created a simple (but we hope effective) little AR app for Android that lets you overlay and fire arrows on a historical model, and use the UI to read up on its historical background in both Korean and English. We also did add an Arduino push button hooked up to get it to fire (with Arduino tied to a hotkey for the animation; conventionally the animation can be fired with a button in the app), but unfortunately the Arduino took a bit of a backseat for us this project; we hope the physical model integration can be considered a physical controller in its own way all the same.

The AR is done using a Unity plugin called Vuforia, using its target model system (Chang goes into that in more depth below) to track the physical model, and then have assets layered over it. The wooden model itself is this one by the company Young Modeler, and we did testing both using an Android phone and webcam for the AR.

Xander assembled the main Unity project (although Chang had one he ran in parallel for testing his assets; we had some trouble using Vuforia collaboratively), and also tested the process and debugging and so on of integrating Vuforia.

Chang has been more focused on creating elements that are needed for an AR space. Chang’s first task was to create an exact same proportion of the 3D modeling of the Hwacha which we are going to track through the camera. To make the same proportions model, the ruler was used to measure the actual wooden model. The 3D modeling was made in Blender with the unit set to centimeters. Tracking an object doesn’t require the detail or colored modeling. Therefore it was made in bold shape. In contrast, arrows are textured in Substance Painter since it will exist in the actual AR space. Aside from that, all arrow animations and particle effects were created within Unity. The UI is a text box that contains information about the fire wagon and supports two languages: Korean, and English.

Collaboration:

We did collaboration mostly through Discord chat and occasional meetings, and also used Github to host our project, and sometimes exchange parts of it by sending Unity packages. We also had a Google Drive folder we used for managing and exchanging more general assets. 

Our collaboration mostly went pretty smoothly; the big issue was that Vuforia wasn't sharable like we first hoped (it requires a license key), so it only ended up working on Xander's computer, and we had to debug and test things back and forth a few times at certain steps. 

Insights and Findings

The big finding was simply how to get Vuforia, and so AR with Unity, to work. It's something we've been curious about for a long time, but it always seemed like a big hurdle to get it working. Now though, that road block is finally done! Xander at least is very excited now to start working on some other AR projects over the summer, probably along similar lines.

We also did our first working app on Android, which was remarkably simpler than we had anticipated it being. On the other hand, we found out that getting an app to work on iOS is way more complicated; that's something to try to figure out next time.

Further improvement:

A way to get it (semi-)playable for people even without the model is something we're sorry we weren't able to do; we felt that the wooden part of it was definitely a core part of the experience, but at the same time it's not simple for everyone to get their hands on this particular model that's only really available in Korea. 

Another big one is more Arduino integration. We had initially planned to integrate a bluetooth sensor, so you could press a button on the model itself, it would send the signal to your phone, and you would see the animation. This turned out to be more than we could handle, but still having more physical integration seems like a natural pairing with our project (considering a core part of it is about physically pairing a model with AR), and definitely something to work on in the future.

Finally, for future projects like this, especially at Parsons where we can use the laser lab, we're definitely curious about trying to make our own models rather than relying on a commercial one, which should hopefully add more flexibility to what we can do.