Will AR lead to the invention of a new instrument?

In September, we launched our hubraum iOS augmented reality innovation program. We invited iOS developers from all over the world to show us their most cutting-edge AR applications which could be boosted by Deutsche Telekom technology like 5G and low latency edge computing. 

When you think of AR, you probably think of gaming. But there’s an entirely different slice of entertainment set to be transformed and enhanced by this new technology: music. Nothing demonstrates this better than two of the startups enrolled on our program: Beatsy, an AR music visualization app that lets you see the world around you being changed by the music you’re listening to, and Magic Piano AR, an app democratizing how aspiring musicians learn how to play instruments by using an AR overlay over piano keys to show which notes to play.

A video should be visible at this point. Due to your cookie settings this cannot be displayed. Please activate cookies for Preferences, Marketing & Statistics to view the video.

Revoke current setting

We brought Beatsy creator Matt Bierner and Magic Piano AR creator Gregoire LeMoulant together for a conversation about how AR will allow you to interact with your favorite popstars from the comfort of your living room; why AR at concerts might take a second and how AR could lead to the invention of entirely new instruments. 

hubraum: Hi Gregoire, hi Matt. Thanks for being with us today. Let’s start with an important question: How does augmented reality help people become better musicians?

Gregoire: I think AR will simplify complex things: firstly, because it allows creators to display content in an intuitive way to the user and secondly because it allows the user to interact with their environment. So I think an important role for AR will be as a sort of teacher, allowing people to understand complicated ideas. It also provides gamification — after all, music theory can sometimes be a bit boring.

Matt: I can definitely see those points as well. Reading notes on a page can be very abstract and I think you can come up with more intuitive ways to represent music. One project I worked on was coloured lights which would change the colour based on whether or not you were playing an instrument in tune. I thought that would be a lot easier to understand than trying to play and listen and trying to match up the pitches. 

hubraum: Yes, for sure. And speaking of presenting music in a different way, how do visual overlays help people better appreciate music?

Matt: With Beatsy, our focus was always on mixing the senses in interesting ways: asking yourself, what does this music make me think of? And then applying that to the world around you. 

When you play a bass note for example, you can envision that note flowing across the world. If you’re listening to a powerful song, you might even close your eyes for a moment and imagine that happening— but it’s a very different experience to see that happen in front of you.  So we really like to get people thinking about music in a slightly different way and forging connections that they maybe wouldn’t otherwise have made. 

Music videos are a good example of this: they take a song that you know, present it in a unique way and often the music video itself becomes its own work of art, like with “Thriller” or Aha’s “Take Me Out” video. Those videos almost overtook the song and they’re what people remember when they think of a certain track. I think augmented reality has similar potential in that you can create these very visual and rich experiences that will be directly tied to the music.

Gregoire: Exactly, there’s a palpable difference between listening to music and attending a real concert — I think AR will occupy a space somewhere between those two experiences because it’s still something you can do on your couch but it will introduce the visual aspect of a concert into your room. 

hubraum: So are you both saying it would add an extra dimension to the music?

Gregoire: Yes, and it’s worth stressing that AR isn’t just about visual content, it could also be about sound. For example, there’s Dolby Atmos, which is a kind of 3D sound — when you couple this with AR, it is very powerful. If you watch a concert in your room, you will be able to feel where the sound is coming from in your room. So you can position a song so the sound comes from in front of you or you can feel it behind you, much as music has a physical dimension when you’re at a concert. 

hubraum: That sounds like a much more vibrant home concert experience! But moving to a trickier question, is the future of AR in music primarily one of education or entertainment and why? 

Matt: I’m biased but I think it’s entertainment — it’s a much larger market segment. The main way people engage with music is by listening to music rather than the education side of things. This said, I don’t think AR will be used for all listening experiences in the same way that not everybody watches music videos, but it might be a good way to augment those creative experiences. It’s easy to imagine AR happening at concerts or dance parties to enhance the experience. 

Gregoire: Conversely, I think education will dominate. Why? Because I think entertainment will be more reliant on VR, whereas AR will be the tool you use to really interact with things around you. This interactive quality is what you need for education and I think it doesn’t have to be limited just to instruments! For example, I could imagine if a person wanted to learn geometry, doing so via AR will be a lot easier. However, I think if AR is used effectively it will erode the border between education and entertainment anyway, since good education is innately entertaining. 

Gregoire LeMoulant, Magic Piano AR creator

hubraum: That’s a very diplomatic conclusion, Gregoire! We can see huge potential in both sectors – but how long will it take until AR is a mainstream part of music and most people are using some sort of AR-based app on a daily basis, whether as part of their free time or to learn music?

Matt: I’d say it’s probably still a way out in terms of people doing this regularly. To get to the point where you’d have a concert enhanced by AR is challenging: A concert is pretty much the worst place to do augmented reality because it’s dark, there aren’t usually walls nearby to project the effects on, so to really get to that sort of experience is really a way off. But I could see it being possible to attend a virtual concert at home in five years. You might pay for music video type experiences where they’re pre-recorded, with a musician coming into your space and being able to interact that way.

hubraum: Yes, that sounds delightful! But let’s get into thornier territory: what would you say the biggest problem facing music AR is?

Matt: As something new, you have to sell people on the idea of why they should be interested in AR. People generally are familiar with this technology from Pokemon Go but these apps are pretty simple in what they’re actually doing. This means if you say you can hold a concert in augmented reality, they might think of a virtual violin that would play that would be entertaining for a minute, but not longer term. But what we’re thinking of is a really immersive experience that’s about taking different parts of your environment and making them part of the performance. 

Gregoire: That does sound tricky. For us, our biggest problem is that we don’t have headsets and to play the piano you need both hands! So how do you hold your phone? For example, on our app, I display AR content to the user and they have to tap on the real key to make a sound but it’s tricky for the user to learn something and play notes without using an AR headset. 

Matt Bierner, Beatsy creator

hubraum: And to end on a more positive note, what do you think is most compelling about AR at the moment?

Gregoire: I think it’s powerful that usually, to play complex music, you’d need to complete years of music theory, but with AR, someone who doesn’t know how to play the piano can just use the AR overlay to play a song.  This reduces the barrier to entry in getting started as a musician — a lot of people are discouraged because of music theory and give up on learning before they even touch a real instrument. 

Matt: Yes, that distinction between real and virtual is interesting. A lot of augmented reality right now is about trying to recreate the real world — like, being able to place this virtual couch in my house to see what it would look like, which is a useful tool. But I think there’s also a lot of opportunities to go beyond that too and to create experiences that wouldn’t otherwise be possible or new instruments that are really compelling and aren’t going to be limited by what we have in the real world. 

These types of experiences or even new instruments that you wouldn’t be able to create in the world are where the potential lies. But those are also the hardest to come up with, because you have to invent them — they don’t exist right now. It can be hard to do that instead of being constrained by thinking of something you saw in a music video or being inspired by an iTunes visualizer from a number of years ago. I think it’s going to take quite a while before people really get comfortable with treating AR as real. Not real like you can touch it, but in the sense that it’s a real perception of the world and if you have a virtual instrument and you master that, you’ve learned a skill, even if that instrument could never exist as a physical thing.

Want more insights into exciting tech developments just round the corner? We’ll be presenting our study on the German AI ecosystem and areas of untapped potential at our Berlin campus on Thursday December 2, 10am. Click here to register for free.

How did you like this article?

Click on the icon to leave a rating.

Loved it!
It was okay!
I didn’t like it!

Don’t miss out.

Want to hear about spots on our limited-place programs before anyone else? Subscribe to our newsletter!