Introduction
“Fail your way to success”. This is a quote I once heard in my second semester. When I think I ruined something or it just does not work, I always try to think of this. As one once said, "Failure is the opportunity to begin again more intelligently". Therefore, I should reflect and learn from it, which is sometimes difficult for someone as stubborn as me.
Who am I? Shihaab Rouine and at the end of the starting semester, I chose for software, which I then changed to media design. But I was also very interested in the other directions, so I guess my interests are broad. I think this reflects on my products. In the previous semester, I focused on interaction design and delivering a whole product. This is something I definitely want to continue, and I think this fits perfectly within the field of creative technology.
Before this education, I graduated from application and media development, which is an MBO study. I mainly learned the skills of application development through practice. I did three internships and one of them was in Spain which lasted six months. By forcing myself out of my comfort zone through these internships I developed my interests in the unknown. I am a curious individual, I’m always fascinated by strange and bizarre ideas.
In the last two semesters, I have learned that I enjoy making products and using the programming skills that I have more as a tool rather than using it as the main focus.
Individual project
What if smart homes were not meant to make humans lazy but instead to provide accessibility for those who actually need it? In my previous semester, I made a smart home app that controls the lights per room. The concept of an ‘accessible home’ fits in this project, so I want to expand this project with new features which are focused on making a smart home accessible.
Seeing as I have only a few weeks, I decided to make only a part of this concept. I had the idea to change the notification manner of a doorbell; instead of the doorbell making a sound it could play with the lights in your room. This would be useful for people who have trouble hearing or are deaf. So, I chose to develop this idea and started coding.
The smart home application is a web app running on Laravel. Conveniently, this made making an API very simple and that is exactly what I need right now to create a proof of concept. So, for starters, I get all the lights in a household, then I turn the lights on and off in a loop. Here are two videos of this prototype in the working.
Camilla in Chronospheres
Instead of having two separate projects we have been given the opportunity to dive into the new technology of volumetric video recording and augmented reality. Our assignment was to create a prototype and concept for a live show of Camilla Blue. She is a music artist that is willing to try this technology for one of her shows. This show will take place in a year or two and the project will be taken over by professionals. We also get to work with five students of the minor transmedia design at ACI (academy of creative industries). It’s really cool to have the opportunity to work with people from such a creative direction and this can really help me develop my creativity. My process of this project can be followed in the document: ‘Process Camilla Blue’.
Client Introduction
Camilla Blue is a Dutch band, led by its singer-songwriter and namesake, Joyce Deijnen. She is a singer, writer, artist, and pianist. She writes and arranges all the songs the band sings, as well as playing piano or guitar to enhance her singing. Joyce is known for not sticking to any particular genre or musical styling, instead, she is taking inspiration from a wide array of different musical influences. In this way, she says, “each song creates its own world.” For her music, Joyce often pulls from her own life experience to write the lyrics and establish the themes that make up each song. These experiences include her relationships with others, such as ex-boyfriends or her parents. Next to that, Joyce’s history within the Dutch foster-care system is also a topic she incorporates into her songs. Joyce was placed in a foster family when she was 2 years old. The first song she ever wrote was about her situation and foster care (‘’When I get home’’). She did not include this song on her debut album. This is because the song was too sensitive. The song is a tribute to her foster parents. Camilla Blue is the ambassador for ‘Pleegzorg Nederland’ “stichting Oosterpoort” 2019/2020. Through this ambassadorship she hopes to connect different foster kids, kids who had similar situation understand each other. Joyce organized the “foster festival”. This is a festival where foster kids could meet each other. Sometimes talking to somebody who had a similar situation works better than a psychologist (Libelle, n.d.)
She says that the musicians who master the instruments improve the settings. Throughout this collaboration the band Camilla Blue is formed. Her compositions are written and associated with colours. Joyce connects the way of playing, the intensity, and the expression with colours. That is the same way she communicates with her band members.
The Effenaar
The Effenaar is a music venue in the centre of Eindhoven. Since its start in 1971, it has grown into one of the larger music venues in the Netherlands. After a big renovation that lasted between 2002 and 2005, the venue upgraded and now possesses two music halls. It also grew into a smart venue, meaning that the Effenaar researches new ways to be innovative in the entertainment industry. That Smart Venue department has also given rise to this specific assignment. Apart from just music, "Effenaar Smart Venue” is also a space for innovative and high-tech experiments in the field of virtual and augmented reality, full-body scans, 3D printing and big data (Wikipedia, 2020).
Dutch Rose Media
Dutch rose is a media production agency that specializes in AR and VR, for a large array of different products and productions. A few examples of their work include:
- A large amount of AR work for different brands and corporations, helping them bring their ideas into a virtual 3D dimension.
- A 3D house tour into the historic Huize Bergen Estate, a massive estate where people can follow a tour and get transported back through the estate’s rich history.
- A mixed-reality escape room for the Effenaar, where they cleverly blended the real and fictional world to give depth and engagement to the escape-room experience.
- The Effenaar Experience app, an app with which users can scan certain books, flyers, and playbills to enhance the experience of visiting their favourite artists.
Target audience
Camilla currently has an audience that consists of people mostly from 30 and up, as well as some of her closer musician-friends and her family. This is an audience she has said to have mostly amassed just by putting out music, as well as word-of-mouth and interactions on social media. However, Joyce herself has admitted to wanting to get in touch with a slightly younger demographic. As she said, she wants to reach more people between the ages of 25 and 30 with her music and believes her next album might be her way to achieve this goal. By writing music that is more in touch with relatable emotions and concepts, the music has a wider and more understandable reach. This would make this next album more attractive to this younger audience.
The AR and VR technologies have been found to be very successful within this age group. As written by Sol Rogers in his 2018 Forbes article Why Millennials Need VR: “For millennial, happiness is about creating memories through experiences. […] All these sensory clues heighten the emotional link with the experience. As a consequence, the memory of the experience is also heightened,” (Forbes, 2018). The fact that creating memories is such a core strength of the VR experience is shaping up to be a large part of the success of our experience if we keep our audience in the back of our mind when designing this user experience.
The team
Our team consists of three ICT students: Aidas, Mitchell and Shihaab, following the specialization creative technology.
Aidas completed his semester in media design before this semester. As he says himself, his expertise is in creating visuals as well as in software.
Mitchell followed his technology trajectory before this and is very interested in VR and AR technologies.
We share this project with five other students of the Fontys-minor Transmedia Design for Creative Industries: Noesch, Gene, Paul, Anne, and Stefan. Having this immense creative involvement has a favouring impact on our creative process.
We combine forces and take this project on as one group.
Vision
You express your vision on future scenarios through research of the evolution of mankind’s relationship with technology.
We created a manifesto for this semester in a workshop with Geert Jan. By reading an article about “Advice for a creative technologist”, I quickly got some points that I wanted to put in the manifesto that are part of my vision about how I want to see the future of art and technology: respect the art and the artist; put yourself in the user’s shoes to see where their hesitations might be; contribute to open-source initiatives.
Before this project, I had never heard about a chronosphere or volumetric video recording. Nor had I ever seen a Hololens in real life. This made me realize that this kind of technology is no longer a dream and is being used for commercial things. This had an impact on my vision of technology, that maybe in a few years, it will be just another smart device you can buy at the Mediamarkt. The Hololens is already being used in the medical sector, where doctors normally depend on 2D pictures on computer screens, but with the Hololens 2 it is even possible to quickly view CT scans. Combined with eye-tracking and speech recognition the interactions are even faster.
So, I believe that this kind of technology definitely has an impact on the way we will live in the future. This impact can be very positive in many ways, but there is also a big danger that this can have a negative impact on a lot of people. A good product is not always good for everyone. Big, money-hungry companies may take advantage of users and can get away with it as the laws are not up to date with the vast growing field of new technology. This should raise a lot of ethical questions on how we design our future technology.
“Technology ethics is the application of ethical thinking to the practical concerns of technology. The reason technology ethics is growing in prominence is that new technologies give us more power to act, which means that we have to make choices we didn't have to make before. While in the past our actions were involuntarily constrained by our weakness, now, with so much technological power, we have to learn how to be voluntarily constrained by our judgment: our ethics.” (Green, n.d.).
We must be careful with our creations, as we have more freedom and fewer constraints. A lot of new applications and technologies have evolved from our dreams. This is someone’s idea of perfection, someone else’s judgment, but that judgment differs for all of us. We all have a different view on what is good and evil, what is ethically responsible. We should bear in mind that certain decisions will not be restrained by this new order of technology. The consequences that this new technology brings about are grave, as we do not have the ability to control the behaviours of its users. We should not only watch the intentions of a product, but also the corollary.
Innovative concept
You discover and define an innovative concept using creative thinking techniques combined with research of technology, context and human.
The live performance will be enhanced by augmented reality, this will be done via a Hololens. Our stakeholder the Effenaar informed us that for the live show, they will make use of a new version of the Hololens 2 combined with the 5G network. Fifty of these devices will be available for the live event; the event attendees will get these at the beginning and, in the best-case scenario (without covid-19 restrictions), can walk around freely. The Hololens will create a mix from Joyce and the on-stage band in real life and a virtual layer that is generated by the Hololens. This virtual layer will bring the music to life with animations and will tell a deeper story. This story will follow the songs and what the album ‘Yellow’ stands for. The live show will contain five different songs of the album, played after each other. To tie these songs together, we as a team came up with the idea to recreate the bedroom of the younger Joyce. Her album ‘Yellow’ is about her growing up and becoming an adult, growing from ‘Blue’ to ‘Yellow’. This room will be on the stage and will adapt according to the songs. For example, the song ‘Umbrella’ is a dark and cold song, heavy rain can be seen in the window of the virtual room. Umbrellas floating around and lightning flashes of a storm outside. This will set the ambience in the room, adding a deeper understanding of the lyric. A little girl, the younger version of Joyce, will be playing in the room and we see her grow up throughout the series of songs.
Concepting an album with augmented reality is on its own an innovative idea; augmented reality creates a new way to express a song or anything really. The power to control and create a new reality for people to experience that can be interacted within the most natural way is ground-breaking. Therefore, I think that this kind of experiences are the new way to approach live shows in the future.
Interactive user experience
You design an experimental interactive user experience that investigates a technology’s potential and its mutual effect on people.
To develop a deliverable prototype for Joyce I first needed to get familiar with the tech we were going to use. I did this with proof of concepts. First, I wanted to deploy an application on the Hololens so that I had a start to build from. This took some time and iterations. This process can be seen in the ‘Process Camilla Blue’ document under the header ‘Getting started with AR in Unity’.
The Camilla in Chronospheres project is one big user experience, where the event attendees will experience a live show that is enhanced by holographic visuals. Although our concept does not involve any interaction between user and technology, there is a great part of the effect the technology has on the user. If we were to have additional time to further work out this project, interactions are definitely things we have thought of. If, for example, bubbles were to float around the viewer, it would only be natural to try to tap them and see if it works and completing that expectation would enhance the experience even more and makes the deep augmented reality even deeper and more vivid.
To create the live show experience as best as possible, two points are vital; immersivity and connection. To enhance the immersivity of the experience, as said previously, a Hololens is used. This is an augmented reality(AR) or mixed reality(MR) headset. We deliberately chose AR/MR instead of virtual reality. Mixing the real and virtual layers will create a profound experience, where the witnessers establish a new connection with the virtual portrayed story of Camilla Blue. By placing the child bedroom on stage, we introduce the childhood of Joyce to the event attendees and strengthen the connection between performer and viewer.
When choosing a platform for the creation of a mixed reality application, we had the following options: a regular phone or a mixed reality headset. The phone would be the best for accessibility; everybody has a smartphone. But using a small screen and holding it up is not really immersive. A mixed reality headset is fixed on the attendees’ head, able to create a virtual world on top of the existing. This is immersive, hence why we chose to use a mixed reality headset.
We chose to use the Hololens, considering the known documentation and support of Microsoft. As well as the fact we have the availability to lend this from the ISSD.
After experimenting with the volumetric video recording, I found out that the Hololens first generation cannot perform at an optimal frame rate. As seen in this video, the frame rate defeats the immersivity of the experience. Therefore, we need to use more suitable hardware; the Hololens second generation. This is the improved version of its predecessor with more processing power that can handle this kind of files.
As said under the header ‘target audience’, “[…] As a consequence, the memory of the experience is also heightened […]“. Using virtual, or augmented, reality for the live show will give the user a better and memorable experience.
Prototypes
You collaboratively realize and test a series of prototypes and their potential, for both users and stakeholders, in a real-world application.
I made numerous prototypes while researching and developing with the Hololens and Mixed Reality.
These prototypes are high and low fidelity. One of the first prototypes was with the Hololens; to get an application on the Hololens was step one. I also had to do a lot of research as I was not familiar with this kind of technology. The process of the Camilla Blue project can be read in the document ‘Process Camilla Blue’, the prototypes are also in this document.
Unity Application on Hololens in 3D with Spatial Perception
Unity Application that is not laggy on Hololens
First XR Unity Application on Hololens
Spatial Processing in Unity with MRTK
Spatial Processing in Unity with MRTK v2
Spatial Processing in Unity with MRTK v3
Hi-fi Prototype, Camilla's Room
We as a team decided to split the work up. We created a storyboard for one or two songs where we would describe what would happen in the live show. This is part of the low-fi prototype I made and can be found in the document ‘Process Camilla Blue’ under de header ‘The low-fi prototype’.
Below on the left is the first screenshot I made on a Hololens, on the right is one of the last I made.
Below is one of the iterations based on feedback. When I placed Camilla in the application, the loss of frame rate is very noticeable. The feedback I got from our group and teachers was that the loss of frame rate breaks the immersive experience. On the second video is the application without Camilla and as can be seen, the frame rate is very high.
Video with loss of frame rate
Video without loss of frame rate
Testing
Concepting and research have been a huge part of this project. Getting started with the Hololens and having to think of the concept and how people should experience this live show took some serious time. Therefore, we will not be able to execute a test with our prototype. I will however make a test plan containing what and how the test would be completed.
To check whether our prototype achieves the intended effects from our stakeholder’s perspective, using appropriate methods, I first need to clearly state what the intended effect is.
The prototype should showcase the possibilities of a live event experience enhanced by the tools of augmented/mixed reality. The intended effect of this enhancement should be an immersive experience where the story of Camilla would also be told.
To test this statement, I would use the following CMD methods from the DOT-framework (https://www.cmdmethods.nl/); thinking aloud and A/B testing. I also want to give a brief questionnaire after the A/B testing to get a hold of the reasoning.
By using the ‘thinking aloud’ method I would get an insight into what the user is thinking through the experience. As cmdmethods.nl states; “As thinking aloud is quite unnatural, you remind users regularly to keep on voicing his/her thoughts.”, I should remind the users to think aloud and put them at ease. By playing out both, a prerecording of Camilla’s show without a Hololens and the prerecording with our Hololens deployed with our prototype. The collected information will answer the question; does our prototype enhance, or better, the experience and portray Camilla’s story?
With the A/B test, a more direct answer will be given. In the execution of this test, the user will be viewed the prerecording of Camilla’s show without a Hololens and the prerecording with our Hololens deployed with our prototype. After the user is confronted with these two scenarios, they will choose one. With the questionnaire in play, an argumentation will give more insights into why this scenario is chosen.
If the Hololens is dominant in the outcome of the A/B test, we could assume that this prototype does in fact enhance the live show. A factor that should be taken into consideration is that giving people a choice between a prerecorded show with and without a mixed reality headset may give them a bias opinion, as this is very new technology and people may think this alone will be an experience. Then again that is the intention. But having a questionnaire where they give an argumentation on their choice, would clarify this.
Thinking aloud would give an understanding of what the user would experience and recognizes/interpret from Camilla’s virtual story. If the user understands more of the story with the Hololens and experiences more than without, this will mean the product achieves the intended effects and this will be considered a success.
Conclusion and reflection
A creative technologist, this has a new meaning for me now. My perspective of ‘creative’ has been broadened, with which I mean I now believe creative is not coming up with something original, it is much more than that; it is a way of thinking and how you connect dots in your mind and a lot more than I can think of. Technology is growing incredibly fast and has already a big impact on the lifestyle of humanity.
The most interesting part about this fast-changing field of creative technology is about the choice one has to make between creating with the purpose to better the world or with the purpose to make a profit. The incredibly large number of different possibilities that the future of creative technology holds for us is dazzling. What I want to achieve with my work is to make sure that these possibilities are morally and ethically responsible and change the world for the better.
As I said, I initially wanted to follow the software direction, yet now I realize that I really know for sure that changing it to media was the best decision here on Fontys. I still want to create, but with meaning. And this specialization for me follows that, and it really fits for me.
Thinking back to the beginning of the specialization, I remember the regret about making this choice. Hearing from game design and seeing their playful projects and presentations made me think I made the wrong decision. Then I realized, our playfulness is in the freedom we get and the support we have from the teachers to explore and unfold our interests within our chosen direction. I am happy that I have chosen this specialization and was fortunately gifted with the freedom our teachers have given us. Discovering this new direction gives me hope in finding the things I am so eager to learn; what I want to create, discover and dream of in life. It has given me the motivation of pursuing the combination of humans and technology. But maybe even more important; I should trust my first instincts, as I should have done with choosing media, instead of thinking that I must continue the software direction after a software MBO degree.
Planning
Struggling with making and following a schedule is something I had trouble with before. As most portfolios follow a more linear increase of content during the semester, mine is mostly flat at the beginning and middle. But when nearing the end, I made a sizable recovery. I talked to Mijke, one of my teachers, about the trouble of not following my schedule. Mijke said that once someone asked her; “Is it a problem, that you don’t follow your schedule?”, she told me that it was not a problem for her, she still got the things done that needed to be done. It is about finding out what works for you. And it made me think; I always thought not following a schedule or not making one is the problem. But it is the uncertainty that I dislike, the discomfort of not knowing what my progress is and the current status of my assessment. Contacting the teachers and asking for feedback on my work so far has helped me defeat this uncertainty. Thus, I should be more proactive with getting feedback on my work and contact the teachers sooner and frequenter.
The sister of my girlfriend, who studies for a bachelor’s degree in communication & multimedia design, explained how she creates a global schedule combined with a plan of action before a project, this gives her an overview of all the tasks.
Creating a global schedule at the beginning of a project can give me a convenient overview during the realization phase. Scattering the design process steps over the weeks will chop up the load and makes it easier to digest. I certainly want to try this method for my next semester, having a better overview may help me focus more on the project.
Presenting
We had multiple meetings with Joyce (Camilla Blue) about our progress and to get feedback. In the first meeting, we discussed the child room concept with her. She immediately became very enthusiastic and came up with all sorts of ideas. One was to have the viewer zoom in on her (chronosphere footage) eye and then into her head. Here we would see what was going on in her head; she proposed one of those monkeys clapping with instrumental discs. I am certainly happy that we inspired Joyce with new ideas by presenting our concept, which has to mean something.
My experience
As cliché as it sounds, this has been an educative semester for me, where I got the freedom to learn about and empiricism my personal and professional interests. I am at the stage where I am still unravelling myself, this freedom to study; me and how I fit in this big, boundless world around me is what motivates me into this daunting journey.
Regardless of the pessimistic effect of the lockdown, I still felt a connection with my fellow students, teachers, and my specialisation. Which was not the case in my previous semesters. Having such a diverse team and special project gave me a lot of energy to move forward with our concept.
I adopted the leading role in the end of my previous semester, I also felt the responsibility to take the lead in this project. I was not happy with the outcome of the project in my previous semester, we as a team did not meet up often and we were not diligent. To prevent this from happening in this semester I took this leading role. To my surprise my team was very proactive, we took the precautions so that we did not procrastinate. We set various deadlines and planned meetings, and this helped for me.
The relation between student and teacher shifts over the semesters. Gradually, over time, the roles have reversed. Where first the teacher was the expert, the student has now taken over this skillful position in the project.
“You can actually do it”, a soothing message, but I have heard it at a disturbing timing at the end of the last three semesters. This confirms the need to be proactive with contacting the teachers and showing my work. Because I can do it, and I even though I could not complete a MBO study. Yet, here I am in the fourth semester of HBO. I deserve to be prouder of my work and my achievements, and most importantly I deserve to be prouder of myself.
Thank you for giving me the opportunity and freedom to discover myself and my interests