Have you ever finished a composition assignment using digital tools like sound recorders or cameras? Or have you ever assigned an audio documentary or a video adaptation of some plays? In our English class, everything can become possible.

In the study of Selfe and Wright’s (2007) Words, audio and video: Composing and the processes of production. Multimodal composition (pp. 13–28), the authors shed new lights on how multimodal composition can be applied in EFL classroom. This research mentioned two main facets of composition with alphabetic texts and digital formats. Firstly, it introduced the process of composition with words, audio and video, each has three moves:

  • Move1: Putting words/ audio/ image or video to a digital format;
  • Move2: Organizing, revising and editing words/ audio/ video resources.
  • Move3: Sharing and publishing in each format.

The similarity is that in the first move, all three kinds of compositions include thinking about the purpose, audience and form, as well as planning and brainstorming various concepts in a word-processing program. Differences among three kinds of composition result from the different digital equipment they use and the nature of the composition. Each composition should use related tools to transform themselves into digital format, for example, typing or scanning words into computers in word composition; recording and downloading audio material into computers in audio composition; shooting video and taking pictures in video composition.

Secondly, there exist some challenges that teachers will encounter when teaching students to compose multimodal texts that contain video or audio. For instance, students may need to learn how to think about, choose, and focus on topics in ways that take advantage of the particular affordances of sound and video; they may also need to learn more knowledge about technologies, like how to operate digital equipment and compress large files.

Here is an example of multimodal assignment-composition with audio. In an EFL class, the teacher asks students to compose an audio essay which allows students to tell about their personal literacy practice with the help of digital sound recording equipment. Students are also allowed hand in their written documents, including detailed plans, storyboard, interview questions, record citations and personal reflections. An evaluation sheet is also attached for students to refer to.

(the screenplay we wrote in Joanna’s class)

In our MES program, we have similar assignment for students to compose with video. Last semester, Dr. Joanna Mansbiridge requested students to make a film project in groups. In the process of creation, students need to develop a story and write a screenplay for the scenes and plots. Students are also asked to draw the storyboards for the stories and made a portfolio for the project.

(the storyboard we drew in Joanna’s class)

In the assessment of the assignment, the projects are evaluated not only by the coherence of the story structure, the format of the screenplay and the development of the characters, the demonstration of the dialogues and clarity of the storyboard, also the application of the film techniques, such as cinematography, editing, lighting, setting and music are taken into account.

Through the process of the project, students successfully learned and developed the skills of composition and technique knowledge of filming.   Meanwhile, the video presented in the final step is a great demonstration of the learning outcomes.

Here are some questions for you:

  1. Do you think it is important to apply multimodal composition to EFL classes? Why?
  2. What are the advantages and limitations of multimodal assignments?

(Edited by Lux and Elle)

37 thoughts on “Multimodal Composition: Words, Audio and Video

  1. Hi, Lux and Elle! I had the same experience as yours in Joanna’s class. It is a tough but really interesting and fruitful task, I think. Undoubtedly, through the process of multimodal composition, learners can not only improve writing and speaking skills but also develop the ability of using multimodal tools. Besides, students might get more enjoyments and enthusiasms from this specific kind of language tasks. Therefore, I think multimodal composition might be a good choice in EFL classes. However, there are some drawbacks or challenges for applying this novel approach in language education. Firstly, just like what you have mentioned, besides language knowledge, students need to learn more about digital tools. Besides this, in order to avoid time wasting or the distraction from digital tech. students have to draw a clear distinction between the primary purpose, language learning, and the minor focus, application of technologies. By considering these, I would prefer to apply multimodal composition to EFL classes as a group task for college students or adults, because those learners might have more knowledge about tech or better cognitive competence for accepting tech knowledge. Additionally, they might have better self-controlling ability.

    • Hi Michelle, I totally agree with your point. It is true that learners need to have a basic knowledge about technology, otherwise, it would take students a lot of time in struggling with the technology thing. What’s more, in order to effectively use this approach, the teacher is required to have a good understanding about the theoretical knowledge over multimodal composition.

    • Ah, ha! Michelle, as well as Lux and Elle, we had the similar experience in Joanna’s Studies in Films and Literature Course, quite interested and fruitful experience!
      Referring to whether we should apply multimodal composition to EFL classes, I support your points of view. This approach is more likely to be suitable in the learners who are in a upper level and have better self-control ability. For younger learners, they might distract on other aspects rather than focusing on language learning and also, they may be too curious about new tools in learning process to lose original enthusiasm on knowledge learning. Besides, their comprehensive skills are supposed to improve before they apply multimodal composition in their language learning.

    • Hi,Michelle, thanks for your reply. I agree with your point that this kind of multimodal assignment which related a lot to the technic knowledge should be assigned to be completed in group. In this way, students can work out the technical problems together.

  2. Seeing that nowadays students are involved in the world with media literacy, it is sound to let students have a try on multimodal composition on the English class. Learning will be more real and permanent when it is linked with learners’ daily lives except for the academic scope.

    I suppose that the advantages of multimodal assignment are as follow:
    1. Help students develop a further and deeper understanding of the knowledge, given that they need to transfer the knowledge into a totally new task;
    2. Offer an opportunity to students to be familiar with new media, getting them catch up with the development of the digital world.
    The limitation of multimodal assignments may be that not all the students have the fundamental knowledge of how to operate multimedia devices, and not all the students are able to afford the devices.

    • Hi Laurel, the two affordances you mentioned are great. I agree that by designing the task skillfully, such as by making it linked with some content-based subjects, students will need to recycle and enhance the knowledge that they have learnt in the new task. Also, the project gives students a chance to explore the new media. They will realize how learning can be connected to their life, and they will also see the practicality of English (such as for authentic communication with the audience on YouTube or other online platforms) in a new light.

  3. Hi Lux and Elle, in relation to Q1, I don’t think most teachers would ask students to create multimodal compositions very often because it is rather time-consuming and sometimes students are obsessed with technical issues more than the language task itself. However, such a project can be desirable. For example, once the NET teacher and I asked one of our weaker classes to video-tape their performance of a ready-made drama script. Students were told to do so because they were uncommitted in class, and their oral skills were lacking. The multimodal composition turned out to be a success, since students were much more serious in producing the drama than they were in class, and we also realized that those F.5 students were quite experienced in making videos (such as for the promotion of the Student Union or for fun) than we expected. They could put on humourous or dramatic scenes quite readily by considering “the purpose, audience and form” simultaneously without a lot of prior preparation. They made use of their voice, facial expressions, the setting, background music and noise in amazing ways. So I guess we shouldn’t underestimate our students’ ability and interest in making these multimodal compositions. The biggest limitation might be that you can’t ask students to do it too often, since the project takes up much time.

    • Hi, Camie, thanks for your reply. Through your case, I think of another affordance of multimodal assignments for students though it is really time-consuming. That is you can get every student’s voice since they are going to present their works. Like you said in your experience, it’s amazing to find the achievements. Sometimes, in class, you might miss some students performance due to their lack of participants because of the introverted personality. However, in this kind of assignment, you might get surprise!

  4. Hello, Lux and Elle. After seeing your example in Joanna’s course, I find that in this kind of multimodal composing task, teachers need to provide the student with proper scaffolding and guidance for the successful outcome, like to develop a story and write a screenplay for the scenes and plots and to draw the storyboards, etc. Of course, your given example is of high difficulty. If for adolescent EFL learners, the multimodal composition task should be set easier, such as storytelling.
    Advantage: 1. Help student foster not only their writing skill but also digital literacies.
    2. Allow students to be creative and more engaged.
    Disadvantage: 1. Time-consuming 2. Need guidance on both the writing process and also the digital techniques, resulting in more workload on teachers.

    • Hi Eva, thank you for your comment. I agree with what you mentioned about the advantages and disadvantages of multimodal composition. Through this work, students can have access to a creative and authentic composition. I think the teacher really plays a very important role in instructing students how to use digital equipment, and how to make better use of them. It is time-consuming for both students and teachers, especially for students who are unfamiliar with techniques.

    • Hi, Eva. You have mentioned that project based approach could be time costuming, which I have never thought about. It may cost a lot of time if the students lack proper guidance.

  5. Hello, Lux and Elle. I do think it’s important to integrate multimodal composition to EFL classes, bcause it is interesting to write screenplay and related images. They are sort of crsftmanship which can attract students’ attention and in this kind of process, students interact with each other to distribute workloads and handle disputes occured. Then they can master some essential skills for better interpersonal relationship in the future. Multimodal composition also provides students with authentic materials and topics, they are not passive receivers anymore, they become active participants, which is beneficial to learner autonomy. While the drabacks of it may be out of time limit and technological skills to back up the whole process. This way of teaching is quite demanding from the aspects of teachers. Some middle-aged teachers are reluctant to incorporate it due to age and lack of confidence.

  6. Hi, Lux and Elle. I do think applying multimodal composition to EFL classes would be an innovative way of learning language. It would be a good attempt to incorporate digital literacies practices into language teaching. The advantages of this teaching method could be as follow: 1) give people opportunities to develop digital literacies 2) enhance students’ learning cooperative competence when they work together to finish the task 3) improve students’ creativity and motivation. Despite that, there are some limitations that is worth noticing. For example, given the student-oriented nature of this task, the effectiveness of this method may be influenced by many subjective factors, like students’ motivation. In addition, they may not be competent enough to utilize such a vast array of digital devices. In view of this limitations, to carry out this method requires teachers’ appropriate guidance and explicit instruction about digital literacies.

    • Hi Evelyn, thanks for your comment. I think what you mentioned about the nature and limitations of multimodal composition are noteworthy. This pedagogical approach is student-oriented, so that students themselves need to be highly motivated and engaged in order to do well. Besides, teachers should considerately guide students through the task in order to equip students with basic competence to finish the composition.

    • Yes. I totally agree with you. And I think the teachers is also required to act as a planner of the project.

  7. Hi, Lux and Elle! I think the multimodal composition is not as important as other approaches exploited in language teaching. As you referred above, it will set a technical challenge for students to finish such an assignment, which is a waste of time for the purpose of learning a language. After all, students in EFL classroom as well as their parents will put language acquisition in the first place rather than the way of using a specific tool.
    However, as far as I am concerned multimodal composition can be used as an auxiliary approach in language teaching because of its advantages. First, it motivates students’ learning interests as a new assignment form, for the features that it well combined their learning process with imagination or the strengths in other aspects so that students’ confidence can be built. What’s more, students’ four basic abilities also can be practiced in this process. The limitations mainly lie in the technical problems students may come across, which will easily discourage their willingness of learning.

  8. HI Luc and Elle. As mentioned in other posts that by using this tool to transform English learning performance into various digital production such as some video clips, it could allow students to constantly replay and revise the production in a self-paced way. And accordingly, the technical problems involved in terms of teachers and students are also enormous particularly when doing a digital project. For teachers, I think we should customize the content to in relation to the learning material as much as possible, and then bring them back to class to further analycize what students have done forming a recycle of learning for several times in a sense that the knowledge and information in this learning can be better reinforced.

  9. Hi, Lux and Elle. Thank you for your post. I believe multimodal composition is useful in EFL classroom. There is no denying that it takes students much time to learn about technology, but it is worthwhile to invest time and efforts in this practice, because the proliferation of digital tools has made it a requisite skill to make use of technology. What’s more, as long as students are familiar with these digital equipment, they will not spend much time on it.

    I, as an English teacher in a senior high, was ever designated to organize a word contest in the school. Then I asked five students in my class to help design the contest in the form of powerpoint. They were responsible for determining the test scope, selecting the phrases, sentences and passages to be tested, and then transforming them into a digital format with sound, animation as well as videos. Their ability to employ digital tools was far better than expected. Furthermore, in the process of making a multimodal composition, they were given the chance to think careful about these vocabulary, including their meaning and usage. For example,they produced a video in which they performed different actions and the contestants were required to describe what they were doing in complete sentences with given words. They must have acquired these words in order to prepare the video. So in my view, students can really benefit a lot from multimodal composition.

    • Hi, Eric, I agree with you that technological skills could hinder or facilitate the learning process. But, although we would not find it difficult to deal with the technical things once getting familiar with this, it would not save much time on tasks. more time will be spent on revising, adjusting and polishing the work. Anyway, it is good and worthwhile for people to constantly strive for perfection.
      last but not least, your experience in teaching is really rewarding and insightful, which set a good example fo me.

    • Hi Eric, thank you for your comment. From your own teaching experience, I notice that to some extent, students could think more deeply and have more sophisticated insight into the knowledge itself in order to finish their multimodal composition task. Therefore, I think it is another advantage that such kind of new literacy practice can bring to student.

  10. Hi, Lux and Elle. Thank you for your post. After getting to know the main idea of multimodal composition, I think it could be a good way to arm students with digital literacies and lead them to learn the target language with multiple modes.

    On one hand, multimodal composition has some merits. It could not be denied that multimodal composition provides students with non-linguistic modes during the language learning, which could stimulate students interest and motivation to some extent. What’s more, with the numerous amount of digital resources, students need to improve their critical reading and critical thinking skills in order to make sure the accuracy and reliability of the materials.

    On the other hand, the demits should not be overlooked. As Camie and Eva mentioned above, this is a time-consuming approach. The students might have to spend much time in figuring out the technology thing. What’s more, in order to maximize the effectiveness of this approach, the teacher has to spend more time in designing the course than a traditional one.

  11. Hi Lux and Elle, thank you for your sharing. I think that applying multimodal composition into EFL classes is important. Because it will make student better learners through providing a more interesting and motivating learning environment, and it also promotes cooperative and participative learning among learners.

    The major benefit of multimodal in EFL classes is that it can promote students’ autonomy and stimulate self-learning. As it provides them opportunities to find the most comfortable ways of learning, which is a challenge in the traditional ways of learning. Besides, it can enhance learners’ engagement and motivation in the class, as students are highly participative in the multimodal classes comparing with the single designed lesson, which consists of lengthy texts without images. Furthermore, multimodal in EFL classes allows for creativity and flexibility in teacher-student interaction and can enhance the learning environment, and produce a creative learning experience and increase the efficiency of language acquisition.

    But as for making the multimodal composition by students themselves, I think it is both challenging and time-consuming for secondary school students, especially with the current exam-oriented learning outcomes in most schools in mainland China. But it is possible for the higher education students.

  12. Hello, Lux and Elle. From your post, I have a better understand about how to accomplish a digital video, from searching about how to write a screenplay, to edit it with some artistic skills like lighting, setting and music. I think making a digital video can be really beneficial for not only writing skill, but also some digital literacies. However, for Q1, whether it is important to apply multimodal composition to EFL classes, my answer may be no, at least for very young learners.
    As Michelle has mentioned, besides language skills like writing or speaking and communication if it is a group project, there are many digital tools and literacies the learners have to learn during the project. The learners can be enthusiastic towards the project, but they may spend more time on these digital tools, and pay much attention to whether these multimodal composition is successful, rather than the language they use. For a EFL class, it may not be so profitable. But I still agree that it is a interesting project that adults learners can approach to, for they need more exposure rather than skills for test.

  13. Hi, Lux & Elle, thanks for your sharing.
    Absolutely, it is very important to apply multimodal composition to EFL classes. As we are in a digitally-mediated society, we have to face a number of multimodal work in our daily life, for example, in education context, we will be required to do a presentation to reflect what we have learnt through power points and in social context, we might be attracted by the multimedia information screen in the shopping mall. Therefore, understanding and being able to working on a multimodal composition project should be well involved into EFL classes no matter for teachers and students. On the one hand, it would help instructors to instruct effective in utilizing multimodal composition, on the other hand, students would be allowed to better adapt to the digital society.

    It should be admitted that multimodal assignment is conducive to learning to some extent. Firstly, it gives students opportunities to independently select the forms to convey their ideas, motivating them to contribute more on assignment work. In addition, multimodal assignment is usually assigned to study groups, connecting learners closely to enhance their cooperation ability and peer relations. As for its constrains, as it is one digital-tool-based project, many students might spend much time on dealing with that technological tools, distracting them from other preparation work.

  14. Hi, Lux and Elle, thank you so much for your posting on the multimodal composition. You did an excellent job in introducing this approach, and I think it have something related to the multimedia and multiliteracies. As to the whether it is of significance to apply this into EFL learning and teaching, the answer is absolutely positive. Based on the researches, it is common to see this approach applied in the last stage of learning, which is included as a stage of output process. EFL learning should be dynamic and intriguing, since English is a language functions the best when put into practices. Such an approach provides learners an opportunity to integrate English learning and technologies, so that what have been imparted could be turn into pragmatic and practical knowledge.
    As to the multimodal assignment, I think the goods are several. First, it could cultivate students’ characteristics and personalities in the process of self-discovery. When accomplishing the tasks, learners have to face with decision-making. What they choose represents what kind of person they are. This is a process to find yourself, your interest, your eager, and your potentials, which could be really rewarding.
    The second one concerns more about the personal skills and their profound influence. The earlier they are exposed to diversified skills, the more competitive they will be in the future career. Technologies have affected deeply the way we work, live and think. So people who are skillful would be at an advantage in job hunting.
    As to the limitations, whether multimodality could be applied depends on the digital and technological devices availability. Students in underdeveloped areas or countries may not have the chance to experience such a high tech-based approach.

    • Hey dear Nicole, I agree with you that EFL learning should be dynamic and intriguing, thus multimodality should be applied into the teaching process. It will broaden their horizons and enhance students’ ability to analyse and compare information. Another benefit of it is that there is a large room for imagination, which help students to develop the cooperation spirit as well as exercise skills.

  15. Hey Lux and Elle, thanks for sharing such an interesting topic.
    I have quite similar experiences of film adaptation when I was a junior student. At that time, we are required to make a film based on the story Blackmail written by Arthur Hailey. Students are given the access to make several changes about the story and incepting manifold sound or visual effects in the video. It’s a really unforgettable experience in my bachelor life, from which I learned the significance of cooperation.
    I hold the view that it is essential to apply multimodal composition to EFL classes and the reasons are shown below: Firstly, it helps students to grab the basic digital tools, which can help them to gain better career prospects and specialized expertise; Secondly, it triggers students’ learning interests and helps to enhance their reading, writing and speaking English abilities; Thirdly, it helps students to develop an appreciation of art, poetry and literature.
    Despite the advantages of multimodal assignments I have discussed above, there are also several constraints of it: Initially, it adds large amount of pressure to students who already have a heavy workload in school tasks; Secondly, it may constitute a distraction of language learning itself.

    • Hi Alice, your experience reminds me of my parallel one when we were assigned a task of shooting a short video based on a scene from the Macbeth written by Shakespeare. However, throughout the whole process the thing that impressed me the most is how much time we spent on choosing costumes, not on preparing the script at all. So we ended up learning very few about the target language. Therefore, I think teacher really plays an essential role in the whole process.

    • Hi, Alice. I believe you are right. Multimodal composition brings with it many affordances, but some potential worries can also be provoked. To my knowledge, it depend on how we teachers use it. With careful planning and designing, the constraints, like consuming much time or distracting students attention from language learning, can be reduced to a minimum. Maybe some people argue that it will increase teachers’ workload, but I don’t think multimodal composition should be and can be used in each class. According to our teaching materials and aims, this practice is encouraged to be implemented when necessary and suitable.

  16. Hi, Lux and Elle. Thanks for your clear post! Although I didn’t attend Joanna’s course last semester, I do find the multimodal composition important in EFL teaching and learning.

    On one hand, we are now living in a society full of advanced technology, which requires us, both as teachers and learners, to take the important affordances of technology into account when considering to improve our teaching and learning outcome. On the other hand, digital and multimodal composition itself is a very intriguing and also special way that is complementary to the rigid conventional way of classroom teaching. When applied properly, it would certainly motivate and stimulate students, helping them improve their English skills.

    However, if not applied properly, the approach might be really boring to students while they conduct the task. Students would find it difficult and complex to combine those technical equipment with their language learning, confused about the meaning and purpose of doing it. Therefore, I think it is essential to equip them with enough necessary information before they actually begin that journey.

  17. As for the task- based teaching style that teachers applied and under-developed technology when I was learning English in school, I have never tried such an interesting activity of making multimodal composition before. However, in order to catch up with social development and make the English study procedure more intriguing,it’s necessary for teachers to include writing multimodal composition in their course plan. Even though it will be a little bit too time consuming for the fast-paced schedule especially for Chinese mainland students as an individual work, it’s possible for teachers to make it as a group work and each group can choose their favorite topic to write about and share. I believe it will be definitely effectively in improving students’ passion in English learning.

    • Hi, Annie, I totally agree what you mentioned about multimodal composition’s positive impact on students English learning. It does motivate students to contribute more on their English learning as it is to some extent the process of free writing practice for students where they are allowed to express their ideas in their preferred form. In addition, to overcome the time-consuming problem, it’s a good idea to assign this task as the group work as you suggested, and from my view , if our curriculum could involve some easy multimodal composition into daily after-class work, the students would be able to effectively do a more complex multimodal composition in assignment.

  18. This case best reminds me of one of the most impressive assignments for me through the MAES project. Last semester in our course of Language, Literature and Culture, our professor conducted a very interesting multimodal composition activity. we could form a group of 3 to 4 students. Each group discussed and chose a literature topic or a piece of work that had been taught before. We were then encouraged to create a new literary form and adapt the original work. For example, we could adapt a poem of Shakespeare and change the means of literary expression from written poems to songs with lyrics, cartoons with plots and stories or even dancing. This activity is very stimulating and creative,. It not only encourages the students to make good use of their teamwork imagination but also serves as a means to check the interpretation of what have been taught. In terms of its disadvantages of extra burden of mastery of new technology, I think that students living in the era of information technology find it much easier to develop their ability to utilize digital tools, which may also be an important skills for their future career.

    • I agree with you, and I did the same project on literacy class too, during the time we adapted our literary work, we did not only adapt it totally same with the original work, we added our own understanding to make it more suitable to be displayed in class by using media assistant tools. And a written explanation of the adaption is required. Although it was a writing task too, it was really interesting. I felt enjoyable when I did my project.

  19. Hello, Lux and Elle. The example you given in Joanna’s course is not easy for EFL learners. It would be better if teachers offer students with enough scaffolding and help ahead.

    As far as i am concerned, this technique is time consuming but is beneficial to students,which may cultivate students creativeness in language learning and application.

  20. Hi, Lux and Elle, thanks for your sharing, I think it is important to apply multimodal composition to EFL class. Firstly, in the future, most of the texts will be changed from paper-based to media-based. Thus, the ability of multimodal composition is very important to students. Secondly, the multimodal composition is really different with the traditional one, students need to know how to read texts combined with different modes and how to create their own text in a new literacy way. Thirdly, the new form of writing may cause students interests since this is what they contact with every online reading.

  21. Hi, Lux and Elle. Thank you for your sharing.

    I believe it is beneficial to apply the multimodal composition to EFL students, they can not only command how to use new technology but also have a better understanding of language. Actually, people nowadays interact with people or reading mostly through digital media, if the classes are incorporated into multimodal compositions, it can arouse the students’ interest and make the language practical.
    They can manipulate language more often, not only a language for a test.

    Regarding its drawbacks, one is time-consuming if the students haven’t practiced that technology before, which will waste too much class time. However, this can be avoided by proper design and well preparation before the class.

Comments are closed.