The LDE CEL Annual Meeting is a yearly event where we showcase our achievements and plans for the year to come. Each year we pick a different theme that we explore together with keynote speakers, workshops and panels. This year’s theme was: ‘Developing a Culture of Learning Analytics'.
After two years of digital annual meetings, finally we could have an annual meeting face to face at the Teaching Lab. We started the day at 9:00 am with the registrations of the participants, some coffee and tea and there was time to catch up before we started. A bit later than planned, due to the hybrid setup for the keynote, Marcus kicked off the day with a word of welcome and what we could expect this day.
We started the day with a great keynote from Australia, Simon Buckingham Shum gave a talk about Cultivating a Learning Analytics Culture at UTS: Boardroom, Staff Room, Server Room, Classroom.
About: Simon Buckingham Shum is a Professor of Learning Informatics at the University of Technology Sydney, where he serves as the inaugural director of the Connected Intelligence Centre. CIC is a transdisciplinary innovation centre, inventing, piloting, evaluating and scaling data-driven, personalised feedback to students, with close attention to the human factors that ultimately determine effective adoption. Simon’s career-long fascination with software’s ability to make thinking visible has seen him active in communities including Computer-Supported Cooperative Work, Hypertext, Design Rationale, Scholarly Publishing, Semantic Web, Computational Argumentation, Educational Technology and Learning Analytics. He has been active over the last decade in shaping the field of Learning Analytics, co-founding the Society for Learning Analytics Research, and helping to catalyse several subfields: Social Learning Analytics, Discourse Analytics, Dispositional Analytics and Writing Analytics.
Abstract: In this talk, I describe and reflect on the last 8 years at the University of Technology Sydney, inventing, piloting and evaluating Learning Analytics tools, specifically, interactive tools focused on data-driven personalised feedback, integrated with the institution’s learning technology ecosystem, and accompanied by staff training and support. I will summarise this as conversations in the Boardroom, the Staff Room, the Server Room and the Classroom, reflecting the different levels of influence, partnership and adaptation required to introduce and sustain novel technologies in the complex system that constitutes a university. The focus is very pragmatic, documenting aspects of our work that are typically not the focus in research papers, although the research-based invention and gathering of evidence is central to our modus operandi, and will be mentioned as relevant. This talk summarises a forthcoming book chapter.
After a short coffee break participants could follow one of the four workshops during our first round of workshops.
Workshop 1 | Digital Skills and CT
Who: Hanno Van Keulen, Marcus Specht, Xiaoling Zhang
Title: Digital Skills and Computational Thinking in Education
Abstract: In this workshop, the motivation for why we need to engage and develop digital skills and computational thinking was discussed. Also, different terms and definitions are used around the term digital skills such as digital literacy, data literacy, algorithmic literacy, or computational thinking. The workshop organizers gave an overview of the current definitions and competencies mentioned and relevant challenges in the field and embedding of digital skills in the curriculum.
The integration and teaching/learning of digital skills were demonstrated with examples from different educational sectors from primary school up to higher education. The assessment of student performance and important competencies and skills often subsumed under digital skills were explained. Third, the workshop has enabled participants to start up a discussion in their own working context on the integration, development and teaching and learning of digital skills in their own curricula.
In this workshop, participants got a 10 minutes presentation by an organizer plus 10 minutes discussion with the audience for each of the following topics:
- Why digital skills? What are digital skills?
- What to assess and how to assess digital skills?
- Example of digital skills implementation in teaching and learning practices.
About:
Hanno van Keulen, Director of Studies / Opleidingsdirecteur MSc Science Education & Communication TU Delft. Technische Universiteit Delft.
Marcus Specht, Professor for Digital Education at the Technical University of Delft and Director of the Leiden-Delft-Erasmus Center for Education and Learning.
Xiaoling Zhang, PhD candidate for Digital Skills at the Technical University of Delft and the Leiden-Delft-Erasmus Center for Education and Learning.
Workshop 2 | Learning by App
Who: Ludo Juurlink, Associate professor Leiden University
Title: Learning by App
Abstract: Altijd al eens willen nadenken en sparren over opzet van onderzoek op gebied van EdTech? In deze workshop werd onderzocht hoe wetenschappelijk onderzoek opgezet zou kunnen worden dat poogt aan te tonen of er leereffecten zijn (specifiek m.b.t. retentie en examenstress) aan het gebruik van een “mobile learning app” door studenten – een app die op hun telefoon werkt. We deden dit aan de hand van een reële casus die recentelijk is uitgevoerd met inzet van de learning app “Cerego” aan de Universiteit Leiden in 3 verschillende faculteiten. Zowel de aanpak als resultaten van dergelijk onderzoek en verschillende typen problemen die het project kunnen frustreren kwamen aan bod.
About: Ludo Juurlink is associate professor bij Leiden University. Hij doet chemisch didactisch onderzoek, is begeleider van DUDOC promotie-onderzoek naar chemische vakdidactiek (2017) en EdTech didactiek met een Comenius Leadership Fellow Award (2019).
Download the slides
Workshop 3 | Analytics and Support for Goal Setting
Who: Gabrielle Martins van Jaarsveld
Title: Designing, developing, and deploying a learning analytics-supported chatbot to support student goal setting
Abstract: The past decades have seen a massive shift in the way we carry out higher education, with technology becoming increasingly common in students' day-to-day learning activities. While these increasing digital learning environments have brought with them a host of benefits when it comes to accessibility, scalability, and personalization in education, they also present their own set of challenges and strong self-regulated learning skills are becoming increasingly important to succeed in these environments. Thus, an important question in higher education has been highlighted: how can we leverage technology to help support students in these increasingly self-directed learning environments? With this in mind, we set out to build a learning analytics-supported goal-setting chatbot, to support and guide students during their self-regulated learning processes. In this talk Gabrielle has shared the process of designing, developing, and deploying this chatbot, and how they intend to use learning analytics to provide students with personalized self-regulated learning support.
About: Gabrielle Martins van Jaarsveld is a third-year PhD candidate completing her research at Erasmus University Rotterdam and LDE CEL. The topic of her PhD is Learning Analytics, with a focus on developing and researching learning analytics-supported goal tools to improve students' self-regulated learning skills and academic performance. With a background in psychology, Gabrielle’s current project takes a multidisciplinary approach guided by insights from the fields of psychology, educational sciences, learning analytics, human-computer interaction, and educational data mining.
Workshop 4 | Using writing analytics to provide instantaneous, formative feedback to facilitate holistic learning.
Who: Ziwei (Jo) Huang | FeedbackFruits
Title: Using writing analytics to provide instantaneous, formative feedback to facilitate holistic learning
Abstract: At FeedbackFruits, we believe that feedback is key to facilitating learners’ growth in a holistic learning experience. However, with more feedback comes more information — increasing volume of academic writing, peer- and self-feedback add to the workload of instructors exponentially. We ask ourselves: how can we ensure that every learner receives personalized feedback on their work while alleviating instructors from unrealistic workloads?
In this workshop, participants learned about several products by FeedbackFruits that seek to address this problem using writing analytics. Participants learned first about the idea, co-creation, and development of Automated Feedback, an award-winning AI-powered learning assistant that provides instant and formative feedback to help students improve their academic writing. Then, participants experienced the tool on FeedbackFruits both as teachers and as students. This includes a beta feature that seeks to facilitate student-directed feedback. In small groups, participants discussed the results they received on Automated Feedback.
In addition to Automated Feedback, participants also got a preview into two work-in-progress features: an automated feedback coach and an insights feature that extracts keywords from student contributions.
In the last part of this workshop, participants were asked to brainstorm ideas for possible implementation of these products in their context to facilitate holistic learning experiences and personalised feedback.
About: Ziwei (Jo) Huang is an Associate Product Manager at FeedbackFruits, whose portfolio includes Automated Feedback and other data products. Previously, she worked as a product and user researcher at various EdTech and consumer product companies. She holds an MSc (Distinction) in applied linguistics from Oxford University and a BA in linguistics (honours) and organizational studies (international education) from Pitzer College. A beneficiary of several educational systems around the world, Jo is a firm believer and advocate for the power of digitalised education to improve individual development and our society.
After the first round of workshops, we had our second keynote of the day. From Germany, Hendrik Drachsler told us about designing highly-informative competency-driven feedback with learning analytics.
About: Hendrik Drachsler is a Professor of Educational Technologies and Learning Analytics affiliated with the German Leibniz Institute for Educational Research and Information (dipf.de), the Computer Science Department of the Goethe-University Frankfurt, and the Open University of the Netherlands. He is also the Scientific Director of the central Technology-Enhanced Learning Innovation unit (studiumdigitale) at the Goethe-University Frankfurt, Germany. His research interests include Educational Technology, Trusted Learning Analytics, Recommender Systems, and Feedback. In the past, Prof. Dr. Drachsler was elected board member of the Society of Learning Analytics Research (SoLAR) as well as a member of the board of the European Association of Technology-Enhanced Learning (EATEL). He has been the principal investigator and scientific coordinator of various national and EU projects and was responsible for a consortium budget of over 45 Million Euro and 15 Million Euro for his research group. Hendrik has chaired international scientific conferences (e.g. LAK16, EC-TEL17, EC-TEL18, LAK2020) and is Special Issue Editor of the Journal of Computer Assisted Learning (JCAL).
Abstract: Highly informative and competency-based feedback is costly and can be best given in small teacher-to-learner settings. According to research, feedback has a powerful effect on learning success; for Highly-informative Competency-based Feedback (HICF), even higher effects are measured. HICF provides actionable feedback that goes beyond correct or wrong replies. It offers the right solutions, possibilities for improvement, hints for self-regulation, and suggestions for effective learning strategies. Such feedback provides suitable conditions for effective metacognitive control of the learning process. Until a few years ago, it was very time-intensive to provide HICF in class; and simply not possible to provide HICF at large-scale university lectures due to a lack of personnel. But with the help of computers and other digital devices, we can design data-enriched learning activities that open up far-reaching possibilities to provide HICF.
Within the talk, Hendrik demonstrated how data-enriched learning activities are designed with domain experts, how the group curates a knowledge base on effective learning analytics indicators, and how the data-enriched learning activities are researched in experimental settings.
After this very interesting keynote, it was time for lunch and networking. To get back on schedule, the lunch was a bit shorter than planned but still, there was enough time to enjoy the lunch.
Right after lunch, we started our second round of workshops. Again participants could choose between four different kinds of workshops.
Workshop 1 | Multimodal Learning eXperiences & Learning Analytics workshop
Who: Bibeg Limbu
Title: Multimodal Learning eXperiences & Learning Analytics
Abstract: This workshop aims to raise discussions at the intersection of two related but distinct domains of Learning Analytics (LA) and Multimodal Learning eXperiences (MLX). Learning analytics (LA) captures the learning process with the promise to foster self-regulated learning in students. To date, LA mostly relies on student data collected from the digital space via learning management systems. This provides only a constrained view of the learning process. Multimodal Learning eXperiences (MLX), however, span across the physical and the digital space, and across modalities which enables instructional design interventions in authentic settings. The LA or MMLA data(in case multiple modalities are used for the data collection) collected can be fed back directly to the Multimodal Learning eXperiences (MLX), or can be presented to the teachers as “Teaching analytics” (TA) to help them make instructional design decisions. In this workshop, you will be introduced to the challenges of enhancing MLX with LA and encouraged to actively contribute towards the design of TA for a specific use case. We aim to provide teachers with easily interpretable analytics and also provide them with flexible tools to improve their teachings.
About: Bibeg Limbu is a post-doctoral researcher at Leiden Delft Erasmus Centre for Education and Learning at TU Delft. In his research he focuses on educational technologies such as Multimodal immersive technologies for expertise development and serious games. He holds a PhD in technology-enhanced learning from the Open Universiteit, Nederland. He received his MSc in Educational Technology from the University of Saarland and BSc in Games technology from Anglia Ruskin University. He also received his Diploma in Multimdedia technology from Hales Institute, Melbourne. He was involved in several European projects such as WEKIT and SafePAT. He is currently working in the project HoloLearn (Holographic Learning) and VR4VET (Virtual Reality for Vocational Education and Learning) where he investigates the use of multimodal immersive learning technologies in vocational education.
Workshop 2 | Time-management and Learning Analytics
Who: Quincy Breidel, Sanne van der Meer & Isabel Bezemer
Title: Time-management and Learning Analytics
Abstract: Het deeltijdonderwijs aan Erasmus School of Law in de vorm van vrijdagmiddagonderwijs is een succes. De groep studenten is zeer divers, met name als het gaat om de (professionele) achtergrond, leeftijd en (werk)ervaring. Veel studenten combineren de studie met gezin, een baan of een andere studie. Het feit dat de groep zo divers is maakt dat de studenten ook behoefte hebben aan onderwijs dat ze in staat stelt om – tot op zekere hoogte – te kiezen op welke manier en op welk moment ze studeren. Door de jaren heen is voor de verschillende vakken in verschillende vormen onderwijsmateriaal ontwikkeld ter ondersteuning van het onderwijs (Flipped Classroom). Echter, deeltijdstudenten geven aan moeite te hebben met het plannen van studeermomenten in combinatie met hun gezin, baan of andere studie. Timemanagement wordt over het algemeen als lastig ervaren door deze studenten en daarom willen wij hen daarin graag beter ondersteunen. Hiervoor is het nodig om tools te ontwikkelen waarmee de student inzicht krijgt in hoeveel tijd bepaald onderwijsmateriaal vergt om tot je te nemen en hoe je vervolgens een goede planning kunt maken. In deze workshop werd samen met de deelnemers gekeken naar hoe een timemanagement tool voor studenten eruit moet komen te zien; welke features zijn bijvoorbeeld noodzakelijk en welke data gebruik je voor de tool?
About:
Quincy Breidel is Learning and Innovation Consultant bij Erasmus School of Law aan de Erasmus Universiteit Rotterdam. Binnen zijn werkzaamheden houdt hij zich bezig met het versterken van het blended onderwijsmodel van de deeltijdopleiding rechtsgeleerdheid en het begeleiden van onderwijsinnovatieprojecten.
Sanne van der Meer is Educational IT-Consultant bij Erasmus School of Law aan de Erasmus Universiteit Rotterdam. Binnen haar werkzaamheden houdt zij zich voornamelijk bezig met digitale innovatieprojecten in het onderwijs.
Isabel Bezemer is Projectmedewerker bij Erasmus School of Law aan de Erasmus Universiteit Rotterdam. Binnen haar werkzaamheden houdt zij zich voornamelijk bezig met digitale innovatieprojecten in het onderwijs.
Workshop 3 | Learning Analytics for Learning Communities in MOOCS
Who: Selma van Esveld and Ali Soleymani
Title: Learning Analytics for Learning Communities in MOOCS
Ali Soleymani, researcher at Leiden-Delft-Erasmus Centre for Education and Learning (LDE CEL) and Selma van Esveld, research coordinator the TU Delft Extension School for Continuing Education, will give a workshop about learning analytics for learning communities in MOOCS. Do you want to learn what insights can be generated from a variety of digital learning traces? Or are you looking for datasets containing these traces to test your educational hypothesis? Then please register and join our workshop.
The learning environments of MOOCs offered by the Extension School collect a variety of digital traces of its thousands of learners worldwide. This creates a valuable dataset for course evaluation and improvement as well as more general educational research. But how to translate the pile of data into usable insights for either teachers or learners? Researchers of LDE CEL, especially within the research topic of ‘Data and AI-enhanced learning’, are skilled to helps learners and teachers make use of these traces for more efficient, effective, and joyful learning. This workshop is a result of the LDE CEL and Extension School collaboration and highlighted the outcomes of the most recent project as well as openings for future collaborations.
About Selma: Selma van Esveld received her PhD degree from Radboud University in Nijmegen, the Netherlands in 2021. The molecular research combined wet lab approaches with in silico analysis to characterize the proteins that interact with mitochondrial RNA. In June 2021, she shifted away from molecular research, but continued to work with big data as research coordinator of the TU Delft Extension School for Continuing Education. In this role she coordinated the data and experiment requests from researchers, the data delivery and translation of the outcomes to practical tips to improve both online and campus education
About Ali: Ali Soleymani is a Ph.D. candidate at the Center for Education and Learning at TU Delft. He has a background in Cognitive Neuropsychology and in his Ph.D. project he focuses on Networked Learning and professional development in high-tech industries like energy management systems. His passion for using and exploring data led him to investigate educational data, especially MOOCs discussion. Using social network analysis, he is investigating these data to have a better understanding of network learning.
Workshop 4 | Peer Review and Assessment
Who: Gillian Saunders, Marcus Specht and Vivian van der Werf
Title: Let’s innovate Assessment: multiplier event for the Erasmus+ project RAPIDE
Abstract: This Multiplier Event is part of the ERASMUS+ project: RAPIDE: on Relevant Assessment and pedagogies for Inclusive Digital Education (https://rapide-project.eu) and was open to anyone who is interested in implementing or improving peer assessment in their courses. At the end of the workshop, participants were able to make an informed decision on a suitable form of Peer Assessment for their courses.
Over the past few years, many of us have faced operating in a frequently changing teaching environment which has made evaluating and assessing students’ learning outcomes and more importantly giving students feedback on their learning much more complicated.
One pedagogical tool that has been increasingly used is that of peer assessments where students give each other feedback and assess each other’s work.
In this workshop, participants were introduced to many different types of peer assessment that can be used in engineering education, such as peer reviewing (each other’s work), peer grading(continuous feedback on mastery), and peer evaluation (group work) whether face-to-face, hybrid or in a fully online environment and how to do so in an inclusive way thus maintaining the important safe place that education should be.
Participants discussed in small groups what types of peer evaluations they use or want to use in their courses and brainstormed on ideas for implementation in their own specific case or for one of the general cases that the facilitators had available.
At the end of the event participants presented their main findings back to the whole group so that they may also learn from each other.
About:
Gillian Saunders-Smits is an Associate Professor at Mechanical Engineering and attached to the LDE Centre for Education and Learning. Gillian is a passionate lecturer and researcher in engineering education and has over 20 years of experience in teaching a variety of engineering courses using a plethora of methods. She has built a solid reputation in managing and organizing education as well as teaching at Aerospace Engineering, be it in project education, traditional mechanics lectures, online education or coordinating a master track. She is a keen engineering education researcher with an extensive international network. She is often invited as a speaker and has over 60 publications to her name. In 2021, she was one of the finalists for the edX prize and in 2022, she was awarded the Open Education Ambassador Award at TU Delft
Vivian van der Werf received the B.A. and M.A. degrees in archaeology from Leiden University in Leiden, the Netherlands in 2013 and 2018, respectively. In addition, she also received the M.S. degree in educational sciences from the same institute in 2018. During and after her studies, she acquired several years of experience in the field of higher education as a Research Assistant, Policy Analyst and Data Analyst, however, she never lost interest in how people learn new skills and how learning can be facilitated. Since 2020 she is pursuing a PhD degree at the Leiden-Delft-Erasmus Universities Centre for Education and Learning and the Leiden Institute of Advanced Computer Science at Leiden University. She focuses her research on programming education. Ms. van der Werf is keen to apply and combine her knowledge from these disciplines in her research projects in order to understand how people learn and how this translates to teaching and education
Marcus Specht, Professor for Digital Education at the Technical University of Delft and Director of the Leiden-Delft-Erasmus Center for Education and Learning.
After the second round of workshops and a small coffee break we started with our last keynote of the day. From New York, Alyssa Wise told us all about, Cultures of Learning Analytics: Understanding and Designing for Variation in Data Use.
About: Alyssa Friend Wise is Professor of Learning Sciences & Educational Technology in the Steinhardt School of Culture, Education, and Human Development and the Director of NYU LEARN, NYU's university-wide Learning Analytics Research Network. Her work is situated at the intersection of the learning sciences and educational data science, focusing on the design of learning analytics systems that are theoretically grounded, computationally robust, and pedagogically useful for informing teaching and learning. Dr. Wise’s most recent efforts center on questions of use and impact, examining how people make sense of and act on analytic information in their educational practices, prioritizing attention to needs, actionability, ethics and equity. Dr. Wise is a Co-Editor of the recently released 2nd edition of the Handbook of Learning Analytics, former Co-Editor-in-Chief of the Journal of Learning Analytics and was a Program Chair for the 2022 International Conference of Learning Analytics & Knowledge.
Abstract: To make a difference in the world, LA tools must not only be technically robust but also useful to real people and able to articulate with existing pedagogical practices. A move towards developing and using such tools can be seen as being part of a larger process of developing a culture of learning analytics. But exactly what kinds of learning analytics cultures do we want to develop and how can we best go about this? In our work at NYU-LEARN we approach this question through the lens of thinking about developing both “for humans” and “with humans,” trying on both sides of the equation to engage, rather than erase, the rich variation in ways that humans go about the business of teaching and learning. In this talk, I will share work from two LA research projects, one reporting an on-the-ground investigation into the different ways that instructors come to use analytics to inform their teaching, and the other describing a participatory design process for actionable student-facing analytics involving multiple stakeholders.
After the keynote is was time for a panel discussion about the topic ‘Data for learning’. Annoesjka Cabo, Jos van der Woude, Willem van Valkenburg and Marcus Specht discussed the topic based on what they learned from this day from different kinds of perspectives.
We ended the day with drinks and networking. We had a wonderful and inspiring day and hope you feel the same. See you next year!
