The DeepClaw Project was recently selected for the UNESCO-ICHEI Higher Education Digitalisation Pioneer Case Award.
The UNESCO-ICHEI Higher Education Digitalisation Pioneer Case Award application highlights the transformative DeepClaw project led by Assistant Professor Chaoyang Song at the Southern University of Science and Technology. This innovative initiative addresses the critical need for accessible robotics education in developing countries like China. It underlines the crucial role of digital learning in bridging the gap by providing inclusive, cost-effective, and globally collaborative education opportunities. The project’s objectives include the development of a low-cost, portable system that combines tactile sensing and machine learning with soft robotics, aiming to democratize robotics education and foster innovation.
The teaching design in the ME336 Collaborative Robot Learning course incorporates the DeepClaw system, offering clear, measurable learning objectives and a multifaceted learning journey. Blended teaching methods enrich the student experience by providing open-sourced resources, online access to the DeepClaw Toolkit, state-of-the-art simulation environments, and various digital technologies. The digital transformation has yielded remarkable learning outcomes, with students excelling in project presentations, publications, and winning awards, while contributing to the evolution of the DeepClaw Toolkit. Theme-based integrated projects have further enriched the learning experience, promoting research and innovation.
The DeepClaw project stands out for its innovative design, bridging the gap between simulation and real-world robotics applications. Its simplicity, affordability, and emphasis on tactile sensing and portability make it a unique and valuable teaching tool. The project’s impact extends beyond national boundaries, as it offers digital resources accessible to students and researchers worldwide. Professor Chaoyang Song’s leadership and commitment to digital education are evident in the project’s ongoing development and its potential to revolutionize how students engage with robotics, fostering inclusivity, innovation, and research in the field.
Reason for Application
Background Introduction
Robotics is a multidisciplinary field of science and engineering focusing on robot design, construction, operation, and application. Robots are autonomous or semi-autonomous machines or systems that can perform tasks in the physical world, often replacing or assisting human labor in a wide range of industries and applications. Learning robotics is paramount for developing countries like China for various compelling reasons, and digital learning plays a crucial role in making this education accessible.
Robotics drives efficiency and productivity in industries like manufacturing and logistics, aligning with China’s goal of industrial leadership. Moreover, it is a gateway to STEM education, fostering critical thinking and practical engineering skills vital for the workforce and innovation. In the era of Industry 4.0, robotics education is essential for technological advancement, especially for a nation seeking modernization.
Digital learning is pivotal in this process, granting access to educational resources, online courses, and virtual labs. It offers flexibility, enabling students to learn at their own pace, making it inclusive for those with commitments. Online robotics courses cover diverse topics, allowing students to select their areas of interest. Virtual labs facilitate hands-on experience, which is particularly valuable for students without specialized hardware. Furthermore, digital learning promotes global collaboration, connecting students and educators worldwide. It offers content in multiple languages, reducing language barriers and ensuring inclusivity for non-English speakers.
Learning robotics is essential for developing countries like China due to its potential for economic growth, technological advancement, and addressing societal challenges. Digital learning is a key enabler, providing access to resources, online courses, virtual labs, and global collaboration. It bridges gaps in access to quality robotics education, empowering students to acquire the knowledge and skills needed to contribute to the country’s technological development and competitiveness on a global scale.
Teaching Design
ME336 Collaborative Robot Learning is an innovative blended teaching course that merges the worlds of collaborative robots and robotic learning, two cutting-edge domains in robotics research and industry applications. This course revolves around the DeepClaw system, an educational tool that empowers students to explore how robots learn to perceive, plan, and interact with the physical world, emphasizing the convergence of robotics and machine learning. Notably, this course is at the vanguard of robotics education, with only a few institutions, such as Stanford, UC Berkeley, and the Max Planck Institute, offering similar programs. The course strategically aligns with the pressing demand for graduates equipped with robust robot learning skills. Over the years, enrollment in this course has surged, attesting to its significance and popularity. The learning journey comprises lectures delving into the theoretical foundations, project-based hands-on exercises, interactive presentations, illuminating talks by industrial innovators, and practical lab sessions utilizing the DeepClaw system.
The core learning objectives encompass:
- Gaining proficiency in constructing real-world robotic systems employing vision-based machine learning and AI.
- Developing insights into the technical hurdles when crafting learning-based robotic manipulation systems.
- Acquiring familiarity with a spectrum of modal-driven and data-driven principles and algorithms integral to robot learning.
- Cultivating the ability to assess, communicate, and apply AI-based techniques to problem-solving within robotics.
To realize these objectives, students will embark on a multifaceted journey: 1) They will partake in the DeepClaw workshop, where theoretical concepts gleaned from lectures will be fortified through hands-on experimentation with the specially designed soft robotic toolkit; 2) Students will engage in collaborative paper reviews within teams, elucidating their comprehension of the latest advances in robot learning, and will be challenged by in-class Q&A sessions conducted by the instructor and their peers; 3) They will be privy to guest lectures delivered by industry luminaries in the field of robotics, enabling them to grasp the emerging trends driven by real-world imperatives; and 4) In the culmination of their academic voyage, students will conceptualize and execute a robot learning project aimed at addressing challenges either inspired by the industry or rooted in academia. They will leverage the DeepClaw Toolkit and the skills cultivated throughout the course through live demonstrations and presentations in the final class.
Digital technology will seamlessly integrate into the students’ learning experience throughout the semester. They will engage with the DeepClaw toolkit in in-class exercises, utilize online learning resources for practical assignments, partake in team-based presentations subject to peer reviews, and benefit from insights from industrial leaders from diverse backgrounds and sectors through guest lectures. This holistic approach ensures that students are well-versed in theoretical concepts and equipped with practical, real-world skills and insights, positioning them as valuable assets in robotics and robot learning.
Teaching Process
Teaching progress is propelled through implementing blended teaching and harnessing various digital media and technologies in the classroom.
- Open-Sourced Learning Resources: All course materials are accessible online via a dedicated website, available to the general public without restriction to enrolled students. This open access allows students to preview course content before the semester, aiding in their decision-making process to align their learning interests with the course. During the semester, students can delve into past years’ projects undertaken by previous cohorts. This enables a better grasp of project and assignment expectations and formats, offering diverse inspiration sources to mold their ideas into a final project. After completing the course, student projects are showcased on the same website, allowing them to review their learning accomplishments while gaining insights from other student teams’ work, enriching their learning journey.
- DeepClaw Toolkit Online: All workshop content, learning materials, and practice programs are available online, replete with executable codes and demonstrative implementations closely linked to the DeepClaw toolkit. This digital repository empowers students to practice these materials during in-class sessions or at their pace post-class. Students who contributed to toolkit development in prior semesters participate in the classroom, sharing their latest innovations through video demonstrations, code walkthroughs, and individual consultations. Their involvement elucidates the workings of the DeepClaw Toolkit, bridging it with lecture content and inspiring ideas for design improvements in a crowd-sourced educational milieu where students take center stage in their self-motivated learning.
- State-of-the-Art Simulation Environment: Simulation is pivotal in acquiring the knowledge and skills for robot learning. This virtual realm undergoes annual upgrades from the course’s inaugural offering in 2019. It began with introducing the Robot Operating System (ROS), an elemental simulation environment for students entering the realm of robotics. The subsequent year, we introduced an upgrade to VRep with PyRep, a well-acknowledged platform supporting robot learning with commercial and educational backing. In 2021, the DeepClaw toolkit hardware was introduced, supported by a dedicated data collection website that enhanced students’ learning experiences, ensuring compatibility across various hardware configurations. In response to the COVID-19 situation in 2022, a wide array of online resources from MIT and other institutions were integrated, leveraging the Feishu App to enrich students’ online learning. In 2023, the simulation environment was elevated to robosuite, offering students a taste of the latest developments in open-sourced robot learning software, harmonizing seamlessly with the custom-designed learning toolkit.
- Various Digital Technologies: Additional digital tools include PowerPoint presentations, video demonstrations, a variety of robotic hardware, online document-sharing platforms, collaborative editing software, and interactive peer-rating systems, all contributing to the digital learning experience at ME336. These tools synergize to facilitate effective learning, engagement, and interaction, enriching the educational journey for students.
Learning Outcomes
Digital education has significantly impacted student learning outcomes. Commencing with just nine students in the Spring of 2019, the course has flourished, boasting five-fold to 44 students in the latest semester of Spring 2023. Throughout this journey, students have excelled in numerous ways:
- Project Presentations and Publications: Students have presented over 30 projects, showcasing their creativity, innovation, and practical skills. Their efforts have culminated in publishing more than ten papers in prestigious journals and conferences dedicated to robotics and artificial intelligence. This robust academic output underscores the students’ commitment to research and ability to contribute to their respective fields.
- Award-Winning Endeavors: The students’ remarkable achievements extend to various competitions and accolades, with several notable accomplishments. They have clinched multiple awards at university and national levels, distinguishing themselves through their problem-solving capabilities and innovative projects.
- DeepClaw Toolkit Evolution: A hallmark of the course’s success is the collaborative development of the DeepClaw Toolkit. Initially created by the course instructors before the Spring of 2019, this toolkit has undergone three significant upgrades: Hardware Design (DeepClaw 1.0), User Interface (DeepClaw 2.0), and Portable Integration (DeepClaw Toolkit). These enhancements align with the course’s mission of promoting digital teaching through an open-sourced approach. At the same time, the first version required dedicated and complex robot hardware. The latest iteration adopts a shareable, lightweight, and low-cost design. Priced at approximately 1,000 RMB or under 200 USD, the DeepClaw Toolkit packs all necessary tools into a foldable fanny bag, promoting accessibility and affordability. The course has produced about 40 DeepClaw Toolkit sets, including a browser-based user interface. This user-friendly interface enables students to engage in hands-on learning, covering various topics from machine vision and data science to advanced subjects such as soft robotics, tactile sensing, neural networks, and reinforcement learning.
- Theme-Based Integrated Projects: In 2020, the course piloted a pioneering international collaboration with MIT to introduce theme-based integrated projects centered on the concept of “wasteless.” This initiative aimed to educate students in designing and constructing machines for autonomous waste sorting. This project unfolded across multiple courses and semesters. Students commenced their journey in the Autumn Semester with ME303 Mechanical Design, continued their exploration of learning algorithms in the Spring Semester through ME336 Collaborative Robot Learning, and finally integrated the system into a functional prototype during ME491 Engineering Practice. Students participated in cross-semester and cross-course projects, gaining in-depth knowledge while developing the system. By the program’s conclusion, students had established four waste-sorting production lines, each comprising three dedicated machines for distinct functions. Furthermore, they implemented four diverse robot learning algorithms for various aspects of human-robot interactions, culminating in developing an integrated waste-sorting robot that entered an entrepreneurial competition and secured several awards.
In essence, the digital transformation of this course has empowered students to achieve remarkable success, from academic publications to competition victories and the co-development of innovative toolkits, thus fostering a rich learning environment.
Innovative Design
The DeepClaw system represents a pioneering digital technology that has revolutionized teaching and learning in robotics, successfully navigating through three development iterations. It addresses a key challenge in robotics education by providing affordable, shareable, and accessible hardware for hands-on practice, bridging the gap between simulation and real-world applications in teaching and learning robotics.
One of the primary challenges in teaching robotics lies in the limited access to hardware for hands-on learning, which can be costly, resource-intensive, and technically demanding. DeepClaw, developed by the SUSTechDL group, was created to enhance robot learning, focusing on integrating robotic hardware as a central component in the educational process.
Initially designed as a reconfigurable workstation for vision-based robotic picking at Monash University, DeepClaw underwent significant evolution upon transitioning to SUSTech. The system was refined to include APIs for seamless communication with various robotic hardware and streamlined mechanical design using standard aluminum extrusion systems. The development progressed further with the soft robotic tongs, enhancing data collection efficiency in grasping tasks. The latest iteration, the DeepClaw Toolkit, incorporates tactile learning and emphasizes portability and affordability. Notably, students actively participated in the iterative development of DeepClaw, cementing their role at the forefront of the system’s evolution.
The DeepClaw Toolkit effectively overcomes a critical challenge in teaching robotics—integrating tactile sensation. While existing simulation software provides valuable insights, the toolkit offers students hands-on experience translating unstructured environmental data into meaningful information for robotic manipulation and machine learning. This learning tool is highly cost-effective, packing the necessary components within a convenient fanny bag.
The simplicity and affordability of modified kitchen tongs make them accessible and easy to comprehend for students. Equipped with soft robotic fingers and vision-based sensing, these tongs become potent tools for data-driven research. The user-friendly browser interface enables plug-and-play functionality, significantly reducing entry barriers and enhancing the learning experience. Furthermore, the toolkit offers extensive learning materials and online tutorials, complete with step-by-step codes and practical exercises, allowing students to progress at their own pace. This flexibility provides ample room for experimentation in various project-based learning activities.
In addition, the DeepClaw system supports remote learning, offering a consistent educational experience even in challenging circumstances such as the COVID-19 pandemic. Students can engage in robotics interactions from the comfort of their homes, ensuring that learning continues seamlessly regardless of location.
The DeepClaw system, through its innovative design and commitment to student involvement, has revolutionized the teaching and learning of robotics. It bridges the gap between simulation and reality, making robotic hardware accessible and affordable for students and promoting an interactive and engaging learning environment.
Project Progress
The DeepClaw project has been a remarkable initiative at SUSTech, supported by three teaching grants since its inception in 2019. Its mission is to create a globally accessible, open-source, and cost-effective toolkit for teaching robot learning, featuring a user-friendly browser-based interface and comprehensive learning materials. Over the years, DeepClaw has undergone three significant updates, aligned with the goals of the ME336 class, generated valuable research output, and benefited from internal SUSTech teaching grants.
In 2019, the project received a grant titled “Using Collaborative Robots for Interactive Teaching in Emerging Engineering Subjects.” The objective was to provide essential equipment and course development support, leading to the development of DeepClaw 1.0. This version was instrumental in supporting the ME336 class in the Spring Semester of 2019 and resulted in a conference publication at AIM2020. DeepClaw 1.0 marked the beginning of an exciting journey in robotics education.
In 2021, the project expanded its horizons with a grant titled “Laboratory Teaching Reform using Cross-Class, Continuous-Project, Big-Team Projects.” This grant fostered international collaboration with MIT, mainly focusing on the thematic project “Wasteless.” It led to the development of DeepClaw 2.0, featuring a dedicated user interface for efficient data collection using soft robotic tongs. The project also contributed to a journal publication in the prestigious Frontiers in Robotics and AI. This step showcased the versatility and adaptability of DeepClaw, extending its reach across multiple classes.
In 2023, the project’s vision continued to evolve with a grant titled “Machine Intelligence Design and Learning Virtual Teaching Lab.” This grant aimed to consolidate the DeepClaw project into a comprehensive blended teaching resource, offering full online access to teaching materials and toolkit resources. The DeepClaw Toolkit underwent further development as part of this endeavor, and a conference paper was submitted to ICRA
2024. This project is ongoing, with expected completion by 2024, reflecting the commitment to continuous improvement and innovation.
Presently, the DeepClaw team is actively upgrading the online platform, enhancing the user interface of the DeepClaw Toolkit, and improving hardware functionality. The goal is to prepare for the following teaching semester of ME336 and explore the potential for broader adoption of the toolkit by other universities and interested institutions. The team envisions achieving this through an open-source approach and potential crowdfunding initiatives, emphasizing accessibility and collaboration in robotics education.
The DeepClaw project is a testament to the dedication, innovation, and adaptability of SUSTech’s teaching initiatives. It not only enriches students’ educational experience but also contributes to the global dissemination of knowledge in robotics and machine learning.
Project Impact and Sustainability
The DeepClaw project is a testament to the unique and innovative work conducted by the applicant’s research team at the SUSTech Design and Learning Lab. It has roots in fundamental and applied research, resulting in a remarkable body of knowledge. This endeavor has yielded high-impact publications in esteemed journals and conferences, underscoring the research excellence in vision-based soft robotic tactile sensing. Additionally, the project boasts a range of patents granted in China and the United States, acknowledging the pioneering core technology behind DeepClaw.
DeepClaw has evolved, with several development iterations culminating in a versatile and accessible system. Notably, the project provides online access, ensuring its reach extends far beyond the borders of China. The toolkit includes a rich repository of online teaching materials replete with videos and demonstrations in English, making it invaluable for not only Chinese students but also international learners worldwide interested in robot learning.
The project’s journey, development, and resources are well-documented and accessible through various digital platforms.
- An overview of the overall development of this teaching initiative can be found at the following link: https://bionicdl.ancorasir.com/?page_id=602.
- The official course website for ME336: https://me336.ancorasir.com/.
- A video summary of ME336 offered for the first time in 2019 is accessible through this link: https://bionicdl.ancorasir.com/wp-content/uploads/2023/10/Teaching-ME336-2019.mp4.
- A video demonstration of DeepClaw Toolkit based on a Conference Paper submitted to ICRA2024 regarding the latest implementation for ME336: https://bionicdl.ancorasir.com/wp-content/uploads/2023/09/BionicDL-Publications-2023-C-ICRA2024-SoftRoboticTong.mp4
Through these digital resources, the DeepClaw project is a remarkable example of digital technology and media design, offering valuable knowledge and insights to students and researchers, irrespective of their geographical location. It has become a global platform for the dissemination of knowledge and expertise in the realms of robotics and machine learning.
Applicant Involvement Level
The lead investigator for this project is Assistant Professor Chaoyang Song, with Assistant Professor Wan Fan contributing critically to the incubation of the idea and its continuous development since the beginning. This initiative has been a collaborative endeavor, with active contributions coming from a diverse group of participants. The team comprises students who are enrolled in the ME336 course, as well as members of the SUSTech Design and Learning Lab.
| Contributors | Original DeepClaw | DeepClaw SUSTech | DeepClaw 1.0 | DeepClaw 2.0 | DeepClaw Toolkit |
|---|---|---|---|---|---|
| Liu Xiaobo | X | X | X | ||
| Guo Ning | X | X | |||
| Ge Sheng | X | X | |||
| Sun Haoran | X | ||||
| He Haibin | X | ||||
| Zheng Shuxin | X | ||||
| Luo Qichen | X | ||||
| Chen Mingdong | X | ||||
| Shi Jianping | X | ||||
| Wang Zhiwei | X | ||||
| Chai Tianhao | X | ||||
| Xu Tianyuan | X | ||||
| Yang Linhan | X | ||||
| Wang Haokun | X | X | |||
| Qiu Nuofan | X | X | |||
| Wang Teng | X | X | |||
| Yang Yuxuan | X | ||||
| Yu Zhiyang | X | ||||
| Dong Yujian | X | ||||
| Xiao Yang | X | ||||
| Wei Jinqi | X | ||||
| Wu Tianyu | X | ||||
| Han Xudong | X | ||||
| Wan Fang | X | X | X | X | X |
| Song Chaoyang | X | X | X | X | X |
| Outcomes | [FROBT, 2018] | [3d Prize for Capstone Project] [ICARM, 2023] [ICARM2023 Best Paper Award Finalist] | [AIM, 2020] | [FROBT, 2022] | [1st & 2nd Prizes for Capstone Project] [UNESCO-ICHEI Higher Education Digitalisation Pioneer Case Award] [MatDes (UR)] [ICRA1 (UR)]、[ICRA2 (UR)] |
Related Publications and Awards:
- Fang Wan and Chaoyang Song* (2018). “A Neural Network with Logical Reasoning based on Auxiliary Inputs.” Frontiers in Robotics and AI, 5(July):86.
doi: https://doi.org/10.3389/frobt.2018.00086 - Fang Wan, Haokun Wang, Xiaobo Liu, Linhan Yang, and Chaoyang Song* (2020). “DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation.” IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Boston, MA, USA, 6-10 July 2020, pp. 2011-2018.
doi: https://doi.org/10.1109/aim43001.2020.9159011 - Haokun Wang#, Xiaobo Liu#, Nuofan Qiu, Ning Guo, Fang Wan*, and Chaoyang Song* (2022). “DeepClaw 2.0: A Data Collection Platform for Learning Human Manipulation.” Frontiers in Robotics and AI, 9(March):787291.
doi: https://doi.org/10.3389/frobt.2022.787291 - Xiaobo Liu#, Fang Wan#*, Sheng Ge, Haokun Wang, Haoran Sun, and Chaoyang Song (2023). “Jigsaw-based Benchmarking for Learning Robotic Manipulation.” IEEE International Conference on Advanced Robotics and Mechatronics (ICARM). Sanya, China, 8-10 July 2023, pp. 124-130.
doi: https://doi.org/10.1109/ICARM58088.2023.10218784
Awards: Best Conference Paper Finalist | About the Award - 3rd Prize of SUSTech Faculty of Engineering’s Capstone Project in 2020: Sun Haoran, Ge Sheng, He Haibin, Zheng Shuxin, and Luo Qichen.
- 1st Prize of SUSTech Faculty of Engineering’s Capstone Project in 2022: Dong Yujian, Xiaoyang, and Wei Jinqi.
- 2nd Prize of SUSTech Faculty of Engineering’s Capstone Project in 2022: Yang Yuxuan and Yu Zhiyang.
- Tianyu Wu#, Yujian Dong#, Xiaobo Liu#, Xudong Han, Yang Xiao, Jinqi Wei, Fang Wan*, and Chaoyang Song* (2023). “Vision-based Tactile Intelligence with Soft Robotic Metamaterial.” Materials & Design.
(Under Review) - Nuofan Qiu, Fang Wan*, and Chaoyang Song* (2023). “Describing Robots from Design to Learning: Towards an Interactive Lifecycle Representation of Robots.” IEEE International Conference on Robotics and Automation (ICRA).
(Under Review) - Yujian Dong#, Tianyu Wu#, Yang Xiao, Jinqi Wei, Fang Wan*, and Chaoyang Song* (2023). “Vision-based, Low-cost, Soft Robotic Tongs for Shareable and Reproducible Tactile Learning.” IEEE International Conference on Robotics and Automation (ICRA). (Under Review)
Notably, the project took a significant step forward in 2023 by establishing the Machine Intelligence Design and Learning Virtual Teaching Lab with Prof. Song Chaoyang as the Principal Investigator. This expansion enabled the inclusion of esteemed faculty members, enriching the collaborative landscape. The team now includes Professor Jiansheng Dai, a distinguished researcher recognized as a Fellow of the Royal Academy of Engineering (FREng) and an Academia Europaea (MAE) member. Additionally, the project benefits from the expertise and involvement of other accomplished faculty members, namely Professor Chenglong Fu, Department Head of Mechanical and Energy Engineering, Assistant Professor Yang Pan, Assistant Professor Zhenzhong Jia, and Research Associate Professor Sicong Liu. This collaborative effort combines knowledge, experience, and innovative thinking to advance the project’s goals and impact.