Rapsai: Rapid Application Prototyping System for AI

The project Rapsai, a.k.a. Visual Blocks for ML, aims to make the prototyping of machine learning (ML) based multimedia applications more efficient and accessible. In recent years, there has been a proliferation of multimedia applications that leverage machine learning (ML) for interactive experiences. Prototyping ML-based applications is, however, still challenging, given complex workflows that are not ideal for design and experimentation. To better understand these challenges, we conducted a formative study with seven ML practitioners to gather insights about common ML evaluation workflows. \n\nThe study helped us derive six design goals, which informed Rapsai, a visual programming platform for rapid and iterative development of end-to-end ML-based multimedia applications. Rapsai features a node-graph editor to facilitate interactive characterization and visualization of ML model performance. Rapsai streamlines end-to-end prototyping with interactive data augmentation and model comparison capabilities in its no-coding environment. Our evaluation of Rapsai in four real-world case studies (N=15) suggests that practitioners can accelerate their workflow, make more informed decisions, analyze strengths and weaknesses, and holistically evaluate model behavior with real-world input. Try our live demo at Visual Blocks for ML and let us know if you find it useful in your classes or project!

Publications

teaser image of Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications Through Visual Programming

Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications Through Visual ProgrammingHonorable Mentions Award, 170K+ views

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.
Keywords: visual programming, node-graph editor, deep neural networks, data augmentation, deep learning, model comparison, visual analytics, interactive perception




teaser image of Experiencing InstructPipe: Building Multi-modal AI Pipelines Via Prompting LLMs and Visual Programming

Experiencing InstructPipe: Building Multi-modal AI Pipelines Via Prompting LLMs and Visual Programming

Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.
Keywords: Visual Programming; Large Language Models; Visual Prototyping;Node-graph Editor; Graph Compiler; Low-code Development; DeepNeural Networks; Deep Learning; Visual Analytics




teaser image of InstructPipe: Building Visual Programming Pipelines With Human Instructions

InstructPipe: Building Visual Programming Pipelines With Human Instructions

https://arxiv.org/abs/2312.09672, 2023.
Keywords: Visual Programming; Large Language Models; Visual Prototyping; Nodegraph Editor; Graph Compiler; Low-code Development; Deep Neural Networks; Deep Learning; Visual Analytics; Interactive Perception



teaser image of Experiencing Visual Blocks for ML: Visual Prototyping of AI Pipelines

Experiencing Visual Blocks for ML: Visual Prototyping of AI Pipelines

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.
Keywords: visual programming, large language models, visual prototyping, multi-modal models, node-graph editor, deep neural networks, data augmentation, deep learning, visual analytics


teaser image of Experiencing Rapid Prototyping of Machine Learning Based Multimedia Applications in Rapsai

Experiencing Rapid Prototyping of Machine Learning Based Multimedia Applications in Rapsai

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA), 2023.
Keywords: visual programming, node-graph editor, deep neural networks, data augmentation, deep learning, model comparison, visual analytics, interactive perception


Videos

Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications Through Visual Programming


Visual Blocks: Ridiculously rapid ML/AI prototyping and deployment to production


How to create effects with models and shaders using visualblocks


How to compare models from web using visualblocks


How to use models and build pipelines in Colab with visualblock


Talks

Interactive Perception & Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Perception & Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at UCLA by Prof. Yang Zhang , Remote Talk.


Interactive Perception & Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Perception & Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at UCLA by Prof. Yang Zhang , Remote Talk.


Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at ECL Seminar Series by Dr. Alaeddin Nassani , Remote Talk.


Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at Empathic Computing Lab , Remote Talk.


Cited By

  • Creating Design Resources to Scaffold the Ideation of AI Concepts. Proceedings of the 2023 ACM Designing Interactive Systems Conference.Nur Yildirim, Changhoon Oh, Deniz Sayar, Kayla Br, Supritha Challa, Violet Turri, Nina Crosby Walton, Anna Elise Wong, Jodi Forlizzi, James McCann, and John Zimmerman. source | cite | search
  • IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency. arXiv.2308.12871.Saeid Ghafouri, Kamran Razavi, Mehran Salmani, Alireza Sanaee, Tania Lorido-Botran, Lin Wang, Joseph Doyle, and Pooyan Jamshidi. source | cite | search
  • Branching Preferences: Visualizing Non-linear Topic Progression in Conversational Recommender Systems. Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization.Lovis Bero Suchmann, Nicole Krämer, and Jürgen Ziegler. source | cite | search
  • Jigsaw: Supporting Designers in Prototyping Multimodal Applications by Assembling AI Foundation Models. arXiv.2310.08574.David Lin and Nikolas Martelaro. source | cite | search
  • Adaptation of Enterprise Modeling Methods for Large Language Models. The Practice of Enterprise Modeling.Balbir S. Barn, Souvik Barat, and Kurt Sandkuhl. source | cite | search
  • Canvil: Designerly Adaptation for LLM-Powered User Experiences. arXiv.2401.09051.K. J. Kevin Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy Zhang, and David W. McDonald. source | cite | search
  • Stay In Touch