InstructPipe: Building Visual Programming Pipelines With Human Instructions

The project Rapsai, a.k.a. Visual Blocks for ML, aims to make the prototyping of machine learning (ML) based multimedia applications more efficient and accessible. In recent years, there has been a proliferation of multimedia applications that leverage machine learning (ML) for interactive experiences. Prototyping ML-based applications is, however, still challenging, given complex workflows that are not ideal for design and experimentation. To better understand these challenges, we conducted a formative study with seven ML practitioners to gather insights about common ML evaluation workflows. \n\nThe study helped us derive six design goals, which informed Rapsai, a visual programming platform for rapid and iterative development of end-to-end ML-based multimedia applications. Rapsai features a node-graph editor to facilitate interactive characterization and visualization of ML model performance. Rapsai streamlines end-to-end prototyping with interactive data augmentation and model comparison capabilities in its no-coding environment. Our evaluation of Rapsai in four real-world case studies (N=15) suggests that practitioners can accelerate their workflow, make more informed decisions, analyze strengths and weaknesses, and holistically evaluate model behavior with real-world input. Try our live demo at Visual Blocks for ML and let us know if you find it useful in your classes or project!

Publications

teaser image of Experiencing InstructPipe: Building Multi-modal AI Pipelines Via Prompting LLMs and Visual Programming

Experiencing InstructPipe: Building Multi-modal AI Pipelines Via Prompting LLMs and Visual Programming

Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.
Keywords: Visual Programming; Large Language Models; Visual Prototyping;Node-graph Editor; Graph Compiler; Low-code Development; DeepNeural Networks; Deep Learning; Visual Analytics




teaser image of InstructPipe: Building Visual Programming Pipelines With Human Instructions

InstructPipe: Building Visual Programming Pipelines With Human Instructions

https://arxiv.org/abs/2312.09672, 2023.
Keywords: Visual Programming; Large Language Models; Visual Prototyping; Nodegraph Editor; Graph Compiler; Low-code Development; Deep Neural Networks; Deep Learning; Visual Analytics; Interactive Perception



Videos

Talks

Computational Interaction for a Universally Accessible Metaverse Teaser Image.

Computational Interaction for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at University of Minnesota by Prof. Zhu-Tian Chen , Minneapolis, MN, USA.


Experiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual Programming Teaser Image.

Experiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual Programming

Zhongyi Zhou

, .


Visual Blocks for ML: Visual Prototyping of AI Pipelines Teaser Image.

Visual Blocks for ML: Visual Prototyping of AI Pipelines

Ruofei Du

CS139: Human-Centered AI @ Stanford , Stanford, Palo Alto.


Cited By

Stay In Touch