Data (In)sight for All: Multimodal Chart Exploration with MAIDR

JooYoung Seo
School of Information Sciences, University of Illinois Urbana-Champaign

AccessComputing Partner Meeting, October 23, 2026

Thank you, Brianna, and thanks to the AccessComputing partners for having me. Today I want to talk about a barrier many of us encounter in computing education and research: charts that assume everyone can see.

In about fifteen minutes, I will introduce a shared multimodal approach and show how to use it through Python, R, and an AI assistant. The examples are meant for the same classrooms, research groups, and communities that AccessComputing and AccessSTEM support. Then I look forward to your questions.

Let’s Think…

What good is a chart you can’t see?

How do you read a chart you can’t see?

All right, let’s jump right in. We’re going to talk about data. You know, it’s pretty much the engine behind everything these days. Business, science, you name it. And we usually understand data through things like charts and graphs. They’re like a universal language. But what happens when you can’t see that language? No, really, think about that. We rely so much on just glancing at a chart and seeing the pattern, right? A spike here, a dip there. So how on earth are you supposed to get any meaning out of a bar chart or a scatter plot if you can’t actually see it?

Data Viz Divide

A Picture’s Worth…?

For Sighted Users

• Instant insight
• Visual patterns at a glance
• Effortless data comprehension

For Users with Visual Impairments

• Wall of inaccessible data
• Locked door, with valuable information trapped inside
• Barriers to understanding

Well, this is the divide. For a sighted person, a chart gives you that instant insight, right? The trends, the weird outliers, they just pop. But for blind and low vision users, that exact same chart can be a total dead end. It’s a wall. It completely locks them out of the entire data conversation.

“Design for Them” Approach

• Sighted person to create a chart and then add a separate description as an afterthought

• One group produces boundaries while the other group just consumes

• One-way street: power imbalance

for those who cannot see it. I am sorry, but that’s their issue.

This quote just nails the real problem, doesn’t it? See, for the longest time, the approach to accessibility has been what you might call design for them. A sighted person makes a chart and then, almost as an afterthought, a separate, often clunky text description gets added for blind and low vision or BLV users. It creates this one-way street, this power imbalance where one group is basically dictating the entire experience for another.

Shifting to “Design for Us”

“For Them”

• Unidirectional, separate tools, one group dictates
• One-way street
• Exclusionary design approach

“For Us”

• Bidirectional, unified framework, all users collaborate
• Two-way conversation
• Inclusive design approach

And this slide just lays it all out, side by side. The old way, the ‘for them’ way, that’s the one-way street we talked about. Separate tools, separate experiences, with the sighted creator basically calling all the shots. But this new ‘for us’ way, well, that’s a two-way conversation. It’s about creating a single unified framework where everyone, sighted and BLV users, can work together on the exact same visualization, just using different senses to perceive it. It’s a huge difference.

Meet {maidr}!

Data Insight for All.

And that’s what brings us to the solution. It’s a bridge, and it’s called maidr. And what’s so cool about it is that it’s not just a tool, it’s a whole new way of thinking. It’s a shift from design for them to design for us.

What is {maidr}? 🔎

• Multimodal Access and Interactive Data Representation

• Decoupling vision-first approach from data interpretation

• Encode data representation in multiple modalities

• Not a separate, watered-down version for blind users

• one unified experience: the same information with preferred modalities

And this really gets to its whole reason for being, its motto: Data Insight for All. The goal here isn’t to make some separate watered-down version for blind users. Nope. It’s about creating one unified experience that lets everybody work with the exact same information together.

It is open source, and it grew out of my IMLS Early Career project. Today it is a family of packages for R, Python, and JavaScript, with more than 119,000 downloads.

Multimodal Data Representation

On top of visualization:

• Touchable (Braille) 🖐️

• Readable (Text) 📖

• Audible (Sonification) 🎶

• Conversable (AI) 💬

The visual stays exactly as you designed it. On top of it, the same plot becomes what we call multimodal. And that just means a user can interact with it in all these different ways. They can explore it on a refreshable braille display, they can read it as simple text, they can hear the data points as musical tones, that’s sonification, and they can even ask it questions with a built-in AI chat. Let me take them one at a time.

Touchable: Feel Data 🖐️

• Data shape and trend pattern in Unicode Braille symbols

• Users can run their fingers across the data to feel its peaks and valleys

• With a refreshable braille display, or as a tactile graph on a Dot Pad or a Monarch

First up, making data touchable. Maidr can represent a whole plot using braille symbols on what’s called a refreshable braille display. So a user can literally run their fingers across the data and feel its shape, its peaks, its valleys.

And on a multiline tactile display, such as the Dot Pad or the Monarch, the plot itself is raised in pins: a tactile graph.

It Feels Like This…

This is a box plot of life expectancy by continent. Above it is the same box plot as one line of braille, and in the photo a hand is reading that line on a refreshable braille display. Each kind of dot pattern stands for a part of the box: the whiskers, the box, the median, the outliers. Read it left to right and you feel where the data is spread out and where it is packed together.

Audible: Listen to Data 🔊

• Sonification turns data into audible tones

• x-axis value: stereo panning position

• y-axis value: pitch

• As a trend goes up, a musical pitch rises with it

But touch is just the beginning. It also uses something called sonification, which is just a fancy word for turning data into sound. Imagine you’re tracking a trend line. As the line goes up, a musical pitch goes up with it. You could literally hear the data.

It Sounds Like This…

Play the audio. It is five seconds.

That is a scatter plot with a strong positive relationship. You heard the pitch climb as the sound moved from your left ear to your right. Nobody had to tell you the trend goes up.

Readable: Understand Data 📖

• A straightforward text of the plot’s key features

• Get a clean, simple summary of what the plot is showing

And of course there is a simple text mode. Each point you land on is read out with its axis labels and its value, and one key gives you a description of the whole plot.

Conversable: Chat with Data 💬

• Powered by AI/LLMs (GPT, Gemini, Claude)

• Ask questions and get co-insight of the plot

• Natural language data analysis

• Example Query Can you summarize the main trend for me?

Now, this fourth mode is straight out of science fiction. Maidr actually plugs into conversational AIs: GPT, Gemini, Claude. This means you can just talk to the chart. You can literally ask it questions like, “Hey, what’s the highest value here?” or “Can you summarize the main trend for me?” It’s like having a data analyst built right into the visualization.

Beyond Theory: {maidr} in Action!

Beyond theory: maidr in action. Let’s cut to the chase.

One Engine, Three Ways In

• Python: import maidr with Matplotlib, Seaborn, Plotly, or Altair

• R: library(maidr) with ggplot2 or base R graphics

• AI assistants: the maidr skill for Claude and ChatGPT

• Each one hands your plot and its data to the same engine, maidr.js

Here is the map for the next fifteen minutes. Three ways in, depending on how you make your plots: Python, R, or an AI assistant that writes the code for you.

Whichever door you take, the same thing happens. Your plot goes out as you drew it, and the data behind it travels along. One engine, maidr.js, turns that into sound, braille, text, and conversation. So what you learn on the first chart works on all the others.

Keyboard-Toggle Modalities

Key	Function
Arrow Keys	Move around the plot
B	Toggle Braille Mode
T	Toggle Text Mode
S	Toggle Sonification Mode
D	Open Plot Description
Page Up / Page Down	Switch Layers
?	Open Chat AI Mode

And the best part, it’s all driven by the keyboard. You’re in complete control. Users can just use the arrow keys to move from data point to data point. And then with a single key press, they can switch modes. B for braille, T for text, S to hear the sonification. D gives a description of the whole plot, Page Up and Page Down move between the layers of a plot, and the question mark opens the AI chat. It puts the user right in the driver’s seat.

Python: Just Plug In

# Step1: Import
import maidr #<<

import matplotlib.pyplot as plt

# Step2: Create a plot
plt.bar(['A', 'B', 'C'], [1, 2, 3])

# Step3: Show with maidr
maidr.show() #<<

Python first. And when I say plug in, I am not kidding. The simplicity here is honestly what makes it so powerful. To make your plot multimodal, you write your normal plot code. Then you add one line, import maidr, and finally, instead of the old plt.show(), you just use maidr.show(). That’s it. Two tiny changes to your code.

Python: Multiline Plot

import maidr #<<

plt.plot(x, life_idx, "bo-", label="Life expectancy index")
plt.plot(x, gdp_idx, "rs--", label="GDP per capita index")
plt.plot(x, pop_idx, "g^:", label="Population index")
plt.xlabel("Year")
plt.ylabel("Index")

maidr.show() #<<

A real one. Three lines for the United States since 1950: life expectancy, GDP per capita, and population, each indexed to 100 in the first year. The two highlighted lines are all that maidr adds.

(Next slide, the plot itself. Tab moves into it. Right arrow walks the years, and the pitch follows the line. Up arrow changes lines: population, then GDP per capita. T turns text on, B braille, S sound. Shift and Tab return to the slides.)

Python: Multiline Plot

Python: Box Plot

import maidr #<<


sns.boxplot(
    x="gdp_k", y="continent", data=df2,
    order=["Africa", "Europe"],  showfliers=True
)

plt.xlabel("GDP per capita (thousands $)")
plt.ylabel("Continent")

maidr.show() #<<

Same two lines, now with Seaborn: GDP per capita in 2007, Africa above Europe, as a box plot.

(Next slide. Tab in. Right arrow moves through one box: from the lower outliers and the minimum, through 25%, 50%, and 75%, to the maximum and the upper outliers. Up arrow goes from Europe to Africa; Down arrow returns from Africa to Europe. B shows the braille, the kind of pattern on the “It Feels Like This” slide.)

Python: Box Plot

{Shiny} + {maidr} integration

# Minimal Shiny-for-Python app structure with MAIDR-ready plots
from shiny import App, ui

# UI with tabs mirroring slides
app_ui = ui.page_fluid( # implement ui.navset_tab with tabs mirroring slides
)

@output
@render_maidr #<<
def create_histogram_output():
    ax = ...  # build matplotlib/seaborn axes
    return ax

app = App(app_ui, server)

Source: xability/a11y_dashboard

Plots do not only live in notebooks and slides. In a Shiny dashboard, one decorator, render_maidr, turns each plot the dashboard draws into the same multimodal chart, and it stays that way as the reader changes the inputs.

maidr Shiny Dashboard

And this is a dashboard built that way, live on shinyapps.io: histogram, box, scatter, bar, line, heatmap, multiline, multilayer and multipanel plots, and one you design yourself. Every plot in it is a maidr chart, so the same keys work: pick a tab, Tab into the plot, and use the arrows.

(Live: the app can take a few seconds to wake, and a tab’s chart a few more to appear. Tab moves into the dashboard; Shift+Tab, a few times, comes back to the slides.)

The Magic behind the Scenes

So, you’re probably wondering, what’s the trick? How do two lines of Python pull this off? It’s a really clever three-step dance, and it happens in a split second. I love the analogy that it’s like a simultaneous translator for data. First, maidr intercepts the command to create a plot. Then it pulls out all the important bits: the data, the structure, what kind of chart it is. And finally, it injects those accessible layers right into the plot before it ever gets displayed. Totally seamless.

The R package takes the same three steps for ggplot2 and base R graphics, and R is where we go next.

R: One Line

library(maidr) #<<
library(ggplot2)

ggplot(mpg, aes(class)) +
  geom_bar(fill = "steelblue") +
  labs(title = "Cars by class", x = "Class", y = "Number of cars")

Now R. Here it is one line: library(maidr). After that, any ggplot you print opens as a multimodal plot. Nothing else in your code changes.

This is the mpg data that ships with ggplot2: how many cars there are in each class.

R: One Line

Same keys as in Python. Tab into the plot, then the arrow keys: seven bars, from two-seaters to SUVs. B for braille, T for text. Shift and Tab bring me back to the slides.

R: Layers

library(maidr) #<<
library(ggplot2)

ggplot(mpg, aes(hwy, after_stat(density))) +
  geom_histogram(bins = 15, fill = "steelblue") +
  geom_density(linewidth = 1) +
  labs(title = "Highway fuel economy",
       x = "Highway miles per gallon", y = "Density")

Plots are often more than one layer. This one puts a density curve over a histogram of highway fuel economy. Still one line of maidr.

R: Layers

Tab in. The arrow keys walk the bins of the histogram. Page Up moves to the next layer, the density curve, and now the arrows follow the curve: listen to it rise and fall. Page Down takes me back to the histogram.

AI: Ask Your Assistant

• AI assistants hand you plots as pictures: nothing to hear, touch, or explore

• The maidr skill teaches them to make every plot multimodal

• It picks Python, R, or JavaScript for you, and checks its own work

• For Claude and ChatGPT, and for Codex, Cursor, GitHub Copilot, and Gemini CLI

And the third way in. Many of you no longer write plotting code at all. You ask Claude or ChatGPT for a plot, and what comes back is a picture. To me, that is the same wall all over again.

So we wrote a skill. A skill is a set of instructions an AI assistant loads when the task calls for it. Ours teaches the assistant to make every plot it draws multimodal, whether or not you ask. It picks Python, R, or JavaScript to match your project, and it checks its own work before it hands the plot over.

AI: One-Time Setup

• Claude Code: /plugin marketplace add xability/maidr-skill then /plugin install maidr@maidr-skill

• ChatGPT.com: Download the ready-made ZIP, then Plugins → Skills → Add → Upload

• Claude.ai: upload the same ZIP in Customize → Skills → Add and turn it on

• Any coding agent: npx skills add xability/maidr-skill

You install it once. For the web apps, download our ready-made ZIP and upload it as a skill, not as a chat attachment. In ChatGPT, this requires a plan or workspace with Skills enabled; both the Illinois workspace and the personal Pro account used in this rehearsal have it. In Claude, enable code execution and turn the skill on. No one needs to clone the repository or build a ZIP. The public plugin directory submissions are not the same thing: Claude is still in review, and current OpenAI approval has not been confirmed. Claude Code can use the third-party marketplace commands on the slide. All of this is documented with official sources in the README of xability slash maidr-skill.

AI: Just Ask for a Plot

Chart the number of AccessComputing partners in each partner category
from https://accesscomputing.uw.edu/partners.

Here is an example from this community, rather than from our own university: the partner network that brings institutions, organizations, and industry together. This is the same original prompt used in Claude.ai and the personal ChatGPT demo. It asks for a chart and identifies the public source; it does not provide any numbers.

In this rehearsal, Claude.ai’s workspace had web search disabled, and its code sandbox could not fetch the source domain. I therefore supplied counts checked against the official roster: 77 Institutional Partners, 29 Organizational Partners, and 17 Industry Partners, captured October 20, 2026. These are partner-category memberships, not student counts or program impact measures.

The first exported chart used Chart.js without a MAIDR binding. A follow-up explicitly requested the installed maidr skill and genuine Python binding output. Keep both follow-ups in the conversation when presenting this example: do not claim that this run fetched the data or automatically activated the skill from the first prompt. The full prompt history and source notes are saved under demo/.

AI: What Came Back

Claude.ai · Open the original demo (presenter login)

This is the chart exported by Claude.ai, not a replacement drawn outside the conversation. The original conversation opens in a new tab and requires the presenter’s signed-in Claude.ai account.

The bars represent listed entries in the three official partner categories: Institutional 77, Organizational 29, Industry 17. They sum to 123 category memberships. Teach Access is listed in both Organizational and Industry Partners, so that is 122 unique partners, not 123. The chart preserves that overlap note and credits the official roster. This is a snapshot, not a historical trend or a claim about student outcomes.

Tab into the chart and use the arrow keys to explore. T toggles text, B braille, and S sound. Physical braille hardware and screen-reader audio were not tested in this browser rehearsal. Use the exported HTML in this slide if the live conversation is unavailable. The MAIDR runtime still loads from a CDN, so an internet connection is required.

AI: ChatGPT Public Demo

ChatGPT · Shared chat history · Open chart (no login)

This is a second run of the same one-line prompt, in a personal ChatGPT Pro account with the MAIDR skill installed. ChatGPT fetched the official partner roster itself, counted partner entries rather than named contacts, identified the Teach Access overlap, and generated a real py-maidr chart. It did not need the counts supplied in a follow-up as the Claude.ai rehearsal did. The counts are Institutional 77, Organizational 29, and Industry 17.

This slide preserves ChatGPT’s generated SVG, MAIDR data binding, and bundled runtime, with only an outer HTML wrapper and source/overlap notes added for presentation. The original file preview passed keyboard and braille checks. After a delivery-only follow-up, the same chart also rendered inside the conversation: Tab entered the MAIDR application, arrows announced 77, 29 and 17, and every highlight matched its original bar. All three values and highlights were checked at actual inline frame widths of 360, 390 and 778 CSS pixels, without horizontal overflow; the full ChatGPT mobile layout was not emulated. B displayed the braille shape. Physical braille hardware, screen-reader audio, and data-dependent sonification in the actual ChatGPT frame remain untested.

The original response printed a bare visualize call instead of rendering the chart. A follow-up corrected the native publication syntax without redrawing it. The new public link includes that complete history and the corrected inline chart; its read-only shared view passed Tab and all three arrow-key/value/highlight checks while signed in. Do not claim that the initial one-line response already rendered inline. A fresh anonymous browser opened the shared conversation without login, but the inline result instead said “Code execution is disabled for this workspace.” The signed-in check is not proof of anonymous chart execution. The converter, guidance and regression tests were corrected in upstream maidr-skill PR #41, which is submitted but not merged. Use the direct interactive chart link or this embedded copy for reliable no-login public interaction. The exported bundle works without downloading maidr.js; its UI begins in English, settings are session-local, and its AI chat is unavailable.

But Does It Work With…?

119,000+ downloads across CRAN, PyPI, and npm

• Python: Matplotlib, Seaborn, Plotly, Altair

• R: ggplot2, base R graphics

• JavaScript: D3, Plotly.js, Vega-Lite, Observable Plot, Chart.js, Highcharts, ECharts, Google Charts, amCharts, AnyChart, ApexCharts, Frappe Charts, uPlot, TradingView Lightweight Charts, React, Recharts, Victory, Nivo, MUI X Charts

• BI and Office: Tableau, Power BI, Excel, PowerPoint, Word

• Where you work: terminal, Jupyter Notebook, Google Colab, Quarto, Shiny, Streamlit, web pages

• AI assistants: agent skill and MCP server, for Claude and ChatGPT

So, does it work with the tool you use? Probably. Python and R you have seen. On the web there are adapters for nineteen JavaScript libraries, from D3 and Plotly to Google Charts. It reads embedded Tableau dashboards, Power BI visuals, and the charts in Excel, PowerPoint, and Word. It runs wherever you work: the terminal, Jupyter, Colab, Quarto, Shiny, Streamlit, or a plain web page. And for AI assistants there is the skill you just saw, plus an MCP server.

Together, the packages have been downloaded more than 119,000 times.

A Future We Build Together

But look, this is about more than just a single tool. This is about a bigger vision. It’s about a fundamental shift in how we think about our community and our work. It’s about building a future that we can all be a part of.

Data Insight for All

Pick your way in:

# Python
pip install -U maidr

# R
install.packages("maidr")

# AI assistants
npx skills add xability/maidr-skill

Three ways in, three lines. Pick the one that matches how you work, and try it on a plot from your own research this week.

Learn More & Contribute

• Start here: maidr.ai

• Python: py.maidr.ai

• R: r.maidr.ai

• AI skill: github.com/xability/maidr-skill

• Source Code: github.com/xability

If you want to play with live demos, read the documentation, or see more examples, start at maidr.ai. The entire project is open source on GitHub, of course, and we would absolutely love your feedback and contributions.

Acknowledgments

Sponsors

• The National Science Foundation #2348166
• The Institute of Museum and Library Services #RE-254891-OLS-23
• Teach Access Faculty Grant
• The Wallace Foundation Grant and the International Society of the Learning Sciences
• The PI’s faculty startup grant

Contributors

• Saairam Venkatesh
• Sean McCurry
• Yu Jun Yam
• Yilin Xia
• Bongshin Lee
• Daksh Pokar
• Krishna Anandan Ganesan

• Niranjan Kalaiselvan
• Sanchita Kamath
• Omar Khan
• Aziz Zeidieh
• Aarthy Kesavan Padmanaban
• Sujithra Rajan

The Insights You Can’t See (Yet)

• We’ve been having a monologue with data 👁️

• Time to start a dialogue 👁️👂🖐️💬

• Your hidden collaborators are waiting

• Don’t lose the opportunity to innovate together

So I’ll leave you with this one final thought. I hope my presentation today has given you some new insights - and it has opened your eyes - and your ears, and maybe even your sense of touch - to new possibilities. Just like I’ve introduced new ways of seeing data to you today, I want you to think about this: We’ve been having a monologue with data for far too long, using just our eyes, just one sense.

It’s time to start a dialogue. Your hidden collaborators are out there - brilliant people who experience data differently, who might hear patterns you can’t see, who might feel trends you’ve never noticed.

When you invite them into the conversation, when you make your data truly multimodal, you’re not just being inclusive - you’re opening doors to innovation you didn’t even know existed.

You see, this isn’t just about helping “them.” This is about what YOU are missing. When we design for us, we all win—new perspectives breed new breakthroughs.

Don’t lose this opportunity. The future of data science isn’t just making things accessible - it’s recognizing that different ways of experiencing data lead to different insights. And those different insights? That’s where innovation lives.

Start today. Install maidr. Open your senses. Open your mind. Let’s discover together what we’ve all been missing.

Thank you.

Thank You!

JooYoung Seo

• Email: jseo1005@illinois.edu
• Web: maidr.ai
• Twitter: @seo_jooyoung