First, the very talented Dion Barja visited me last year from the University of Manitoba for a University of Waterloo Undergraduate Research Fellowship, funded in part by an NSERC USRA award. He will present our paper “Glass Chirolytics:
Reciprocal Compositing and Shared Gestural Control for Face-to-Face Collaborative Visualization at a Distance” in the Collaborative/Shared XR session on Monday April 13 at 11:15. Read the pre-print at https://arxiv.org/pdf/2603.05864

Next, my co-authors Maxime Cordeil (University of Queensland) and Christopher Hurter (University of Toulouse) will present “Challenges in Synchronous & Remote Collaboration Around Visualization” at the “Methodological Foundations” paper session on Thursday April 16 at 09:00. This paper follows a recent NII Shonan seminar and synthesizes the perspectives of Takayuki Itoh, Wolfgang Büschel, Mahmood Jasim, Arnaud Prouzeau, David Saffo, Lyn Bartram, Sheelagh Carpendale, Chen Zhu-Tian, Andrew Cunningham, Tim Dwyer, Samuel Huron, Masahiko Itoh, Alark Joshi, Kiyoshi Kiyokawa, Hideaki Kuzuoka, Bongshin Lee, Gabriela Molina León, Harald Reiterer, Bektur Ryskeldiev, Jonathan Schwabish, Brian A. Smith, Yasuyuki Sumi, Ryo Suzuki, Anthony Tang, Yalong Yang, and Jian Zhao. Read the pre-print at https://arxiv.org/pdf/2603.05871 and watch my talk video:

Finally, my collaborator Narges Mahyar (City St George’s, University of London) will attend the Workshop on Craft-Based Data Physicalization Wednesday April 15, where she will represent our workshop submission “Community Craft and Embodied Input Physicalization at the Playground,” which we wrote along with Foroozan Danehszand, Negar Ghorashi, and Sydney K Purdue. Check out our extended abstract at https://osf.io/a5u63/files/hz6au.

This workshop will focus on research and technologies aiming to address the main challenges of future remote communication and collaboration around data. With this workshop, our goal is to grow the community around this area of research, especially in Europe, help spawn and advance research collaborations by connecting like-minded researchers, and foster deeper international networking.

Our first MERCADO workshop took place in October 2023 at IEEE VIS in Melbourne, Australia. The workshop featured six workshop paper presentations, a keynote by Tom Bartindale of Monash University, and a panel discussion. Last year, we (Maxime Cordeil, Christophe Hurter, Takayuki Itoh, and myself) continued this initiative with a 4-day seminar at the NII Shonan Village Center in Kanagawa, Japan. Twenty-seven attendees contributed their expertise and perspectives on visualization, computer-supported cooperative work (CSCW), immersive analytics, social XR, and application scenarios relevant to the intersection of these topics.

One of the outcomes of this seminar (summarized in a report) was the identification of four categories of challenges that will motivate future research and ground the discussion at this year’s MERCADO workshop: opportunities for new tools and techniques, how individual differences and interpersonal dynamics affect synchronous and remote conversations around data, the employ of AI assistants or agents in these activities, and methodological considerations for the evaluation of MERCADO interventions.

If you have early ideas provocations to share, or if you are looking to recruit collaborators for projects related to these challenges, I invite you to submit either a short paper or an abstract by August 15th and September 15th, respectively (submission instructions are on the workshop website).

The RemixTape project makes three visualization design study contributions in the enterprise business intelligence (BI) domain. These include a formative understanding of what it means to “remix metrics”, an interactive presentation tool that was grounded in this understanding, and a critical reflection on our design informed by an evaluation with enterprise data professionals.

Motivation

In the past few years we’ve witnessed an evolution in terms of the ecosystem of enterprise business intelligence tools, an evolution centred around metrics (or key performance indicators / KPIs). One manifestation of this was Tableau’s Pulse product, a cloud-based platform for creating, collecting, and sharing views of metrics. Tableau was not the only player in this evolution, as several major enterprise BI vendors have moved toward a so-called “headless BI” model of cloud-based metric layers and away from the idiosyncrasies of individual dashboards. Across an enterprise BI deployment, metrics now appear in dashboards, but also found in gallery collections or on presentation slides used to inform business decisions.

Meanwhile, our own work with an interview study with Tableau customers on the topic of remixing visualizations and dashboards. This complements another interview study conducted at Tableau around the same time by Melanie Tory and colleagues, where interviewees did however express an interest in “slicing and dicing” data artifacts like published dashboards, workbooks, visualizations, …and now metrics. The key findings from our interviews included a desire for an interface design to collect and repurpose visualization content, an “additional workspace where you can rearrange and move stuff from different dashboards.” Such an interface would also allow you to arrange and coordinate this content in a storyboard, complete with commentary and annotations that you could present to your team. Finally, it would allow you to draw specific attention to metrics (or KPIs): quantitative values that change over time.

Combining what we learned from the remixing interview study and by the industry focus on metrics, we arrived at three design imperatives: First, to support arranging, slicing + dicing, and semantically aligning sequences of charts depicting metrics; Second, to support the enrichment and annotation of these chart sequences with situational context; And third, to provide context-aware recommendations of complementary metric charts that enrich a partially-constructed narrative sequence. With these imperatives in mind, we build RemixTape between 2021 and 2022.

Inspiration from Music Production

The name RemixTape is a portmanteau of two terms: the first is remix. As inspiration, we looked to the original connotation of remixing, to music production and the combining of musical samples from different sources. And as inspiration for our interaction design, we looked to the slicing and dicing and arranging of these samples as performed in contemporary digital audio workstations. The other part of the name is “MixTape”. In a mixtape, a narrative forms when arranging a curated set of songs by different artists or producers in a particular order. Mixtape also has a broader meaning if you’re familiar with mixtapes in hip hop culture in which an MC raps over sequences of mashed-up samples.

Application Design

With this inspiration in mind, let’s take a brief look now at its functionality. RemixTape is a presentation authoring tool with multiple views. There’s a presentation canvas where we find scenes containing visualization cards and text cards. We have a browser to collect metrics from different data sources, and a recommendation list of new visualization cards to add to the canvas.

Speaking to our first design imperative of arranging and semantically aligning sequences of charts, we implemented interactions on cards that allow us to slice and dice charts, as well as juxtapose and coordinate charts by truncating and relativizing their scales and when two charts are adjacent, we reveal an affordance to merge them and superimpose their metrics.

Speaking to our second design imperative of adding situational context to charts, we added functionality to both annotate + obfuscate parts of charts along with the ability to add free-form text commentary and interactively coordinate a text card with its adjacent visualization card with linked temporal highlighting:

And finally, our third imperative was to provide context-aware recommendations of complementary metric charts that enrich a partially-constructed narrative sequence. These could be recommendations of broader overview views of metrics already on the canvas, or views that zoom in on notable spikes or troughs, recommendations to break down a metric by a categorical dimension, and cold start recommendations of metrics exhibiting notable variation, a type of recommendation which I’ll reflect on momentarily.

Finally, there’s a presentation mode of the app that hides the authoring UI, which could be used as an interactive alternative to presenting static slides.

Scenario

On my YouTube channel you’ll find a 5 minute end-to-end scenario depicting how a real-estate analyst remixes metrics from residential listings, COVID-19 cases, and weather to construct a presentation about real estate trends.

Evaluating RemixTape

In the evaluation of RemixTape that we conducted with six enterprise data professionals, we asked them to reproduce and then extend a narrative presentation involving several sets of metrics, our full methodology and observations can be found in the paper. For now, I’ll focus on our key high-level takeaways from this evaluation. First, in this post-dashboard era of business intelligence, product designers should exercise caution when proposing alternative narrative BI experiences that deviate from the expected conventions of dashboard or slide presentation interfaces. Second, designers should consider scene + card-based presentations about metrics with cards of varying visibility and prominence, and by that I mean thinking about selectively hiding and showing cards, or allowing for multiple levels of hierarchy between canvas, scene, and card. Third, with respect to recommendations, their scope should be narrow: otherwise, recommendation features give an impression that the application is intended for exploratory data analysis rather than for communicating narratives.

Next Steps

Plans to productize aspects of RemixTape are unknown at the time of writing, though two patents related to the project were issued in 2025: Semantic alignment of text and visual cards to present time series metrics (US12229856B2) and Visualization recommendations for time-series metrics presentations (US12260478B2). Several additional patent applications are pending.

In terms of functionality, one potential future consumption experience for interactive RemixTape narratives could be a swipe-able sequence of cards embedded into a communication / collaboration platform like Slack or Teams (echoing an approach from our 2024 VIS paper).

Finally, with respect to future research directions, RemixTape represents a point in a nascent design space for visualization remixing, one worthy of further exploration. It also serves as a call to mine the interface and interaction design paradigms of music production for applications to visualization and visual analytics.

Matt Brehmer joined the Cheriton School of Computer Science as a tenure-track Assistant Professor in August 2024.

Previously, he was a lead researcher at Tableau Research in Seattle, where he specialized in new experiences for communication and collaboration around data. Before joining Tableau, he was a postdoctoral researcher with the Extended Perception, Interaction & Cognition (EPIC) and Human-Computer Interaction groups at Microsoft Research. He completed his PhD and an MSc at the University of British Columbia, where he was a member of the InfoVis group and the Multimodal User Experience lab. He also holds an undergraduate degree in cognitive science from Queen’s University.

In 2022, Professor Brehmer was elected to the Visualization Conference (VIS) Executive Committee, which oversees the planning and success of the IEEE VIS conference, and he was appointed to the IEEE Visualization and Computer Graphics Technical Community’s executive committee. He is also dedicated to connecting research with practice: he currently serves as an advisor to the Maryland Institute College of Art (MICA)’s Master of Professional Studies in Data Analytics and Visualization, has served on the executive nomination committee of the Data Visualization Society, has co-organized the VisInPractice event at IEEE VIS, and has spoken at practitioner events including OpenVisConf, the Microsoft Data Insights Summit, and the Tableau Conference.

As of February 2025, Professor Brehmer’s research has been cited more than 2,900 times with an h-index of 24 according to Google Scholar.

What follows is a lightly edited transcript of a conversation with Professor Brehmer, where he discusses his research, his advice for aspiring computer scientists, and what excites him about joining the Cheriton School of Computer Science.

Tell us a bit about your research.

My research specializes in information visualization. It’s a blend of human-computer interaction (HCI), graphics and databases, but I come at it from the HCI side. Since my PhD, I’ve been focusing on visualization in the context of storytelling and visualization beyond the desktop, or beyond mouse and keyboard interaction: from tangible interaction on mobile devices to gestural interaction in augmented reality environments.

Currently, I’m focusing on what I call “ubietous information experiences.” “Ubiety” is a rare but useful word with a specific meaning: it means being sensitive to place. There’s a related term that people may be more familiar with, “ubiquity,” which is about being “everywhere, all at once,” with an implication of pervasiveness. When I say “ubietous information experiences,” it is information designed for a specific place and time: using the right information in the right presentation format at the right time. I’m interested in interactive and dynamic information visualization that fulfill this criteria, particularly for places where people collaborate and communicate, either in person or virtually.

What do you see as your most significant contribution?

It depends on who you talk to. I’ve had one foot in the academic world and another in the practitioner / industry world.

Among the academic visualization community, many are likely familiar with some of my PhD dissertation. It was a structured way of describing why people visualize data. It is an approach to classifying tasks that is both hierarchical and sequential. It helps visualization designers and researchers articulate tasks in terms of sequences of actions, allowing them to determine what visualization or interaction design works for a particular task or activity.

Researchers found the related paper from my dissertation to be a useful way of thinking about visualization tasks. Whether that was in the context of designing experiments, justifying design choices, or doing fieldwork to observe how people use visualization tools and techniques. Before, visualization design was more of a process of “I have this data, …so which techniques could I use to visualize it? Instead, our paper can link someone’s goals (why) and their data (what) with techniques (how). One benefit of this approach is that it can lead to cross-pollination of ideas across application domains with common task and data abstractions.

This paper, co-authored with my PhD advisor Tamara Munzner, was the centrepiece of my dissertation, and I used it throughout the other chapters, including design or evaluation projects. It continues to be widely cited. I’ve seen people using it to explain what tasks they chose for a particular user study or an evaluation. And at the 2023 IEEE Visualization Conference (VIS) conference, we received a Test of Time award for the paper. It was great to see that this work still resonated 10 years after it was published.

Meanwhile, I’ve been in industry for the better part of the last decade. The visualization practitioner community may know me as an expert on time-oriented data visualization and timelines. I’ve spoken about this at industry conferences and visualization practitioner events like OpenVisConf. I’m also a member of the Timeline Consortium, which has proposed a standard for timeline data. Most recently, I was working with Tableau in Seattle, which is devoted to helping people see and understand data, building data visualization and business intelligence software. The Tableau “DataFam” might know me as someone who has worked on bringing Tableau to new modalities and platforms, including spatial computing with Tableau for visionOS. Another project, Tableau Gestures, focused on compositing interactive data visualization over video in live teleconference calls: you could use gestures and body language to point at data and direct your audience’s attention without using screen-sharing.

What challenges in HCI, data visualization and ubietous information experiences do you find most exciting to tackle?

The journalist and writer Ellen Ullman has written about how technology has brought about a “disintermediation” between people since the dot.com era and through the rise of social media and AI. There is a mentality that the individual customer or user always knows which data, product or service best suits them. But there remain many situations in our daily lives and professional environments where we benefit greatly from intermediaries. So, you could say that I’m broadly interested in “reintermediation.” Another way of putting it is that I want to augment and mediate rather than replace or automate human-to-human communication. Particularly in cases where there are asymmetries in knowledge levels, like between teachers and students or journalists and the general public. Information visualization can help mediate between people with different levels of expertise, especially during real-time communication. How can you have an engaging and informative experience or interaction with another person that’s augmented with information? Or can data-rich hybrid or remote meetings be something that’s qualitatively different to in-person meetings while still being similarly rich and informative? Some of my research is driven by the belief that there’s great potential in augmented reality technology and using multiple modalities to present information in these contexts.

There’s also a growing body of research about AI assistants, and how they might help you perform a task in real-time. I’m interested in applying that concept to multiple people having a meeting that is grounded in data. How can AI assistants help people reach a consensus or agreement? Could they raise everyone’s understanding to a baseline level during the meeting or presentation? Can they help with negotiation or conflict resolution?

Finally, it’s not clear how we evaluate these interventions that we build. They’re inherently collaborative. They’re often distributed, especially if you’re talking about supporting remote or hybrid meeting scenarios. Often you don’t see the effects of these interventions after one meeting. It’s really challenging to determine how to validate the efficacy of some of the work that we do. What methodologies should we consider? Even beyond HCI, do we look to the humanities and educational psychology? How do we test these interventions that we build, both longitudinally and in these distributed formations of people, where multiple people are involved in a classroom, discussion or decision-making activity?

How could AI help people to come into agreement?

Even in systems like Zoom, there’s an AI assistant that’s built-in. It’ll give a distilled summary of the meeting, such as who spoke and what were the major topics and action items. But these summaries are retrospective. A potential intervention would be used as the meeting unfolds, where we could show representations of data that are relevant or complementary to the conversation. Hopefully, this intervention allows people to reach a common understanding or provide some participants with the missing context to help them catch up.

What advice would you give to students interested in pursuing research in your area of expertise?

HCI and visualization researchers and practitioners come from many different disciplines. So, having a multidisciplinary background is a tremendous asset if you want to pursue research in these areas. Students can seek out opportunities to work with someone in a particular domain that they don’t have previous knowledge in. These experiences could be rich sources for informing new techniques or applications that could potentially be used across other domains.

It can also be useful to read broadly, so you can see how others relay complex data-rich concepts to a lay audience. I like to read popular science, science history, science communication, and long-form journalism. These genres or mediums have the same goal as my research, just with a different modality.

Finally, it’s helpful to have side projects where you can collect and visualize data in different ways. Maybe it can be your personal data like your daily exercise or step count, or things you’re interested in like video games or manga. Even if your project doesn’t lead to a novel technique, it’s a way of practicing the craft and building your portfolio. Put your work out there and receive feedback. I recommend getting in touch with online communities, local meet-up groups or professional societies like the Data Visualization Society, which started in 2019: it is really welcoming and has a number of programs dedicated to early-career professionals.

Do you see opportunities for collaborative research at the Cheriton School of Computer Science?

Yes, it’s a big reason for what drew me here! The HCI group within the Department of Computer Science and the broader HCI community at the University of Waterloo are well-known, respected, and varied in terms of their interests. I was already collaborating with Professor Jian Zhao and one of his students before joining, and I look forward to more collaborations like this in the months and years to come.

Starting this term, I’ve started collaborating with some of our computer science undergraduate students through the Undergraduate Research Assistantships (URA) program. I knew that Waterloo’s undergraduate students are strong, and I’ve been impressed so far with their fresh perspectives and initiative.

Beyond the HCI community at Waterloo, there are also some exciting research opportunities that could be fun to collaborate on with other department members. Explainable AI is an emerging topic in the visualization community: using interactive visualization to explain how these algorithms or processes work. More broadly, CS education present many opportunities for using data visualization in explorable explanations. Finally, I’m also keen to work on visualization in the context of the ethics of computing, such as with visualization for platform or algorithm audits to illustrate how data flows through these sociotechnical systems.

What aspect of joining the Cheriton School of Computer Science excites you the most?

I’ve returned to academia after spending nearly a decade in industry. There, I would have a PhD intern for about 12 weeks for a very focused project. I had many ideas, but I could only hire one intern per year. So, having this ability to form a group around my research ideas is very exciting. This term, I’ve started working with undergraduate students through the Undergraduate Research Fellowship and URA programs. These programs make it easier to work with students at different levels of their careers, and I’m eager to explore other similar programs. Another aspect of working in industry research was that I had to align my projects with a productization mindset and with top-down corporate goals like increasing productivity or monetizing new product features. Here, I can have a broader range of projects and outcomes, and I feel more free to work on curiosity-driven ideas.

Who has inspired you in your career?

My PhD advisor Tamara Munzner certainly influenced my early career work. Since then, another frequent collaborator has been Bongshin Lee, who’s now at Yonsei University. I’ve had so many inspiring discussions with both of them.

Beyond my collaborators, Bill Buxton is also an inspiring figure. I first met him when I was undergrad, when he received an honorary doctorate at Queen’s University. After my PhD, we ended up being on the same team for a while at Microsoft Research. He was remote, though, so we didn’t work on projects together. But what I admired about Bill is the breadth of knowledge he brought to his visits. It was him that introduced me to this concept of “ubiety,” or designing something that appears at the right place at the right time. It’s been years since I learned about that term from him, but it’s still stuck with me. I find it a useful framing concept for what I’m thinking about now.

There are also historical figures that I find inspiring, especially in the field of visualization and information design. One such figure is Otto Neurath, a member of the Vienna Circle, from around 100 years ago. Along with Gerd Arntz, he pioneered this system of pictographic representation called Isotype for communicating statistical information to the working class, so that they might better understand their working conditions and the inequalities they faced.

There are also contemporary visualization and information design practitioners that I greatly respect like Alberto Cairo from the University of Miami, a mediator between the visualization and journalism communities. Then there’s Nigel Holmes, who was at Time Magazine for the longest time, who created information graphics were really fun and whimsical. Like Neurath, he uses pictographic elements to draw people in and help them understand the story. Finally, Giorgia Lupi has designed detailed information graphics that would be the equivalent of a long-read piece of journalism that you would sit down and read on a Sunday afternoon. Lupi’s work is beautifully done, incorporating visual metaphors and pictographic forms to convey multi-dimensional information to a general readership. As you can tell, I’m generally inspired by people who have this shared goal of conveying complex topics or statistics to a general audience.

What do you do in your spare time?

I have an active toddler, so there’s not a lot of spare time between researching and being in “dad mode.” But when I need to unwind, I like to read for leisure, both fiction and non-fiction. With respect to the former, my favourite literary genre is hysterical realism. As for the latter, I typically like books covering topics at the intersection of technology, culture, and history.

I’ve also been collecting vinyl records for my entire adult life. I like to unwind by putting on an LP, sitting back and just listening to it with a good pair of headphones. I listen to a lot of genres and eras, but in general, I like music with a lot of dynamic range, varied instrumentation, and a fair bit of reverb. Recently, I discovered that the Waterloo Public Library has a vinyl record library, which I’ve been enjoying.

When the weather’s good, I like hiking. I miss the mountains of the West Coast, but it turns out that Southwestern Ontario isn’t as flat as I thought it might be. I also enjoy being out on the water. I started paddleboarding during the pandemic. Now, it’s something my dog and I do in the summertime. When we were in Seattle, we enjoyed paddling in Puget Sound.

Before my dad era, I would spend my winter weekends and evenings on music production. Once my toddler is older, I hope that we can jam and produce something together.

Earlier this year at the Tableau Conference, attendees got a preview of Tableau Gestures, a new way to present data to remote colleagues and customers, one that overlays interactive visualization over webcam video. Instead of screen sharing and hoping that the audience follows along, Tableau Gestures allows presenters to point directly at parts of a chart to highlight and annotate it. In case you missed the showstopper of a demo, be sure to check out the Tableau Conference opening keynote on demand here. You can also watch Iron Viz host and Tableau evangelist Andy Cotgreave demo Tableau Gestures here. Since these demos, we’ve taken Tableau Gestures further by adding speech interaction. 

When someone uses hand gestures while speaking, these are known as co-speech gestures. These gestures are particularly useful when using a visual aid (like a chart) to direct attention: for example, pointing gives context to words like “this”, “that”, or “those”, which could refer to marks or values in a viz. With Tableau Gestures, we use computer vision to take advantage of this connection between a speaker’s gesture and a viz to highlight, draw reference lines, and annotate marks with helpful labels that draw the audience’s attention. 

Highlighting and annotating are not the only things that need to do be done when presenting data. Often the data in a chart needs to be filtered, aggregated, disaggregated, or sorted. Certainly, some of these decisions can be anticipated for a scripted presentation, and the appropriate charts can be constructed in advance. However, whether in the flow of an improvised presentation or in response to detail-oriented questions from audience members, presenters cannot rely solely on pre-constructed charts. Unfortunately, there does not exist an obvious set of hand gestures for filtering, sorting, or otherwise transforming charts; even if such a set of gestures existed, they might require more preparation and concentration, they might be distracting to audiences, and they might interfere with gestures used by different cultural groups. 

In response, my Tableau Research colleague Arjun Srinivsasan and I considered using the modality of speech to transform a chart while retaining the existing benefits of gestural interaction. We modified the Tableau Gestures application to additionally listen to voice input, and we demonstrated the results in recent video. We also described our approach in a recent paper that Arjun presented at the MERCADO workshop, a half-day event held at the IEEE VIS conference in Melbourne, Australia last month, the preeminent forum for visualization research. The workshop was devoted to the topic of Multimodal Experiences for Remote Communication Around Data Online, making it the perfect venue to demonstrate our idea and solicit feedback from experts from both academia and industry working in this burgeoning area of research.

Our three-minute demonstration presentation (illustrated in the annotated photo montage and in the video) illustrates a scenario that combines speech and gesture interaction. This presentation centers around a dataset of about 500 American colleges containing both categorical dimensions (e.g., Region, Type) and quantitative measures (e.g., SAT Average, Average Cost). 

To initially orient audiences to the data, the presenter begins by showing a pre-specified unit column chart, where each college appears as one mark and is grouped according to its geographic region. In explaining the distribution of colleges, the presenter points their finger at different columns, highlighting the number of colleges in each region. 

Next, the presenter indicates that the Mid-Atlantic and Southeast regions have the most number of colleges, and he presses a button on his remote clicker (or a key on his keyboard) to trigger voice input while continuing his monologue, stating that “we can see that Southeast and Mid Atlantic have a higher number of schools than the other regions, so let’s go ahead and drill down into the Mid Atlantic and Southeast regions to learn more about the schools here.” After uttering this, colleges from regions aside from Mid-Atlantic and Southeast are removed from the chart.

Moving on to focus on the differences between college types, the presenter states “I’m a little curious to see what the distribution of different school types in these densely populated regions is, so to do that, maybe what we can do is color these points based on the school type that they have.” Following this utterance, the initially white marks turn orange or blue depending on if they represent Public or Private colleges, respectively. Highlighting the associated college types by pointing at the color legend, the presenter then draws attention to the fact that there are more Private schools than Public schools in the two regions.

Delving deeper into the two shortlisted regions, the presenter then moves on to focus on the competitiveness of colleges using the Admission Rate attribute as a measure for the same. Narrowing down to schools that have low admissions rates, the presenter says “… let’s go ahead and focus on schools that have an admission rate of less than 25%.” Pointing at the highlighted colleges, the presenter calls out that there are only three public schools with such competitive admission rates, whereas there are sixteen private schools. Discussing if the type of area a college is located in could lead to more insight, the presenter then says “…to see that, let’s update this layout to group schools based on the locale that they belong to.” This results in the view changing to a new unit column chart with colleges being spatially grouped by their locale type (e.g., Large City, Small Suburb).

To summarize, throughout this presentation, the presenter uses gestures to highlight and annotate (groups of) marks in the chart along with speech utterances that transform the chart via filtering, aggregation, re-coloring, and sorting. Important to note is that this presentation only required specifying a single chart in advance; all of the transformations to the chart were performed on the fly.

We’re continuing to demonstrate Tableau Gestures and collect feedback from customers. We aren’t yet sure if speech interaction will continue to be part of its development; it will undoubtedly require further research, prototyping, and focused feedback from customers. For instance, we don’t want to rely on a keyboard hotkey or remote clicker to trigger speech capture, but we also want to limit unintended transformations to the viz. When (not if) these unintended transformations take place, we need to make sure that they can be easily undone or recovered from without significantly disrupting a presenter’s delivery. We also need to better understand how this speech interaction flows in the context of (or inhibits) natural-sounding speech. It is one matter to utter a terse imperative to a computer when no one is listening, but these utterances might seem unnatural to audiences watching someone give a presentation. Finally, we’re curious as to what else we can achieve with multimodal interaction in the context of augmented reality video presentations, such as whether the combination of speech and gesture could be used to create new charts on the fly, provided a data source is available. 

In his 2016 essay collection The Weird and the Eerie, the late writer Mark Fisher set out to define these two terms. He references a selection of critically notable novels, films, television series, and pieces of music that evoke the weird and the eerie. And his writing led to my realization that data visualization may also be capable of instilling these unique affects.

Fisher argues that the weird and the eerie are distinct from one another, even though they are both often associated with the genres of horror, science fiction, and post-apocalyptic fiction. However, the weird and the eerie are not necessarily horrific or devastating.

According to Fisher, the weird is “that which does not belong,” that is, the presence of something that seems wrong or at least strange; the placement of unfamiliar things within a familiar context. And it’s not necessarily a negative affect; there can be enjoyment in seeking out the weird. 

In contrast to the weird, Fisher defines the eerie as a reaction to situations “when there is something present when there should be nothing, or when there is nothing present when there should be something.” In both cases, there is a question of agency, a question of who or what is responsible for the lack of absence or the lack of presence, and what motivated — or continues to motivate — this unseen agent.

In my essay, I contextualize the weird and the eerie by discussing a few related affects in visualization and HCI research: fear, thrill, uncertainty, ambiguity, doubt, a profound feeling of loss, and even humor. 

Ultimately, the affective goal of communicative visualization is persuading viewers to take action. To compel to action, however, first requires strengthening or changing viewers’ beliefs. Encountering the weird is an opportunity to change what one believes; Fisher remarked that the weird forces us to acknowledge that “the weird thing is not wrong [—] it is our conceptions that must be inadequate.”

Another motivation for this essay stems from my interest in visualization techniques that integrate semantic cues into encodings of data. While it may be easy to imagine semantic cues that indicate how particular visual patterns exhibited by the data are good or bad, it is less clear how to effectively communicate that some patterns are simply weird or strange. Similarly, it is not universally obvious how visualization designers can communicate that a particular pattern is present or absent, as well as how to express a lack of an explanation for these presences or absences. 

Visualizing the weird

Visualization practitioner Andy Kirk remarked that people “are naturally drawn to gaps and exceptions and things that don’t really fit in with the rest.” And yet, designing representations of data that emphasize these aspects can be “a very fiddly challenge.”

Real datasets “frequently contain outliers, missing data, and “just plain weird” distributions.” [Leland Wilkinson and colleagues, 2006] A professional data analyst is trained to spot these weird distributions, distributions that seem strange or wrong given the context of the data. But how can the analyst effectively communicate that these distributions are weird to a lay audience?

Despite the precision and generalizability that scatterplots and their variations offer, using other forms of representation can, according to Enrico Bertini and colleagues, promote “serendipitous discovery, educational impact, hedonic response, or changes in behavior”. Like the hedonic enjoyment of the weird in popular culture, it is similarly possible to enjoy weird visualization design choices. 

In other cases, weird choices of representation can accentuate weird patterns in data. Consider the connected scatterplot: it can resemble a line chart, but it can also exhibit visually prominent line reversals and loops, indicative of weird events in the relationship between two variables. Absent weird events, the connected scatterplot is often a poor choice for showing bivariate temporal relationships

The connected scatterplot is not alone in its capacity to draw attention to the weird. Visualization designer Maarten Lambrechts has amassed a catalog of techniques that he calls xenographics, or “weird but (sometimes) useful charts.” The concept of xenographics suggests that given a weird observation in data, there may (or should) exist a suitably weird way to represent it

Visualizing the eerie

Fisher’s definition of the eerie as a failure of presence brought to mind Nicole Hengesbach and colleagues’ review of the four types of missingness in data. If an entity exists in reality and is either completely or partially captured as data, the representation is complete (or at least partially complete). However, if the entity is present in reality but unaccounted for in the data, this is an absence, while the converse is an emptiness. Lastly, there is nothingness: entities are neither present in reality or represented in the data. Given these categories or missingness, an absence or an emptiness could be seen as eerie if there is ambiguity with respect to the processes of data collection and redaction: who was responsible, what their motivations were, and if any external forces acted to bring about absences or emptiness. Nothingness can also be eerie, even when the processes of data collection are relatively transparent, and particularly in cases where something or anything was expected.

In communicative visualization design, evoking an eerie affect around absences in data requires establishing what expected patterns look like or how they have manifested in the past, contrasting presence with absence, and withholding or deferring possible explanations for this absence. An example of this formula is Isao Hashimoto’s 1945 – 1998, a multimodal combination of visualization and sonification, animating every atomic detonation during this time span on a world map, with different colors and tones indicating the nation responsible for each detonation. Following several cacophonous decades with hundreds of detonations and a barrage of color and sound, the last decade is markedly quieter: a relatively eerie calm has set in. By withholding any verbal or written explanation in the video itself accentuates an eerie affect.

An eerie affect induced by a failure of absence could arguably be easily triggered given our propensity to apophenia, a bias in which we recognize patterns and attribute their appearance to an entity (or entities) that have agency. In most cases, however, there is no agent, and the pattern is fleeting, illusory, or insignificant. Visualization designers must exercise care to ensure that their viewers do not succumb to apophenia unless this is the intent, such as in Michael Brenner’s humorous Viz in the Wild project, in which spurious visual patterns appearing in photographs are captioned as charts.

However, in select cases, there is an explanation for salient patterns where none were expected or designed. Dietmar Offenhuber’s notion of autographic visualization is useful here, as the interpretation of material traces left by environmental phenomena can often yield meaningful measurements as well as revelations regarding the process of data generation. Meaningful measurements depend on several design operations, including the juxtaposition of frames, annotations, and decoding scales alongside material traces. By undertaking these operations and measurements, practitioners can arrive at explanations for the patterns, which may include an attribution of agency, such as air pollution caused by human industry and transportation. 

A sense of the eerie is instantiated when an autographic visualization interpretation either fails to attribute this agency or fails to explain the motivations of an identifiable agent. It is similarly eerie when an expected material trace vanishes, and we are bereft of any explanation for this disappearance. I’ll offer a personal example: for years I regularly visited a Florida beach that was pockmarked with tiny holes made by sand fleas, a visual pattern distributed evenly along the shoreline. One year, this pattern had vanished altogether along one stretch of the beach, and in that moment I could not explain this lifeless failure of presence. (I would later learn that the adjacent resort hotel had began importing sand from elsewhere, destroying the natural habitat of the fleas.)

Visualization for weird and eerie time

We are living in weird times: from weird weather fluctuations to invasive species, algal blooms, and disruptions to currencies, real estate markets, and supply chains. How we represent these and other weird happenings, whenever we capture them as data, should be commensurately weird, particularly if we are to communicate just how weird they are.

For example, new visual idioms like Ed Hawkins’ Warming Stripes are useful in communicating how weird extreme weather events are, though even these elicit questions of how we should visualize future extreme values: while we could renormalize the color palettes to make deep blues and deep reds more apparent, renormalization may fail to capture the severity of the next weird event.

Our world is also increasingly eerie. Jenny Odell recently wrote about the feeling of living in apocalyptic times, which is fitting given how post-apocalyptic films and novels often evoke a sense of the eerie. Biodiversity loss has resulted in eerie landscapes and oceans, while human migration has resulted in eerily empty urban centers, empty rural settlements, and conversely crowded interstitial spaces: Why are there absences in places where entities are expected, but present in places where they are not? 

Odell’s writing also mentions that the etymology of apocalypse is Greek, meaning to reveal, and that prior to its modern English usage signifying ‘an ending’, apocalypse was closer in meaning to insight. Given the old adage “the purpose of visualization is insight, not pictures”, could there be some utility in the notion of “apocalyptic data visualization”? In experiencing the eerie, each viewer can arrive at some degree of insight, that is, to identify the forces acting on world and the data collected.

In conclusion, for much of what could be constituted as weird or eerie in our world, we have the potential, via data visualization to document this reality, and ultimately, the potential to change what viewers believe. I leave you with several design implications, including some (*) that I didn’t discuss here, so please read my full essay if you want to go further down the rabbit hole of the weird and the eerie:

• Acknowledge situations where it is appropriate for data visualization to make viewers uncomfortable.

• Avoid manipulating viewers’ impressions by pairing an expected or familiar data observation with a weird choice of representation, reserving the grotesque art of xenographics * for communicating truly weird patterns in data.

• Employ egocentric perspectives and doorway motifs for distinguishing the familiar from the weird.*

• Provide subtle indications of agency via animation.*

• Withhold or delay revelatory explanations or annotations for failures of absence and failures of presence.

I want to thank the IEEE VIS community for recognizing the value of this work.

At the outset of my doctoral studies, I aspired to make a methodological contribution to the field of visualization: to rethink how we evaluate visualization experiences beyond controlled experiments of response time and accuracy, to introduce a qualitative approach leveraging my background in HCI and cognitive science.

The challenge I faced was that an evaluation should result in implications applicable beyond a single application domain. This challenge manifested in the first two projects of my PhD as a lack of a consistent code set for abstractly describing why and how people visualize data. Because of this challenge, these projects stalled.

It turned out that Tamara also faced this challenge in the process of writing her book.

In response, I set out to read everything related to tasks, interactions, insights, both communicative and analytical goals with respect to visualization. While this literature survey is represented in the paper, the other and more exhaustive forms of this literature review include an annotated bibliography appearing in my dissertation and an interactive wiki, which is still accessible today at mattbrehmer.ca/wiki.

The paper followed several months of intense diagramming, white-boarding, and marathon one-on-one meetings with Tamara. The paper was originally titled Mid-level tasks for Visualization Design and Evaluation, aiming to fill the gap between prior high- and low-level classifications. This evolved into the Multi-Level Taxonomy of Abstract Visualization Tasks, and thanks to a knowledgeable reviewer who pointed out the difference between an exhaustive taxonomy of objectively observable entities and a typology of abstract classes, we finally settled on A Multi-Level Typology….

InfoVis 2013 was our first attempt at submitting this work, and the reviews were mixed but leaning positive, with scores 3, 3, 4, and 5, with a summary score of 3. The primary reviewer wrote: “The InfoVis conference might not be the best avenue to deliver this work. A journal article (such as a regular TVCG paper and InfoVis journal) might be better venues.” However, with a strong revision and cover letter, we managed to win them over.

Presenting this work was my first time speaking at VIS. I still recall the session well. I presented last, following two other thematically similar presentations by Robert Roth and Hans-Jorg Schülz. The session chair was Bongshin Lee, who I would later collaborate with extensively. After my presentation, first to the microphone to comment were Ben Shneiderman and my Tableau future colleague Jock Mackinlay. Talk about intimidating for a newcomer to VIS! Both expressed enthusiastic appreciation for the work, for which I’m grateful.

In addition to its later incarnation in Tamara’s book, this typology would also be the backbone of my dissertation work (see the chronology diagram). I used it in two evaluation studies presented at VIS 2014, and in a design study presented at VIS 2015. Since graduating, this typology has also informed my subsequent work and collaborations, from contextualizing experimental tasks at VIS 2018 to a grammar for specifying dashboards at VIS 2020, and most recently with respect to Tableau Research’s open source Visual Analytic Intent Language.

At the end of my dissertation, I reflected on the impact of this work and how others in the community had used it in their work between 2013 and 2016. I suspect this is still true in the years since, but I observed that by citing this work, many people were citing the terms used in Figure 1 and Table 1: the classification of why and how people visualize data. I would challenge new readers of this paper to read beyond Figure 1 to learn about how to use the typology in the context of sequential and hierarchical task analysis, which I demonstrate in the other papers of my PhD.

A final reflection. This paper was directed at the visualization community. It is not very accessible to newcomers. The summary reviewer stated “It would be difficult to understand for the people who are new to this topic.” Neither is the paper very friendly if you’re a practitioner. It prompted my Tableau colleague, Iron Viz host, and author of The Big Book of Dashboards Andy Cotgreave to ask in a tweet “Why is the language so intentionally obtuse?” Sorry, Andy. He would later add: “Don't get me wrong, the info is generally utterly fascinating, but some of the conventions feel like barriers.” We can and must do better as a field to ensure that our papers have reach beyond visualization researchers.

I’d like to thank many people who provided ideas, feedback, and support related to this work: the UBC InfoVis Group circa 2011 – 13 (Stephen Ingram, Miriah Meyer, Michael Sedlmair, Jessica Dawson, and Joel Ferstay), my co-advisor Joanna McGrenere, Ron Rensink, Colin Ware, Silvia Miksch, the InfoVis reviewers, Tamara (or course), and lastly my partner and colleague Ana Crisan.