Kyungja, by Eunsu Kang

Collaboratively Designing Metrics to Evaluate Creative Machines

Sunday, October 18

15:00pm - 19:00pm (Montreal Time, GMT-4, EST)

ISEA2020 - Why Sentience

Speakers Schedule Questions Participants Call for Participation Metrics Organizers

How do we make a creative machine? This workshop follows a series of studies conducted in the classroom setting at CMU and UC San Diego. For ISEA, participants will collectively find a definition of creativity specific to machines, addressing follow up questions: Can we computationally model ambiguity? Would a novelty search result in valuable discoveries? Where is the threshold between randomness and creativity? How do we evaluate the creativity of an algorithm? To answer these questions, participants develop sets of criteria to assess their own and peer groups’ creative AI and ML projects. Although such a human-centered method is subjective, we anticipate discovering ways to describe and interpret dimensions of algorithmic creativity. This half-day workshop engages a broader arts and machine learning community to collaboratively define these quantitative criteria, in a first attempt to collectively establish evaluation metrics for the area of creative AI.

ISEA 2020 is a virtual event and this workshop will be conducted online. Our invited speakers will provide prerecorded video talks and participate in live panel discussions.

NOTE: This page will be updated frequently as we confirm workshop participants.


Introductory slides from Eunsu Kang: Measuring Computational Creativity (pdf)


Allison Parrish (New York University,

Devi Parikh (Georgia Tech/Facebook AI Research)

Aaron Hertzmann (Adobe Research)

Roger Dannenberg (Carnegie Mellon University)

Fabrizio Poltronieri (Institute of Creative Technologies at De Montfort University,

Haru Ji (OCAD University, and Graham Wakefield (York University,

Jun-Yan Zhu (Carnegie Mellon University)

Ahmed Elgammal (Rutgers University)


tentative schedule (online videos: invited speaker talks, participants will watch it beforehand)

Time                   Activity Location
3:00 - 3:20 Introduction (20 min) main room
3:20 - 3:50 Discussion 1 - Elements of Creative AI (30 min) breakout rooms
3:50 - 4:05 Q & A (15 min) main room
4:05 - 4:40 Guest Speaker Panel 1 (35min) main room
4:40 - 5:10 Discussion 2 - Evaluating ML/Art Projects (30 min) breakout rooms
5:10 - 5:25 Q & A (15 min) main room
5:25 - 6:10 Guest Speaker Panel 2 (35 min) main room
6:10 - 6:30 Discussion 3 - Revising Metrics, Evaluation 2 (20 min) breakout rooms
6:30 - 7:00 Presentation of Results and Q&A main room


Panel 1: Graham Wakefield, Haru Ji, Fabrizio Poltronieri, Allison Parrish, Aaron Hertzmann

Panel 2: Ahmed Elgammal, Devi Parikh, Jun-Yan Zhu, Roger Dannenberg

Additional Questions


This session is an actual working session: together we will collaboratively define metrics for Creative AI.

Tuomas Sandholm Superhuman AI for heads-up no-limit poker: Libratus beats top professionals

Carlos Castellanos PlanetConnect

Austin Lee

Nick Fox-Gieg


Peter Schaldenbrand

Kat Mustatea What is the Value of Art in the Age of Intelligent Machines?

Dana Sperry

Kim Baraka

Rodrigo F. Cádiz

Jie-Eun Hwang

Joel Ong Terra et Venti

Alex MacLean Switch Jockey

Claire Jervert

Aaron Oldenburg Desert Mothers

Amalia Foka The Invisible Structures of the Artworld

Weihua Zhao 404 Not Found

Vít Růžička GAN Explorer

Patricia Alves-Oliveira

Violaine Lafortune Organisme No. 17

Donald Craig

Andrew Brown

Johnny Diblasi

Kiel Howe Baby’s First Image Classification Training Data Set - An Opensource Children’s Book

Jon Paden Dadum

Pablo Sotres Uruguay25

Participants will be updated as they are confirmed.

Call for Participation

Participants in this workshop, as a group, will examine a number of Artificial Intelligence (AI) Art projects, and articulate metrics to evaluate dimensions of creativity in those works. Workshop participants are key contributors to this research in Measuring Creative AI, and we plan for every participant to be named on future mCreativeAI publications (website, papers, etc.) as contributors. This is a novel research project with no prior examples as far as we know, and this workshop will be the inaugural event for this effort as this exercise has only been conducted with students previously.

How to participate: If you are interested to participate, please fill out the following google form by Friday, September 25th, midnight Eastern.


We will send out an acceptance notification by Friday, October 2nd.


Here is our metrics worksheet: google docs

We will publish our results after the workshop.


Eunsu Kang is an artist, a researcher, and an educator who explores the intersection of art and machine learning, one of the core methods for building AI. Her works have been presented at conferences including ACM, ICMC, ISEA, SIGGRAPH Asia and NeurIPS. Kang earned her Ph.D. from DXARTS of the University of Washington, an MA from MAT, UCSB, and an MFA from Ewha Womans University. She was a tenured Associate Professor of New Media Art at the University of Akron and currently is a Visiting Professor of Art and Machine Learning at Carnegie Mellon University’s School of Computer Science.

Jean Oh is a faculty member at the Robotics Institute at Carnegie Mellon University. Her research addresses the creativity in decision making in the problem domains including self-driving cars, disaster response, healthcare, and arts. Her team’s work won the Best Paper Award at the 2018 IEEE International Conference on Robotics and Automation, and also the Best Cognitive Robotics Paper Award at the IEEE International Conference on Robotics and Automation in 2015. Jean received her Ph.D. in Language and Information Technologies at Carnegie Mellon University, M.S. in Computer Science at Columbia University, and B.S. in Biotechnology at Yonsei University.

Robert Twomey is an artist and engineer exploring the poetic intersection of human and machine perception. He has presented his work at SIGGRAPH, the Museum of Contemporary Art San Diego, and his research has been supported by the National Science Foundation, Microsoft, Amazon, and NVIDIA. Twomey received his BS from Yale with majors in Art and Biomedical Engineering, his MFA from UC San Diego, and his PhD in DXARTS from the University of Washington. He is an Assistant Professor with the Johnny Carson Center for Emerging Media Arts at the University of Nebraska-Lincoln, and a Visiting Scholar with the Arthur C. Clarke Center for Human Imagination at UC San Diego.


This workshop is made possible, in part, by generous support from the Johnny Carson Center for Emerging Media Arts.