Artificial intelligence continues to push boundaries, and at Brown University, one project in particular is at the forefront of explainable AI. ClickMe AI, an initiative from the Serre Lab under Professor Thomas Serre, is leveraging human psychophysics data to build AI models that see the world more like humans do.
The project is driven by a dedicated team, with student leader Jay Gopal playing a significant role in its development and growth. To provide a deeper understanding of the project, Professor Serre shared his valuable insights via an email interview upon request, in addition to Gopal’s contributions. The full interview is included at the end of this article.
READ: Automating healthcare administration: In conversation with Eden’s Naomi Rajput and Evan Smith (January 2, 2025)
ClickMe AI is more than just a research project; it represents the future of explainable AI. With its ability to align AI vision with human perception, the platform has applications in medicine, law, finance, and more.
In an interview with The American Bazaar, Serre and Gopal discuss the inspiration, challenges, and vision behind ClickMe, an explainable AI initiative from the Serre Lab.
The American Bazaar: Can you start off by telling me about your origin story?
Jay Gopal: ClickMe is a large-scale explainable AI project at Brown University in the lab of Professor Thomas Serre—the Serre Lab—and our focus is on building more brain-aligned models of vision. So, we’re working with a lot of deep neural networks and we’re collecting human psychophysics data to try to improve the training procedures for these networks at a very high level.
As for my background, my name is Jay Gopal, I’m a student in Brown University’s eight-year program in liberal medical education, so I’ll be getting an MD in 2029 from the Warren Albert Medical School. I am really interested in medical AI and explainable AI. I’ve been working with AI models since early high school, so I have years of experience working with tools like PyTorch and TensorFlow. I think that’s what drew me to Brown [University] and the Serre Lab, and why I’ve been able to contribute to the Serre Lab for years now, since even before my first class at Brown freshman year.
A big part of what we do is we include students of diverse backgrounds, students from very different educational levels. Even undergrads are very involved in this project. I am a student concentrating in computational neuroscience, so neuro AI is a big passion for me. I’ve taken a lot of graduate-level computer science coursework at Brown with a focus on deep learning. As project lead for ClickMe at the Serre Lab, I’ve gotten the chance to meet a lot of people, and I would say the biggest thing I’ve learned beyond the coding and the conversational skills is how to inspire and motivate other people with a shared vision.
What exactly is ClickMe AI, and how does it work?
ClickMe is an online game where we collect psychophysics data in order to understand where humans think is important in an image. In other words, ClickMe answers the question, ‘What parts of an image do humans think are important for classification?’
It could be an image of a dog, a certain type of dog, and the human might think the ears are very important, so they’ll draw over the ears. And then we can later constrain AI models to be aligned with this view and essentially teach models to look where humans look.
READ: Brown University students launch AI startup Kyron Medical to revolutionize medical billing (November 6, 2024)
This is very important because we’ve had multiple papers come out of the lab over time that show that this type of work is timely, it’s important. A lot of traditional methods of deep neural network training don’t account for the importance of explainability in the same way the Serre Lab is. So essentially, we’re creating, in my view, the next generation of explainable AI models.
You can help contribute to the next generation of human aligned AI models. The research is supported generously by the Serre lab, and we’re able to give out cash prizes every week. Follow this link to play right now and you can directly contribute to improving AI: https://bit.ly/clickmeai
When was ClickMe officially launched, and how has it evolved over time?
The first version of ClickMe was launched a long time ago, even before I joined the Serre Lab. At the Serre Lab, we often operate like a startup in that we like to fail quickly if we do fail to make sure that we can get the right results as fast as we can. And on that front, it’s not like we perfected the website and then launched it at the end. We launched a working website, and we’ve been launching multiple versions of ClickMe.
The version now is up and running, and we are continuing to improve ClickMe. I want to emphasize that ClickMe is not just a finished platform. We’re always coming up with new ways to ensure integrity of the game and make sure that the people who win are playing fairly. We’re also constantly coming up with new incentives to get more players on the platform.
What would you say is the main aim or vision of ClickMe?
We’re collecting data on multiple data sets that are collecting click maps on multiple data sets that are widely used in computer vision, in the computer vision literature.
One of those is called ImageNet.
Now, importantly, we want to collect many maps on each image on ImageNet.
This is important to us because we can check inter-participant agreement and average across participants to get one unified representation of where humans think is important or one unified representation to teach models to look where humans look, right?
So, the vision is to get many maps on all of ImageNet as soon as possible.
And actually, we’ve exceeded all expectations on ClickMe and it has succeeded beyond our wildest dreams. I will say, Professor Serre is a visionary. It’s very easy to become inspired and to share his vision for the future of explainable AI.
What were some of the biggest challenges you faced while working on ClickMe?
One of the biggest challenges is debugging deep neural networks that sometimes take entire days to train. If there’s a small misalignment in preprocessing or model accuracy, it can take time to troubleshoot. But we run controlled experiments before generating results to ensure reliability.
Another challenge is constantly improving ClickMe to handle large-scale data collection. We’re collecting millions of maps and adding thousands of users, so ensuring the system remains scalable is an ongoing effort.
And just a follow up to the previous question, how do you think being a student specifically at Brown University helped you ease those challenges?
First of all, without the detailed mentorship and attention to detail from people like Professor Serre, who has decades of experience in the field and is one of the pioneers of explainable computer vision, a project like this would be impossible.
Brown University gives us access to Oscar, a compute cluster where we do a lot of our modeling. Brown also gives access to an intellectually active ecosystem of like-minded and driven students that we can give opportunities to.
I am very thankful for the ongoing guidance and mentorship from Professor Thomas Serre, Professor Drew Linsley, Thomas Fel, and Ivan Felipe-Rodriguez as we develop a novel training regimen called Harmonization that utilizes the data from ClickMe to create human-aligned deep neural networks.
What hopes do you have for ClickMe’s progression and growth? Where do you see the project in the next few months?
I’m hoping that we can get thousands more users onto the platform and that the data we collect will significantly improve model training. I’m very focused on developing harmonization and building the next generation of human-aligned deep neural networks. We want as many people as possible to participate so that we can create AI models that are better aligned with human perception.
READ: AI to ease healthcare burden: Northeastern students develop ‘Eden’ (November 11, 2024)
The reason that we’re doing ClickMe, the reason we gamified this platform and we’re collecting the data is so that we can do model training. And I really want to include as many people as possible.
Email interview with Professor Thomas Serre
The American Bazaar: What is ClickMe?
Professor Thomas Serre: ClickMe is a game that collects data to help us identify the visual features (i.e., the image regions, i.e., the eyes or the tail of an animal, the wheel of a car, etc) used by human observers when recognizing objects. In prior work, we have shown that while AI systems are now performing on par or better than humans on specific object recognition problems, they are leveraging a very different strategy from the one used by human observers. That is, they use entirely different visual features. ClickMe helps us align AI systems with human vision.
How do you and your Lab envision the long-term evolution of the ClickMe project? Are there specific milestones or ultimate goals you are aiming to achieve in explainable AI?
The long-term vision for ClickMe is to help us bridge the gap between human and machine vision. Our dream is to build AI systems that not only perform at human level but also do so by leveraging human-like visual strategies. This will help increase the trustworthiness of these systems because the visual strategies used by these systems will be more interpretable by humans if they are better aligned.
How does ClickMe handle scaling data collection across large datasets like ImageNet, given the number of images and human annotations required?
Traditionally, psychophysics was run in person in labs across the world. This would typically constrain these experiments to a few dozen participants and on the order of 10K trials/annotations. With ClickMe, by “gamifying” data collection, we can collect on the order of 10M trials/annotations from thousands of participants. Jay led the design and development of “ClickMe 2.0” (our current iteration of the game/website). But this is a team effort, and numerous undergraduate and graduate students contribute to the game in various ways weekly.
ClickMe involves contributions from undergraduates to post-docs. How do you foster collaboration across such a diverse group, and what impact does this inclusivity have on the project’s outcomes?
We cultivate an inclusive environment by assigning roles that align with each contributor’s expertise, ensuring everyone feels valued and empowered. Weekly lab meetings and brainstorming sessions promote open dialogue and knowledge-sharing across all levels. This diversity of perspectives has been crucial in refining the game design and improving the research, as it fosters creativity and ensures the project benefits from both innovative ideas and rigorous scientific methodologies.
How do you view the role of students like Jay Gopal in the success of the lab’s projects?
Students like Jay are pivotal in driving these types of projects forward with their initiative and creativity. They show how undergraduates, medical students, graduate students, and others can work together to produce results.
Jay has been instrumental in leading the ClickMe project while also utilizing the data collected to build brain-aligned deep neural networks with applications across a diversity of fields. Jay sets an inspiring example for how students at all levels can bridge research and practical applications.
What industries do you believe stand to benefit the most from explainable AI models developed through projects like ClickMe? Are there specific medical, legal, or educational applications in mind?
Explainable AI models from ClickMe have broad applications, but industries like healthcare, education, and law stand to benefit the most. For instance, these models can enhance diagnostic tools in medicine by offering clear reasoning behind predictions. Neuroscience also has a lot to gain. As we have shown in prior work, AI models aligned with human vision using ClickMe better explain the response of neurons in our visual system.
Are there plans to collect multimodal data, such as combining click maps with eye-tracking or reaction times, to further enrich the dataset?
Other projects besides clickme involve collecting human data, including reaction times and eye movements, to help us further align AI models with human vision. More to come soon!