AI Toy Dissection Remix with Cognimates

Last week when I was in New York for the Fab Learn conference, I was invited to be guest presenter at friend of WICO Stefania Druga’s class NYU ITP on “hacking smart toys for AI learning”.

It was really inspiring to discuss the prototypes that her students are working on and share about WICO’s experiments with toy dissection activities and installations. The next day at the conference, Stefania and Nancy Otero (along with the students from the seminar) shared their research on democratizing AI and giving kids and adults tools to create with this new technology. Although there are many interpretations for using AI in education, I’m most interested in platforms like Cognimates that allow learners to take control of the technology and be creative with the tool.

At the toy dissection and remix workshop that I led at FabLearn with Joel Gordon and Christa Flores we introduced possibilities of adding Micro:bit programming to the toys. After the trip, I was inspired by deeper conversations with Stefania to try to connect her collection of AI based scratch extensions on the Cognimates website to make an “smart” twisted toy hack.

Here’s a description of my starter project:

First I grabbed a stuffed bunny that survived the workshop at FabLearn and collected the dissection tools including scissors, an exacto blade and wire snips.

I removed the mechanisms from inside the toy and set them aside (eventually it would be great to reuse the motors/lights from the toy but right now the Scratch/Cognimates/Micro:Bit extensions can’t control outputs).

At first I tried to connect the toy speaker to a headphone jack to play the sounds from the computer, but the sound was too low to really hear.

So I grabbed a bluetooth speaker, a Micro:Bit and a computer and started exploring the Cognimates extensions. I taped the Micro:bit on the inside of the bunny’s belly with the lights facing out and I stuffed the speaker on the inside of the toy. Once the bluetooth connection with the computer was established, all the sounds from the Scratch program appeared to be coming from the toy!

It was really quick to start a simple program with the ‘speech to text’ and ‘text to speech’ blocks (using the microphone on the computer). I programmed the bunny to ask “how are you feeling today?” and then used the AI based ‘analyze sentiment’ blocks to have the toy come up with an appropriate answer.

With simple inputs/outputs of the Micro:bit, I could add animations on the LED array, include an agitated response when the bunny was picked up and code some fun surprises when the buttons of the Micro:bit were pressed. I also added a sprite and a background to match the theme of the project.

Here’s a video of the bunny going through the entire program! There are so many more possibilities to add extra switches, tweak the questions and answers, make the code more complex and re-purpose the elements inside the toy.

It’s super fun to share the creation with others in our Maker Ed Community Studio and continue to tweak the program. We’re hoping to collaborate more with Stefania and the Cognimates team to develop more connections between tinkering activities and the new technologies of programming, AI and machine learning!