Week 10: Interactive Environments

Day 1
Slides
Video of our session

Niki is back with the infamous (but powerful?) ESP32! We experimented with our boards to send out messages through OSC and wifi. (Your TA didn’t have the right microcontroller board, so she literally missed out the whole lab time…) Finally, our day ended with our minds blown away by Kaii’s mesmerizing butterfly work!

Day 2
Slides
Video of our session

The second day of interactive environments brought the ESP32 and touch sensing to another level that we can send messages to Mad mapper through OSC! Niki also gave an introduction proximity sensors and MQTT.

Day 3
The last day of our class passed by in a blink listening to everyone’s thoughts and concepts about their final project, it was very inspiring to be able to hear all of your ideas😎.

And…this brings to an end of our final class recap….
It had been such an enjoyable journey to learn with you all! Bon courage on everyone’s final projects!💛

Week 9: Projection Mapping with Madmapper

Day 1
Class resource
Video of our session

Finally getting started with the long anticipated projection mapping! Taurin gave an introduction about projection mapping’s history starting from Disney in the 1960s to the breathtaking works in modern days. Then after an hour of exploration in Madmapper, we all got to experiment with something that reacts to audio inputs!

Day 2
Class resource same as day 1
Video of our session

The second day of projection mapping class kicked off with everyone sharing their final projects ideas (looking back, everyone’s brain must have been storming over the past few weeks!)
Taurin continued the class with a thorough walk through of different types of projectors available on the market. (Watch out for Lumens! Contrast Ratio! and Throw!) And we learned more about mad mapper, even controlling it through TouchOSC!

Day 3
Class resource same as day 1
Video of our session

Workshop day! Our last day of projection mapping was packed with a bunch of inspirational projects! (I personally love the music video of Emmy Curl, so delicate🤩) and I hope everyone had fun creating their own projection mapping mini projects during the class.
Can’t believe this is the ninth week, next post will be our final class recap…

Week 8: Physical Computing with the ESP32 board!

Alrighty! After learning about Node.js, we’re back to the second week of physical computing!

Day 1
Slides
Video of our session
Github repo

The first lab in pcomp2 is to get to know how a joystick works! Isn’t it quite mind-blowing to find out that a joystick is actually a combination of two potentiometer?! We got the FastLED library in our Arduino IDE and connected our Uno board with the joystick and the WS2812 LED strip, then experimented with some light art with them!

Day 2
Slides
Video of our session
Driver for ESP32 board

Finally we’re on to the powerful ESP32 board. Unfortunately your one of your TAs wasn’t able to get her board running, so she didn’t really have the chance to grasp how powerful ESP32 is…😅 We were able to turn our board into the output/input device and control the LED strip with TouchOSC!

Popular (commercial) software packages that use OSC to communicate with other devices and software
1. Ableton Live
(& article about using OSC for a reactive audio-visual performance set up)
2. Max
3. TouchDesigner

Extra resources provided by the instructor Michael:
1. Bob Jarvis’ art work using OSC and the making of it
2. More in-depth video about OSC, also by Bob Jarvis

Day 3
Slides
Video of our session

Workshop day! We were divided into groups and experimented with how to hook all of our boards together through wifi (AMAZINGGGGG!) Imagine all of the possible works we can create with this tiny micro controller and our understanding of the communication between OSC, ESP32, and our browser! Boom! More exciting news, next week we’re jumping into projection mapping!

Week 7: Node, what is it good for?

On the heels of our first week of physical computing, we dove into the vast and mysterious world of Node…

Day 1
Slides
Video of our session

After reviewing some cool projects from PComp 1, we began our journey to Nodeland. Stephanie explained why Node is useful, helped us get it installed and running, introduced us to NPM, showed us how to connect P5 to Node, and explained what APIs are.

Day 2
Slides
Video of our session

Serial ports! Sockets and the BART API! We talked about Johnny 5 (Still alive! (and Ally Sheedy’s second best film)), and spent some time on micro projects.

Day 3
Slides
Video of our session

Workshop day! Graduation day!

We got our jamboard on to review concepts from the previous week (which led to an amusing moment between your TAs), learned about deploying a Node app, and dove into our micro projects.

Week 6: Physical Computing with Arduino

Day 1
Slides
Video of our session
Github Repo for class examples

The class started with the introduction of basic Arduino components, how does a micro controller work? After the basics, we soon got into soldering our own Arduino boards and accelerometer! Did everyone succeed without burning anything? Later in the class, Stephanie introduced diodes, simple/series/parallel circuits, and the PWM.

Extra Materials:
1. How to use a Multimeter
2. Voltage, Current, and Ohm’s Law

Day 2
Slides
Video of our session
Github Repo for class examples

The second class of physical computing, we spent a moment to refresh everyone’s memory from Tuesday on a google Jam board, then everyone was sent to a breakout room to tinker their own simple/series/parallel circuit on TinkerCad! From the single-colored red LED to the tri-color LED, then the buttons, did everyone’s circuit worked?

Homework from this class:
1 – complete the button exercise and share per above; go further if you feel comfortable and have time to do so!
2 – download Processing (https://processing.org/)

Day 3
Slides
Video of our session

Workshop Day started with more circuit exercises! We all made a capacitive sensor with conducive materials we’ve found in our homes, beer cans, bottle caps, aluminum foil, or even a spared metal material! Then the exercises became more complicate such as connecting with Processing and installing a potentiometer.

Homework from this class:
optionally if you have time and energy:
Continue working on adapting/expanding what you started on Saturday. Some suggestions mentioned were:

  • continuing with your conductive materials
  • adding more leds or more sensors
  • exploring new components
  • changing your Processing sketches
  • trying the piezo labs (there’s guidance for this in the slides!!!)
  • etc.

Journey through Goaland

The workshop theme for 3D week is to build a scene in aframe that has a little twist.

My initial idea was to make a dog that poops coins when you click on it. I learnt that you cannot put onclick event handlers directly on 3D objects on aframe the way you would with normal HTML DOM objects.

Since there are no easy ways to interact with objects in aframe through javascript, I decided to go a different route.

I really like the environment presets in aframe and their extensive customization options. I decided to turn my favorite one, the psychedelic landscape of goaland, into a little audio environment.

I also managed to add in a little twist. As you can see in the video, it turns out that you are not taking a trip in the car as you are originally lead to believe , but are actually viewing the scene from the eyes of a dog that is following a tiny toy car, surprise !

As a side note, even though this is not an intended twist, I like how the car is able to pass through the mushrooms, which gives goaland a much appreciated dreamlike quality.

Since this is an audio environment, being able to control the audio is important. One thing I had trouble with was making the audio pause and resume in aframe.

When I used the pause function in aframe, the sound did not resume at the place where it was previously stopped.

Marc was able to reproduce this issue and we figured that I may have discovered a bug in the library. I ended up just using HTML5 to handle the audio, which worked out well because it comes with a nice set of built in player controls.

You can visit https://tabris-journey.glitch.me/ to take a musical journey through Goaland.

The project repository is available at https://glitch.com/~tabris-journey.

A-Frame Workshop

aka why must a thousand different ways to link everything exist and why is it all so tedious to figure out these these ways

For my final project, I’m going to be working on making 3d objects and showing them in a browser-based environment. I’ve been learning Cinema 4D for a little bit and was excited to figure out a workflow for importing these models using A-Frame. I’d worked with gltf files in the past and recently discovered there’s a beta plugin that will export a Cinema 4D scene into gltf.

For this workshop, I thought it would be fun to put this clamshell into the browser, and that it would open and the pearl would float on click (please ignore the crunchiness of the gif, it got compressed weirdly in photoshop). model

Two problems came up with this method:

1) I’m using a 3rd party renderer with Cinema 4D, so those beautiful materials are lost in the export, as they aren’t the standard material settings. Ok, no big deal, I’ll just change them back to standard later, I think.

2) The second is that animations are exported in a not straightforward way-this is what comes up in the gltf tester that I usually drop my files in to view:

Screen Shot 2020-05-02 at 3.32.26 PM

Looks like the top shell didn’t rotate at all but the pearl animated its movement?

So long story short, it looks like it’s going to be painful to figure out this workflow, but I’m invested, since it’s my final project!

Week 5: 3D with A-Frame

Day 1
Slides
Video of our session
Github Repo for class examples

We saw some cool webVR examples made with A-Frame, and learned the basics of building entities in the 3D environment. The 3D class kicked off with us creating funky shiny boxes in the web browser!

Day 2
Video of our session
Github Repo for class examples

How to import an obj. file? The class started with us importing a flying car model into the environment, and decorating it into a Tiger King themed vehicle! Then, we learned how to create a magic eight ball with the help of Blender. Did everyone succeed in placing the shell into the 3D environment? We’ll continue to make the transparent cover on Saturday!

Day 3
Video of our session

Workshop Day! We picked up from the Magic Eight Ball example on Thursday, and continued to make it shiny! After an hour coding along with Marc, we split off and spent an hour creating our own project with a twist at the end.

CloudChime

My mini-project for the Web Audio week is called CloudChime, a page which makes music out of hand-placed clouds over a satellite image, using a Leap Motion Controller for gesture recognition.

The ability to create an appealing melody without needing to play an instrument allows people without musical training to engage in the experience. The act of ‘dropping’ clouds as if they were notes on a staff breaks down the barriers of dry musical notation and helps provide an intuition for pitch and rhythm. A cursor on the screen corresponds to the hand’s position in the horizontal plane, and when the hand is opened, a cloud is spawned.

One can access tools for gesture recognition from the Leap Motion Javascript API website. In this case, the position of the palm above the camera is provided out of the box, and by processing the graspStrength attribute, one can determine whether the hand is open, closed or transitioning between these states. One of the challenges was working out a technique for opening the hand to allow it to reliably detect such motion. Another challenge was the skipping between estimated palm positions in these transitional states. I could apply a low pass filter to the cursor position to mitigate this effect.

The position of the dropped cloud was mapped to 2 variables: on the horizontal axis, the relative position in a single, looping, measure, and on the vertical axis, one of four ascending wind chime pitches. Practically, no two patterns are alike, which allows for continual experimentation and discovery.