Hey there. It's me, Daniel. You can find more about me here if you'd like.
Week 1 | Project Kickoff
Reading Reflections:
Molly Steenson
This talk helped illuminate for me the origins of AI/Machine learning from a design/historical perspective. Previously, my experience with the development of machine learning had primarily been from a technical perspective or through the lens of science fiction. It helped me understand that even if machine learning applications did not become prominent and widely used until recently, the underlying narratives and hopes for application had remained relatively the same since the early 1960s. Seeing early versions of conversational user interfaces like ELIZA compared to modern VUIs like Siri and Alexa feels like the adaptation of early ideas and narratives for a profit focused context.
Anatomy of AI
This article served as a sobering reminder of the costs of technology and progress in a capitalist system. I had been aware of the aspects data collection, mechanical turks, and the physical costs of mining rare earth minerals, however, this article really brought them into full focus in contrast to the image of AI and machine learning that is commonly presented. I was particularly struck by the discussion of the perception of AI as a black box and how that perception vastly simplifies the work and resources that go into it, both from the perspective of the work and resources extracted and involved in its creation and in the workings of the algorithm itself.
Enchanted Objects
This reading was really interesting reading the fantasies and speculations of someone from the recent past. While some of the fantasies that the author illustrates have been becoming a part of our daily live and current reality, ex. in Amazon Echo and Alexa or AI driven facial recognition systems, it seems that much of the hopeful, collaborative visions he had have not yet come to pass. I think that AI and machine learning are still far more privatized and controlled to reach the point of the author’s AR tagging system for identifying objects/people for example. I also found the section on creating technology to silence or combat technological advances very insightful. I felt that society’s reaction to each technological advance has continued along the path of simultaneously embracing the new but also thirsting for a simpler less noisy lifestyle and environment.
Project Preparation:
Create a diagram or a sketch of how you could interact with data of a space in a shared work environment like the TCS hall.
Pose a research question you want to explore for this project. It could be more than one, we will discuss it in the studio. Find at least two examples or case studies related to your research question.
Research Q1: Can lighting our environment responsively augment or enhance our daily experiences and activities? Alternatively, can we tie a human input to a characteristic of their environment? Ex. Adjusting light level + temperature with conversation volume, varying lighting arrangements with context, lighting responding to activity (movement, sound), lighting conditions dependent on number of people present.
Research Q1B: Can we make the data collection system transparent as to what's being collected in real time? Ex. displaying each metric in a visual, augmenting the environment or projecting data in real time.
Research Q2: How can we visualize an individual's/group's impact on a space. Ex. A person's sonic footprint, motion, vibrations, etc, Tracking nth order statistics such as occupancy counters, activity analysis, etc.
Research Q3: Can we help people find spaces to occupy by visualizing/building profiles on different rooms based on their characteristics? Ex. Students who are looking for a space with specific characteristics to work in (quiet, strong wifi, relatively cold), Administrators looking for irregularities in building/room conditions.
Week 2
Reading Reflections:
IOT Data in the Home:
Going into this study, I was skeptical as to the methods and low sample sizes that the researches used as I was worried they might be statistically insignificant or non-representative of the true experience. However, I found that the anecdotes and discussion that this study provided were far more interesting than I had imagined.
One thing that really caught my attention was the notion of Data as Breadcrumbs. I found as one participant recalled searching through his Google Home history for recipes he had used in the past, I have already been using very similar methods with my non-iot devices and work-flows. This example really helped me realize how data exploration and the navigation can be manipulated and utilized in unintended and "hacky" ways.
Additionally, the example where Lucy defers to the data on how long she spent in the elevator compared to her perception was really enlightening for me. It was really interesting how the data forced her to revaluate how she felt about the elevator and changed her mindset on how she viewed that experience. That calls to mind the discussion of two different kinds of "truth," the factual truth of how much time she actually spent in the elevator and the "truth" of the way she felt about that experience. Ultimately, it's super interesting how one kind of truth defers to the other.
Finally, the section on Data Epics was super interesting to me because it helped me understand that we see can start to see our data as a way we tell stories about ourselves and who we are. Knowing that that data is being collected about us can in turn make us more conscious of what we are doing in that moment so we change our behavior to record the data to tell the story that we want. In some ways, I see this as similar to the mindset of some who might be religious. With the attitude that God is always watching, some might change their behavior to reflect what they might perceive God to approve of. However, in this case, the data collection, story, and observation is much more tangible. Because the record of our behavior is stored as cold hard evidence, there is no way to delude the data, although our interpretation is still up for grabs.
Podcast Takeaway:
One thing I connected with was the Yanni's initial anecdotes about dual identity. As someone who has a foot in design and a foot in computer science and a foot in a bunch of other things, I also often find myself casted as the coding guy in a design project or the design guy in a coding project. The way he talked about those experiences felt cathartic in a way.
Another thing I found interesting was how he emphasizes the importance of place and context, both in how data is collected and also how it is received. Reflecting on my own thought process, I found that I had not explored the latter as fully as I wanted in my previous work. Instead of creating non-interactive graphics or animations, I would like to create interactive interfaces that are made for a particular setting. I feel like taking this extra step as opposed to a universal app could elevate the experience.
Additionally, the way Yanni describes "all data as being local" really helped me visualize what it means for all kins of bias and error to be introduced into a dataset. It also helped me gain a new perspective for how the research paper mentioned that data is not smooth. It's organic and prone to flaws, etc.
Finally, I found Yanni's stressing of the importance of talking to someone with direct contact with the data to be an important reminder. I had previously had a similar experience of having data being cast in a different perspective after my own set of interviews. Even though I had felt safe only working with the data, those interviews helped me contextualize and course correct in a way that ensured I stayed on track with the local reality.
Choose a similar system or a case study that inspired you. Analyze how it works, the system components, the interactions, document it in your blog, use a diagram/sketch.
One system that I feel is a rough parallel to the mites system in the home is the Google Nest ecosystem. It's a collection of various IOT devices that collect and monitor a variety of data related to home security, temperature, etc.
Google Nest Products:
Cam (indoor)
Cam (Outdoor)
Cam (with Floodlight)
Doorbell
Hub
Audio
Chromecast
Thermostat (multiple variations)
Wifi
Protect
Temperature Sensor
For smart home devices like the thermostat, Nest collects data about resident usage and uses machine learning to optimize its behavior. All data collected is centralized in the hub where Nest can not only interact with nest devices but also connect to Google's external features and and integrations.
Storyboard example of usage:
Concept Development:
Week 3
Reading Reflections:
Designing the Behavior of Interactive Objects
Data Imaginaries
Each of the initial examples from the data imaginaries video left an impression on me, allowing me to see a different way data could be represented and experienced. The first project,
Project Development:
Visiting TCS
Personality Traits
Ambient
Playful
Alert
Updated Storyboard
Case Study
Prototype
Reading Reflections:
Giorgia Lupi
I found Giorgia's talk really interesting for the way she visualizes and collects data. The first thing I found really striking about Giorgia's work was the way she chose to visualize data across multiple projects. Building on the base of familiar charts and graphs, she chooses to abstract the data into aesthetic symbols and shapes, with a key to interpret them, and often hand draws the final plot. This approach was really interesting to me for two reasons. Firstly, it was really interesting to see how the aesthetic representation of the data was beautiful and fun to look at, but also provided meaning at a macro and micro scale once the viewer chooses to reference the key. I also found the process of representing data in an analog way really interesting. Aside from the aesthetic, it seemed like a really interesting process to draw out each data point by hand, which forces you to interact with the data in a much different way than I typically do when I work digitally.
The other thing I found really interesting was how Giorgia actually creates a lot of the data that she works with from historical accounts and second hand sources. Typically, I have always tried to to find existing sources of data through APIs or databases. Creating a dataset from scratch seems really time consuming yet might be helpful in helping the person working with it understand its 'local' characteristics more deeply.
Project Development:
Isha's Talk Values
Reflecting on Isha's talk, the values that are most important to this project are transparency, awareness, and expanding perception. Due to the invisible nature of the electronic world and potential technical ignorance, most people are able to perceive the electronic activity around them and their affects on it. I'd like for this project to not only make the invisible tangible, but also expand their awareness of how the electronics around them operate and how it affects them and vice versa.
Storyboard Interactions
Environment
The ideal environment for this project would be a smaller enclosed room where people go to study either individually or in small groups. The readings of Wifi strength, electromagnetic noise, and so on may vary based on location very rapidly. To have the most accurate representation of the space and to fit a more ambient peaceful mood, I think it would be optimal to limit the scope to these spaces.
Week 4
Concept Development:
Silicone Techniques
Andrew D. Marchese, Robert K. Katzschmann, and Daniela Rus. A Recipe for Soft Fluidic Elastomer Robots. Soft Robotics, 2(1):7–25, March 2015. doi:10.1089/soro.2014.0022.
Concept Analysis:
Things: Mites Sensors, Tactile Panels, Key (Attached?)
Human Actions:, Looking at/Touching the panels, Affecting the data (personal devices, electronic activity)
Feedback: Panel lights, Panel texture
Connectors: Mites -> Arduino -> Panels
Data Channels: Mites Sensor Data: electromagnetic noise, bluetooth connections, wifi strength, and magnetic field strength.
Themes: Each sensor contributes to painting a picture of the electronic activity that is usually invisible around us.
Criteria: Can visitors make the connection between changes in the data, changes in the tactile representation, and potentially connecting the data to the real world?
Can users grab, feel and move “the important stuff”?
Yes.
Is there rapid feedback during interaction?
Yes.
Can users proceed with small, experimental steps?
Yep.
Can users experience the interaction straight away, from the start?
Yeah.
Do people and objects meet and invite into interaction?
Mhmm.
How can the human body relate with the space?
Through the tactile and visual components of the display.
Can you create a meaningful place with atmosphere?
Does shifting stuff (or your own body) around have meaning?
Interacting with electronic devices in various ways might have different impacts on different sensors. For example, switching on or off the bluetooth sensing function of a device would be reflected in the height, oscillation speed, or other properties of the tile.
Can everybody see and follow what‘s happening?
Yeah
Can you use your whole body?
Nope
Can users take ownership of space by physically moving there?
No
Can users be proud of skilled body movement? Can they develop skill over time?
No
Can you communicate through your body movements while doing what you do?
No
Are actions publicly available?
Yes
Does the physical set-up lead users to collaborate by subtly constraining their behavior?
Yes
Is there a physical focus that draws the group together?
Yes
Can all users get their hands on the central objects of interest?
Yes
Can you hand over control anytime, and fluidly share an activity?
Yes
Does the representation build on users’ experience and connect with their skills?
Not necessarily
What is the entry threshold for interaction?
Touch, sight, perhaps owning a device of some kind
Are representations legible, meaningful, and expressive? Are they of lasting relevance?
Yes
Are physical and digital representations of similar strength?
No
Can users think or talk with/ through objects, using them as props to act with?
Yes
Does the representation give discussions a focus and provide a record (trace)?
Yes -> No
How easy is it to understand the relations between action and effects?
There is a layer of abstraction between the action and effects in that the precision of numerical values is lost. However, change in the data in terms of increasing, decreasing, etc should remain apparent.
Are there powerful representations that transform the problem?
Yes
Week 5
Project Feedback:
Class Feedback:
- Where is it placed in TCS? Why is it placed at that location?
- Like how you approached this project with unseen/unknown data types.
- Like how you made it an organic/tactile form.
- How is this site specific? This interaction could be placed in any building that can pick up on electronic data.
- If the size of the physical artifact is too small is it possible ppl wouldn't notice it?
- What if you explored the different implications of size in the experience.
- Do you image the tile being much larger in the end?
- What sort of specific actions can people do?
- Interested to see personality through interaction.
- Is the measure how "organic" it ends up being? (Criteria)
- I love the soft robotics.
- How do people interpret the tactile pattern.
- Where are these being displayed? How many in one area?
- How many states can you represent in one board configuration?
- Adding personalities to the creature?
- Tabletop Tactile Device?
- How would people read the physicalizations?
- What if they don't know what they are seeing?
- How does that interaction match their mental model?
- I like the idea of having different displays.
- You might need to explore how users will know what each one means.
- Is it important for people to know the source of the data?
- How long do you imagine users to engage w/ this?
- Where do you visualize this tactile display? (wall, desk, location?)
- I really like the idea of connecting to a tactile display + working w texture based data/
- How can you make this visual more understandable for someone in the space? How can you make it more straightforward how data is affecting the tactile display?
- Your system diagram gives a nice sense of your system + how you plan to move forward.
Feedback Daniel:
- Justify the reasoning for including each sensor type
- Be clear on the interactions and what a user might get from each display
- Maybe explore expanding the size of the organism to fit an entire wall with panels
Feedback Elizabeth:
- How can you make the "organism" fit into the context of both the environment itself, but also the type of data it's displaying.
- Make lots of smaller scale prototypes and sketches instead of focusing on a few more refined prototypes to speed up development.
Feedback Reflection:
Much of feedback revolves around refining the focus and narrative behind the data visualization for the "electronic organism." I plan to take up Daniel and Elizabeth's advice by refining storyboards and interactions for the organism as a whole and each of the sensors. I'd plan to speed up the process of research and prototyping.
Concept Development:
Research
Magnetometer: A magnetometer measures the strength of magnetic fields in the area. While it measures the environment broadly, the EMF is affected by electric currents and devices as well. Depending on the device and the frequency it emits as well as the distance from the source, the strength of the signal will increase or decrease. Environmental and artificial signals tend to have different frequencies/energy levels. Magnetometer measurements can stand as a proxy for the general density/strength of the devices/electrical activity in the area with additional environmental noise.
Wifi: Wifi signal strength of the Mites Device. This metric could be used as a proxy to estimate the signal strength for devices in the room/immediate area.
Bluetooth: # of bluetooth devices in range to connect with. While Mites does not give specific details on how this data is collected, I assume it works like a computer/phone/device would. If a device is powered on with bluetooth enabled, it is visible with varying levels of connection.
Microphone: Captures sound from its surroundings. This could potentially be used to isolate white noise from electronic sources in the environment. Examples include the mains hum (a low frequency hum emitted by devices plugged into the grid), AC, Fridge noise, etc.
Much of the feedback I received during the feedback session seemed to question the relationship between using the microphone for white noise and the overall concept of the invisible electronic world. While some of this confusion might be alleviated with better presentation of the idea, I decided to cut the microphone data stream to simplify the concept and focus on refining Magnetometer, Wifi, and Bluetooth instead.
What are the precedents informing your design?
How does your system make connection to the data being collected.
How would the users understand how to interact with your system?
Remember that the sensors collect data in space. Not about each occupant.
What is the value that you’re basing your concept on? Data awareness? Transparency? ….
How are you going to build your concept? Remember you have to show a full demo or partial
demo+movie for your final deliverable.