Canon EOS 5D Mark II
ƒ/2
100 mm
1/640
4000

Imagine having a searchable video recording of everything that happens in every room of your house.

MIT cognitive scientist Deb Roy started the recording when his first child arrived home, and has kept recording for 5.5 years now.

He now has a 200TB archive, with a 70-million-word transcript, and can do a virtual 3D fly through any moment across all rooms. He can view all developmental milestones, from baby’s first steps to the mastery of any spoken word. He can roll back and see what verbal and physical interactions preceded the acquisition of language. He can draw movement traces through the house of all caregivers and their proximity to his son. From these traces, he can see the social hotspots of interaction. See examples below.

“You are looking at a piece of what is by far the largest home video collection ever made.”

His TED Talk went online today. It is one of my favorites.

21 responses to “Lifestreams”

  1. He can search for every utterance of the word “water” and map it to the location in the house:

    Water Wordscape 4

    Here is the resulting Wordscape, a landscape of where “water” was spoken, with peaks in the kitchen (left), living room (right), and a sharp peak near the bathroom sink (rear right):
    Water Wordscape

    Then he can perform retrospective analyses. For example, here you see a graph of the complexity of sentences uttered by caregivers that include the word “water” in them.

    Water

    The vertical dotted line shows the moment where baby first said the word water. In a feedback loop they were unaware of, all caregivers used simpler and simpler sentences with the word “water”, dropping to two word utterances like “want water?” and then single-word practice at the moment of mastery. Then they ramp back up to complex, normal usage. It reminds me of the “baby talk” parents naturally perform in the early years, a form of phoneme practice where baby’s attention goes to the speaker’s lips when it’s interesting and new, another feedback loop leading to repetition by the speaker.

  2. Simply superb, the photograph and the photoclips from the talk! Great explanations too.

  3. Fascinating use of technology to further developmental psychology. Thanks for posting

  4. Wonder if one of the video streams was patched…they did have a second baby…
    🙂

  5. The magic part is "searchable". I gather that took a looot of manhours.

    Great to have analysis.

  6. It also makes me think how much information we pass to our babies subconsciously…naturally…without any forceful rationalization… how much we know without knowing…very interesting talk in terms of research as well:) will revisit it…

  7. great talk.
    we’re going to have to be responsible in the way all that data is going to be used.

  8. I had the chance to talk to him at the MIT dinner and ask him a question… if anybody had yet or before filtered or contrasted his work with others’ works known so far on children learning like Piaget, for example. He told me that yet nobody did contrast his work with these other thinkers but that yet, he had his favorites before doing his experiment.

    So this brings the question: how much of this is data or already driven information? He had already a bunch of theories of how should things unfold when a child is learning a language, the fact that he recorded it doesn’t add much to the already existent theories in learning.

    In my impression, I would have liked it a lot more had he mentioned that he was working on this with a lot of a-prioris that he wanted to guide his work with (whether for or against).

    The very concept of DATA gathering is confusing, when you were already biased by some previous knowledge when you did the set up of your experiment.

    The talk opened a lot of questions which were unasnwered rather than bringing me answers to questions I had before…

    That’s my impression, feedback welcome.

  9. you mean… he already knew "water" would spike in the bathroom?!

    =)

  10. Something like that, but far beyond that… He sets the spaces for certain things to unfold the way they were "expected". My only question is regarding the concept of DATA against INFORMATION gathering. His work is very important, I may just want to add that he didn’t gather data (raw), he gathered information (cooked)… Don’t blind yourself in the DATA GATHERING and VISUALIZATION fashion which seems so overwhemling at TED… I am into deeper understading… and deeper understanding was done way before this talk, and should be acknowleged, even more so if it was foundational for the experiment….

  11. what was that that the environment is learning from him and that hasn’t been observed ’till now … nobody has heard of relationship dynamics?
    very cool visuals.. that’s it for me..i guess didn’t understand it…except for –i now have an answer to who uses the feedback form things like yahoo articles..and similar.. (if i understood even a bit of that talk)

  12. oh, and picked up something about society behaving as a wave cased by a particle 😉
    :p
    will have to watch it again ..later

  13. @jurvetson: Scatological! 🙂

    @alieness: Are you saying that the family behavior could have changed because they knew video cameras were installed…that’s a whole other sociological discussion…read "on Keyhole morals" by AG Gardiner…behavior of humans changes when you know there’s someone behind the keyhole watching you. Or are you saying that he knew how the data would look, so he positioned his cameras in a particular way, and hehaved accordingly. The latter would be hard…it takes n months to learn the word water no matter how many cameras are pointed at you.

  14. Hours and hours of video in Deb Roy’s house seems like raw data to me. There are many different topics which research could facilitate, the simple answers “where is word “water” used” is obviously not a purpose of the research itself. One application is the software which can analyze, slice and dice the information from the video (heard about different approaches in this regard), he also has demonstrated how it can be used in tracing broader audiences reactions to certain political events, marketing campaigns etc.
    What about privacy issues of all parties involved and legal issues (in this case solved very easily by asking permission from 5 years old – like his sense of humor) could become very touchy. Who is watching us when and why and how this data is going to be used?
    As for my own company, video is one of our verticals for a reason, all this video will have to use storage and the software is continually evolving. So I can be happy personally, since global storage software is in my professional area and it is growing rapidly.
    As parent, however, I loved this idea of subconscious feedback loops by which we communicate without being consciously aware… rationalizing them is like asking us how do we breeze, so we stop breezing while trying to rationalize.

  15. @alienness, are you talking about confirmation bias? Or are you suggesting that empirical observations in the past is made more concrete by the video, but the video per se is not adding anything new?

  16. well.. it looks like things started going towards the word "water"….it’s like trying to talk to a foreigner…then things got established …so? (ok, i perhaps have to shape up)
    yeah..ok, how about some other word..and everybody was told a different meaning…i bet there will be still a graph..(that’s cool! )

  17. ..imagine their shorter sentences were related to higher temperatures which happened to make the kid very thirsty..so he decided to be more explicit … bummer :p
    Eppie..what you saying sound correct to me..i’m not sure if a particular development of events is of interest..or just the ability to record it and get some analysis..but anyways–interpretation can be sure wrong(or manipulated)…whatever it might be about… but can be a useful tool as well… i make fun of usage of data, too..despite the fact i don’t know too much about it..but if it looks funny to me, i laugh 😉

    and that just reminded me of a photon~wave’s post

  18. Sorry for my delay, I was out all day.

    As per your questions, it’s exactly what Eppie (plus tull with the confirmational bias) say, what I said (tried to!). What Drona mentions is also related, but secondary to my observation. Ande veneta’s last post before me rings for me too.

    It’s not "data", imo, when you already read the whole work of Piaget and have it as a favorite prior to make the experiment. It’s rather information gathering. You had already sort preconceived the outcome. Did you hear about the Pygmaleon effect? Something like that. I consider this experiment would have been called "data gathering" was it done in a family were the members didn’t have a clue about raising a child. And it would be a further plus, if they didn’t know they are being recorded, or at least, not know the exact purpose of the recording.

    He should have mentioned, imho, that he "inspired" this research in previous studies, and that would have made it more valid to me. Piaget experimented with all many kids and his own family. The fact that Piaget wasn’t able to record in video this sessions and years of studies don’t change the facts.

    "To test his theory, Piaget observed the habits in his own children. He argued infants were engaging in an act of assimilation when they sucked on everything in their reach. He claimed infants transform all objects into an object to be sucked. The children were assimilating the objects to conform to their own mental structures. Piaget then made the assumption whenever one transforms the world to meet individual needs or conceptions; one is, in a way, assimilating it. Piaget also observed his children not only assimilating objects to fit their needs, but also modifying some of their mental structures to meet the demands of the environment. This is the second division of adaption known as accommodation. "
    en.wikipedia.org/wiki/Jean_Piaget

    Piaget in a way, was in a halfway of confirmational bias (but of his own theories) and gathering raw data by pure observation (when he didn’t have a clue of what went on).

    The addition of this kind of technologies to the study of whatever it might be of use for, is nor negated, neither belittled by the fact that it’s an addition or a continuation of previous research. On the contrary, to me, puts it all in proper and more valuable perspective. Because I think that many of us thought, after the awe of the impressive amount and filtering through visuals and audio of the "data gathering"… How this can be of use? How we can make sense of it? This [perspective] would have been a palatable answer to give, imo.

  19. I ended up reviewing the translation into Spanish of this talk… kind of a curse or what? 😉

  20. Marvelous image, nicely composed and shot. Very well done.

Leave a Reply to angry shock Cancel reply

Your email address will not be published. Required fields are marked *