HICSS Workshop – Collecting Data - SoMe Lab

cakeexoticInternet και Εφαρμογές Web

13 Δεκ 2013 (πριν από 3 χρόνια και 10 μήνες)

140 εμφανίσεις

@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Data Collection
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
{"text":"JP Morgan-Chase Mic-Checked at Princeton University #OWS #princeton
#jpmorgan #occupy http:\/\/t.co\/fwsvTCar","entities":{"hashtags":
[{"text":"OWS","indices":[52,56]},{"text":"princeton","indices":[57,67]},
{"text":"jpmorgan","indices":[68,77]},{"text":"occupy","indices":[78,85]}],"urls":
[{"indices":[86,106],"url":"http:\/\/t.co\/fwsvTCar","display_url":"lnkd.in\/
ADVeBa","expanded_url":"http:\/\/lnkd.in\/ADVeBa"}],"user_mentions":
[]},"contributors":null,"place":null,"id_str":"147103601754517505","coordinates":nu
ll,"source":"\u003Ca href=\"http:\/\/
www.linkedin.com
\/\" rel=\"nofollow
\"\u003ELinkedIn\u003C\/a\u003E","retweet_count":
0,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"favorited":false,"possib
ly_sensitive":false,"geo":null,"in_reply_to_screen_name":null,"truncated":false,"in
_reply_to_status_id_str":null,"user":
{"default_profile_image":false,"show_all_inline_media":false,"profile_image_url_htt
ps":"https:\/\/si0.twimg.com\/profile_images\/1193557876\/
Brian_BW_filter_normal.jpg","lang":"en","notifications":null,"profile_use_backgroun
d_image":true,"followers_count":513,"profile_image_url":"http:\/\/a0.twimg.com\/
profile_images\/1193557876\/
Brian_BW_filter_normal.jpg","default_profile":false,"time_zone":"Pacific Time (US &
Canada)","profile_text_color":"333333","url":"http:\/\/
imonad.wordpress.com","id_str":"42944839","contributors_enabled":false,"profile_bac
kground_image_url":"http:\/\/a0.twimg.com\/profile_background_images\/80107544\/
1AhHaSurfing_243x178_Name.JPG","favourites_count":17,"description":"Ecology-of-Mind
* Human Singularity * Autopoiesis * Innate Gift-Centric Humanity * Synergy *
StartUps * Art Beauty Meaning &
Purpose","following":null,"profile_link_color":"0084B4","location":"California,
USA","is_translator":false,"profile_background_image_url_https":"https:\/\/
si0.twimg.com\/profile_background_images\/80107544\/
1AhHaSurfing_243x178_Name.JPG","friends_count":
1361,"profile_background_color":"9AE4E8","listed_count":
13,"profile_background_tile":true,"protected":false,"statuses_count":
6008,"verified":false,"profile_sidebar_fill_color":"DDFFCC","name":"J. Brian
Hennessy","follow_request_sent":null,"geo_enabled":false,"id":
42944839,"utc_offset":-28800,"created_at":"Wed May 27 18:32:31 +0000
2009","profile_sidebar_border_color":"BDDCAD","screen_name":"thejohnbrian"},"retwee
ted":false,"id":147103601754517505,"in_reply_to_user_id_str":null,"created_at":"Thu
Dec 15 00:00:06 +0000 2011"}
{"text":"RT @KeithOlbermann: #ShowPlug 5: Obama's path to reelection in 2012 goes
straight through the premises of #Occupy. Guest @RBReich on his ...","entities":
{"hashtags":[{"text":"ShowPlug","indices":[20,29]},{"text":"Occupy","indices":
[106,113]}],"urls":[],"user_mentions":[{"indices":
[3,18],"id_str":"130617778","name":"Keith Olbermann","id":
130617778,"screen_name":"KeithOlbermann"},{"indices":
[121,129],"id_str":"148529707","name":"Robert Reich","id":
148529707,"screen_name":"RBReich"}]},"contributors":null,"place":null,"id_str":"147
103603344150528","coordinates":null,"retweeted_status":{"text":"#ShowPlug 5:
Obama's path to reelection in 2012 goes straight through the premises of #Occupy.
Guest @RBReich on his powerful video","entities":{"hashtags":
[{"text":"ShowPlug","indices":[0,9]},{"text":"Occupy","indices":[86,93]}],"urls":
[],"user_mentions":[{"indices":[101,109],"id_str":"148529707","name":"Robert
Reich","id":
148529707,"screen_name":"RBReich"}]},"contributors":null,"place":null,"id_str":"147
103311621922816","coordinates":null,"source":"web","retweet_count":
5,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"favorited":false,"geo":n
ull,"in_reply_to_screen_name":null,"truncated":false,"in_reply_to_status_id_str":nu
ll,"user":
{"default_profile_image":false,"default_profile":false,"show_all_inline_media":fals
e,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/1616674758\/
n8dpX_normal.jpg","lang":"en","notifications":null,"profile_use_background_image":t
rue,"followers_count":354663,"profile_image_url":"http:\/\/a1.twimg.com\/
profile_images\/1616674758\/n8dpX_normal.jpg","time_zone":"Eastern Time (US &
Canada)","profile_text_color":"333333","url":"http:\/\/current.com\/
countdown","id_str":"130617778","contributors_enabled":false,"profile_background_im
age_url":"http:\/\/a3.twimg.com\/profile_background_images\/196246275\/
KO1971YankeeStadium.jpg","favourites_count":5,"description":"There is Baseball, and
occasionally there is other stuff of note.
","following":null,"profile_link_color":"0084B4","location":"New
York","is_translator":false,"profile_background_image_url_https":"https:\/\/
si0.twimg.com\/profile_background_images\/196246275\/
KO1971YankeeStadium.jpg","friends_count":
478,"profile_background_color":"C0DEED","listed_count":
14583,"profile_background_tile":false,"protected":false,"statuses_count":
21650,"verified":true,"profile_sidebar_fill_color":"DDEEF6","name":"Keith
Olbermann","follow_request_sent":null,"geo_enabled":false,"id":
130617778,"utc_offset":-18000,"created_at":"Wed Apr 07 20:43:27 +0000
2010","profile_sidebar_border_color":"C0DEED","screen_name":"KeithOlbermann"},"retw
eeted":false,"id":
147103311621922816,"in_reply_to_user_id_str":null,"created_at":"Wed Dec 14 23:58:57
+0000 2011"},"source":"web","retweet_count":
5,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"favorited":false,"geo":n
ull,"in_reply_to_screen_name":null,"truncated":true,"in_reply_to_status_id_str":nul
l,"user":
{"default_profile_image":false,"default_profile":false,"show_all_inline_media":fals
e,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/1629885754\/
radSpacer_normal.jpg","lang":"en","notifications":null,"profile_use_background_imag
e":true,"followers_count":61,"profile_image_url":"http:\/\/a0.twimg.com\/
profile_images\/1629885754\/
radSpacer_normal.jpg","time_zone":null,"profile_text_color":"333333","url":null,"id
_str":"351316269","contributors_enabled":false,"profile_background_image_url":"http
:\/\/a0.twimg.com\/profile_background_images\/360909480\/
keith0NYCSkyLighgtsUP.jpg","favourites_count":26,"description":"plowing the skyZ
for truth, justice & the gamma
rayZ","following":null,"profile_link_color":"0084B4","location":"#occupyXXXXXX","is
_translator":false,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/
profile_background_images\/360909480\/keith0NYCSkyLighgtsUP.jpg","friends_count":
37,"profile_background_color":"C0DEED","listed_count":
2,"profile_background_tile":true,"protected":false,"statuses_count":
13914,"verified":false,"profile_sidebar_fill_color":"DDEEF6","name":"Teip
Cire","follow_request_sent":null,"geo_enabled":false,"id":
351316269,"utc_offset":null,"created_at":"Tue Aug 09 02:35:07 +0000
2011","profile_sidebar_border_color":"C0DEED","screen_name":"londonriots8"},"retwee
ted":false,"id":147103603344150528,"in_reply_to_user_id_str":null,"created_at":"Thu
Dec 15 00:00:06 +0000 2011"}
{"text":"RT @KeithOlbermann: #ShowPlug 5: Obama's path to reelection in 2012 goes
straight through the premises of #Occupy. Guest @RBReich on his ...","entities":
{"hashtags":[{"text":"ShowPlug","indices":[20,29]},{"text":"Occupy","indices":
[106,113]}],"urls":[],"user_mentions":[{"indices":
[3,18],"id_str":"130617778","name":"Keith Olbermann","id":
130617778,"screen_name":"KeithOlbermann"},{"indices":
[121,129],"id_str":"148529707","name":"Robert Reich","id":
148529707,"screen_name":"RBReich"}]},"contributors":null,"place":null,"id_str":"147
103607894982657","coordinates":null,"retweeted_status":{"text":"#ShowPlug 5:
Obama's path to reelection in 2012 goes straight through the premises of #Occupy.
Guest @RBReich on his powerful video","entities":{"hashtags":
[{"text":"ShowPlug","indices":[0,9]},{"text":"Occupy","indices":[86,93]}],"urls":
[],"user_mentions":[{"indices":[101,109],"id_str":"148529707","name":"Robert
Reich","id":
148529707,"screen_name":"RBReich"}]},"contributors":null,"place":null,"id_str":"147
103311621922816","coordinates":null,"source":"web","retweet_count":
6,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"favorited":false,"geo":n
ull,"in_reply_to_screen_name":null,"truncated":false,"in_reply_to_status_id_str":nu
ll,"user":
{"default_profile_image":false,"show_all_inline_media":false,"profile_image_url_htt
ps":"https:\/\/si0.twimg.com\/profile_images\/1616674758\/
n8dpX_normal.jpg","lang":"en","notifications":null,"profile_use_background_image":t
rue,"followers_count":354663,"profile_image_url":"http:\/\/a1.twimg.com\/
profile_images\/1616674758\/n8dpX_normal.jpg","time_zone":"Eastern Time (US &
Canada)","profile_text_color":"333333","url":"http:\/\/current.com\/
countdown","id_str":"130617778","contributors_enabled":false,"profile_background_im
age_url":"http:\/\/a3.twimg.com\/profile_background_images\/196246275\/
KO1971YankeeStadium.jpg","favourites_count":5,"description":"There is Baseball, and
occasionally there is other stuff of note.
","following":null,"profile_link_color":"0084B4","location":"New
York","is_translator":false,"default_profile":false,"profile_background_image_url_h
ttps":"https:\/\/si0.twimg.com\/profile_background_images\/196246275\/
KO1971YankeeStadium.jpg","friends_count":
478,"profile_background_color":"C0DEED","listed_count":
14583,"profile_background_tile":false,"protected":false,"statuses_count":
21650,"verified":true,"profile_sidebar_fill_color":"DDEEF6","name":"Keith
Olbermann","follow_request_sent":null,"geo_enabled":false,"id":
130617778,"utc_offset":-18000,"created_at":"Wed Apr 07 20:43:27 +0000
2010","profile_sidebar_border_color":"C0DEED","screen_name":"KeithOlbermann"},"retw
eeted":false,"id":
147103311621922816,"in_reply_to_user_id_str":null,"created_at":"Wed Dec 14 23:58:57
+0000 2011"},"source":"web","retweet_count":
6,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"favorited":false,"geo":n
ull,"in_reply_to_screen_name":null,"truncated":true,"in_reply_to_status_id_str":nul
l,"user":
{"default_profile_image":false,"show_all_inline_media":false,"profile_image_url_htt
ps":"https:\/\/si0.twimg.com\/profile_images\/1670512672\/
FreeWorld-1_normal.PNG","lang":"en","notifications":null,"profile_use_background_im
age":true,"followers_count":74,"profile_image_url":"http:\/\/a3.twimg.com\/
profile_images\/1670512672\/
FreeWorld-1_normal.PNG","time_zone":null,"profile_text_color":"333333","url":"http:
\/\/shadowysalmon.wordpress.com
\/","id_str":"405438023","contributors_enabled":false,"profile_background_image_url
":"http:\/\/a0.twimg.com\/images\/themes\/theme1\/bg.png","favourites_count":
375,"description":"Sometimes live RTs of breaking news, reports and opinion about
the global struggle for human rights. Sometimes a blog post. Always a citizen of
the world.","following":null,"profile_link_color":"0084B4","location":"Portland,
Oregon","is_translator":false,"default_profile":true,"profile_background_image_url_
https":"https:\/\/si0.twimg.com\/images\/themes\/theme1\/bg.png","friends_count":
1051,"profile_background_color":"C0DEED","listed_count":
1,"profile_background_tile":false,"protected":false,"statuses_count":
3914,"verified":false,"profile_sidebar_fill_color":"DDEEF6","name":"Shadowy
Salmon","follow_request_sent":null,"geo_enabled":false,"id":
405438023,"utc_offset":null,"created_at":"Sat Nov 05 09:41:24 +0000
2011","profile_sidebar_border_color":"C0DEED","screen_name":"ShadowySalmon"},"retwe
eted":false,"id":
147103607894982657,"in_reply_to_user_id_str":null,"created_at":"Thu Dec 15 00:00:07
+0000 2011"}
{"text":"RT @MissAnonymous74: #FreeSpeech @Twitter Bomb!! #NDAA Summon the
#failwhale! Why is Twitter not trending #NDAA? #RodgerThat copypaste
# ...","entities":{"hashtags":[{"text":"FreeSpeech","indices":[21,32]},
{"text":"NDAA","indices":[49,54]},{"text":"failwhale","indices":[66,76]},
{"text":"NDAA","indices":[106,111]},{"text":"RodgerThat","indices":
[113,124]}],"urls":[],"user_mentions":[{"indices":
[3,19],"id_str":"422031118","name":"Miss Anonymous","id":
422031118,"screen_name":"MissAnonymous74"},{"indices":
[33,41],"id_str":"783214","name":"Twitter","id":
783214,"screen_name":"twitter"}]},"contributors":null,"place":null,"id_str":"147103
605596491776","coordinates":null,"retweeted_status":{"text":"#FreeSpeech @Twitter
Bomb!! #NDAA Summon the #failwhale! Why is Twitter not trending #NDAA? #RodgerThat
copypaste #youneedspeed #OWS #DERP","entities":{"hashtags":
[{"text":"FreeSpeech","indices":[0,11]},{"text":"NDAA","indices":[28,33]},
{"text":"failwhale","indices":[45,55]},{"text":"NDAA","indices":[85,90]},
{"text":"RodgerThat","indices":[92,103]},{"text":"youneedspeed","indices":
[114,127]},{"text":"OWS","indices":[128,132]},{"text":"DERP","indices":
[133,138]}],"urls":[],"user_mentions":[{"indices":
[12,20],"id_str":"783214","name":"Twitter","id":
783214,"screen_name":"twitter"}]},"contributors":null,"place":null,"id_str":"147094
786132803585","coordinates":null,"source":"web","retweet_count":
7,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"favorited":false,"geo":n
ull,"in_reply_to_screen_name":null,"truncated":false,"in_reply_to_status_id_str":nu
ll,"user":
{"default_profile_image":false,"show_all_inline_media":false,"profile_image_url_htt
ps":"https:\/\/si0.twimg.com\/profile_images\/1659202003\/
Anon_Logo_normal.jpg","default_profile":true,"lang":"en","notifications":null,"prof
ile_use_background_image":true,"followers_count":59,"profile_image_url":"http:\/\/
a3.twimg.com\/profile_images\/1659202003\/
Anon_Logo_normal.jpg","time_zone":null,"profile_text_color":"333333","url":"http:\/
\/
www.youtube.com
\/watch?v=czY-dZQsd-
k&feature=feedu","id_str":"422031118","contributors_enabled":false,"profile_backgro
und_image_url":"http:\/\/a0.twimg.com\/images\/themes\/theme1\/
bg.png","favourites_count":7,"description":"[#opblackout will not be silenced. We,
#Anonymous, shall prevail! http:\/\/
www.youtube.com
\/watch?v=czY-dZQsd-k
#(#TouchDown)]","following":null,"profile_link_color":"0084B4","location":"","is_tr
anslator":false,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/
images\/themes\/theme1\/bg.png","friends_count":
175,"profile_background_color":"C0DEED","listed_count":
4,"profile_background_tile":false,"protected":false,"statuses_count":
896,"verified":false,"profile_sidebar_fill_color":"DDEEF6","name":"Miss
Anonymous","follow_request_sent":null,"geo_enabled":false,"id":
422031118,"utc_offset":null,"created_at":"Sat Nov 26 18:22:13 +0000
2011","profile_sidebar_border_color":"C0DEED","screen_name":"MissAnonymous74"},"ret
weeted":false,"id":
147094786132803585,"in_reply_to_user_id_str":null,"created_at":"Wed Dec 14 23:25:04
+0000 2011"},"source":"\u003Ca href=\"http:\/\/twitter.com\/download\/android\"
rel=\"nofollow\"\u003ETwitter for Android\u003C\/a\u003E","retweet_count":
7,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"favorited":false,"geo":n
ull,"in_reply_to_screen_name":null,"truncated":true,"in_reply_to_status_id_str":nul
l,"user":
{"default_profile_image":false,"show_all_inline_media":false,"profile_image_url_htt
ps":"https:\/\/si0.twimg.com\/profile_images\/1077607269\/
mongoose_normal.jpg","default_profile":false,"lang":"en","notifications":null,"prof
ile_use_background_image":true,"followers_count":543,"profile_image_url":"http:\/\/
a2.twimg.com\/profile_images\/1077607269\/
mongoose_normal.jpg","time_zone":null,"profile_text_color":"663B12","url":null,"id_
str":"55949344","contributors_enabled":false,"profile_background_image_url":"http:
\/\/a1.twimg.com\/images\/themes\/theme2\/bg.gif","favourites_count":
366,"description":"occupy anonymous wikileaks topiary manning truth-supporting
progressive 99percenter mom of a great kid who really loves whales
pickuplitter","following":null,"profile_link_color":"1F98C7","location":"Maine","is
_translator":false,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/
images\/themes\/theme2\/bg.gif","friends_count":
773,"profile_background_color":"C6E2EE","listed_count":
31,"profile_background_tile":false,"protected":false,"statuses_count":
25750,"verified":false,"profile_sidebar_fill_color":"DAECF4","name":"pj","follow_re
quest_sent":null,"geo_enabled":false,"id":
55949344,"utc_offset":null,"created_at":"Sat Jul 11 22:34:30 +0000
2009","profile_sidebar_border_color":"C6E2EE","screen_name":"wheeliesmom"},"retweet
ed":false,"id":147103605596491776,"in_reply_to_user_id_str":null,"created_at":"Thu
Dec 15 00:00:07 +0000 2011"}
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW

What data do you want to collect?

From what services?

Point in time vs. longitudinal data
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
APIs
Graph API
Search and Streaming APIs
Query
Query
Data
Data
https://developers.facebook.com/
https://dev.twitter.com/
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
140kit
yourTwapperKeeper
https://140kit.com/
https://https://github.com/jobrieniii/yourTwapperKeeper/
http://discovertext.com/
http://datasift.com/
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
How Do You Choose?

Data Source

Correct API

Understand the limits and matching

Fields you need
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Where Does It Live?
Desktop
Cloud
Closet/Under Desk/
Server Room
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Problems / Goals

Clear data chain of custody

Automation

Preservation of metadata

Scalability

Data export (connect to the best tools)

Low Cost
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Clear Data Chain of
Custody

Where, when, and how was the data
collected.

Why does it appear in our dataset?

Evolving curation of keywords.
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Search API
Streaming API
Firehose
Tuesday, January 8, 13
“Big Data”

Ephemeral data

355 search terms

Oct 19 - Jun 6, 2011 :
64,298,104 tweets

350,000 - 1M new
tweets each day

Twitter, Facebook,
YouTube, Meetup
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
THE
INTERNET
Meetup.Com
Twitter
Facebook
Servers
Servers
Raw
Parse
Expand URLs
Analytics
Expand
Metadata
Processing
DATA COLLECTION
EXTRACT - TRANSFORM - LOAD
ANALYSIS
So
Me
Tools
http://somelab.net/
Data Exploration
Processed
User Jobs
Database Storage
Processed
Data
Analytics
Descriptive
GIS / Geo
Data Export
Keyord
Matching
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Raw
Servers
Servers
THE
INTERNET
Meetup.Com
Twitter
Facebook
Data Collection
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Explore Keyword List
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Processing
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Parse
Expand URLs
Analytics
Expand
Metadata
Processing
Raw
Processed
Database Storage
Processed
Data
Analytics
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Exploring
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Database Storage
Processed
Data
Analytics
Data Exploration
User Jobs
Descriptive
GIS / Geo
Data Export
Keyord
Matching
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Tuesday, January 8, 13
@somelabresearch
So
Me
Lab
Social
 Media
 Lab  @  UW
Database Storage
Processed
Data
Analytics
Data Exploration
User Jobs
Descriptive
GIS / Geo
Data Export
Keyord
Matching
Tuesday, January 8, 13