Data Retrieval Basics

Sergey Chernov
Center for the Study of New Media and Society
www.newmediacenter.ru

Data Expert Evolution

1995 1999 2013

Everyone is a data expert today!

4/11/2013

Sergey Chernov, Information Retrieval Basics

Outline
Open Data Online and Social Media Data Sources Multimedia Archives Hands-on Assignment

Sergey Chernov, Information Retrieval Basics

Outline
Open Data Online and Social Media Data Sources Multimedia Archives Hands-on Assignment

Sergey Chernov, Information Retrieval Basics

Open Data
Open data is the idea that certain data should be

freely available to everyone to use and republish as they wish, without restrictions from copyright, patents or other mechanisms of control.

4/11/2013

Sergey Chernov, Information Retrieval Basics

Open Gov Data

Several national governments have created web sites to

distribute a portion of the data they collect.

4/11/2013

Sergey Chernov, Information Retrieval Basics

Data.gov US Open Gov Data

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://www.facebook.com/pages/OpenGovDataru/139336139422921

Opengovdata.ru

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://../

Open Gov: Open Data

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://www.rsoc.ru

Roskomnadzor (Federal Service for Supervision of Communications, Information Technology and Mass Media)

4/11/2013

Sergey Chernov, Information Retrieval Basics

Roskomnadzor Open Data

4/11/2013

Sergey Chernov, Information Retrieval Basics

Roskomnadzor Open Data (cont)

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://./%D0%BE%D1%82%D0%BA%D1%80%D1%8B%D1%82% D1%8B%D0%B5-%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D0%B5

Russian Ministry of Education and Science

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://www.gks.ru

Rosstat (Russian Federal State Statistics Service)

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://cbsd.gks.ru/

Rosstats Interactive Database

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://www.fedstat.ru/indicators/start.do

Single Cross-Departament Infomation Statistical System ()

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://data.mos.ru/

Moscow Open Data Portal

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://data.mos.ru/datasets/426_muzei/map/

Example: Moscow Museums

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://egip.mka.mos.ru/egip/egip.nsf/va_GeoDataCatalogByCat?OpenView

Moskomarchitecture Geoportal

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://budget.mos.ru/gp_xml

Moscow Open Budget

4/11/2013

Sergey Chernov, Information Retrieval Basics

Openpolice.ru

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://apps4russia.ru/

http://www.mysocialcard.ru/

Apps4Russia
Annual contest of

open data-based mobile applications

Last winner project

Sotskartochka (My Social Card)

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://gis-lab.info/start.html

GIS-LAB

4/11/2013

Sergey Chernov, Information Retrieval Basics

Outline
Open Data
Online and Social Media Data Sources

Multimedia Archives
Hands-on Assignment

Sergey Chernov, Information Retrieval Basics

http://www.dowjones.com/factiva/

Factiva

-28 languages from 200 countries over 35 years - Top news sources include The Wall Street Journal, Dow Jones Newswires, The New York Times, The Sydney Morning Herald and Le Monde.
4/11/2013 Sergey Chernov, Information Retrieval Basics

mlg.ru

Medialogia

4/11/2013

Sergey Chernov, Information Retrieval Basics

Integrum.ru

Integrum

4/11/2013

Sergey Chernov, Information Retrieval Basics

www.mediacloud.org

4/11/2013

Sergey Chernov, Information Retrieval Basics

Media Cloud Twitter vs LiveJournal

4/11/2013

Sergey Chernov, Information Retrieval Basics

Russian Media Cloud

4/11/2013

Sergey Chernov, Information Retrieval Basics

Russian Media Cloud (cont)

4/11/2013

Sergey Chernov, Information Retrieval Basics

Russian Media Cloud: Association Golos

Elections Elect- Pressure Fine for Monitorin ODIHR oral on Golos illegal Media / Cluster g and obserLegis- observers agitatio Violations vers lation in regions n Reports kommersant.ru 51 19 -9 2 golos-org LJ 29 24 7 3 -gazeta.ru 22 2 -1 9 echo.msk.ru/blog 13 18 1 -1 newsru.com 11 ---3 svpressa.ru 9 2 --1 lenta.ru/ 2 ---3 www.kavkazuzel.ru 12 1 --1 aif.ru 11 3 ---anticompromat LJ 5 2 --1 vz.ru 3 ---4 bfm.ru 2 ---4 www.news2.ru 3 2 --3 fontanka.ru 1 1 --3
4/11/2013

NTV and "Surkov' s Propaganda" --3 -3 -1 --3 6 1 1 2

ShibaRoskomnova's DDOS nadzor TOTAL noteattacks warning book --6 -1 2 3 --2 1 -1 2 --1 -2 -2 ----1 1 1 --2 1 2 3 4 1 -1 -4 1 1 81 63 46 34 22 17 15 15 14 14 14 12 12 11

Sergey Chernov, Information Retrieval Basics

Visualizing Data with Gephi

gephi.org

4/11/2013

Sergey Chernov, Information Retrieval Basics

Cities

Facebook Users city -

Facebook

city_code1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

60 10660 280 60 3820 20 520 320 20 3820 6400 620 160 3580 380 40

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://habrahabr.ru/post/123856/

Vk.com

4312 60 - Server with 12 cores and 40Tb of space 89141 6336 1805 24208 3548 12697 11280 5918 42679 68572 21929 1603 27148 2368 10660 280 60 3820 20 520 320 20 3820 6400 620 160 3580 380

- Several weeks of crawling

- PostgreSQL database
- 120 mln user profiles
4/11/2013

Cities city_code 1 city 1 2 3 4 5 6 - 7 8 9 10 11 12 - 13 14 15

VK Users Facebook Users

Sergey Chernov, Information Retrieval Basics

Twitter API

4/11/2013

Sergey Chernov, Information Retrieval Basics

Yandex Blog Search

4/11/2013

Sergey Chernov, Information Retrieval Basics

Topsy

4/11/2013

Sergey Chernov, Information Retrieval Basics

SocialBro

http://userguide.socialbro.com/post/40596560247/using-socialbro-to-reach-themaximum-audience-with-your
4/11/2013 Sergey Chernov, Information Retrieval Basics

Social Bro (cont)

4/11/2013

Sergey Chernov, Information Retrieval Basics

http://nodexl.codeplex.com/

NodeXL

4/11/2013

Sergey Chernov, Information Retrieval Basics

Outline
Open Data
Online and Social Media Data Sources

Multimedia Archives
Practical Assignment

Sergey Chernov, Information Retrieval Basics

Library of Congress

4/11/2013

Sergey Chernov, Information Retrieval Basics

Russian State Library

4/11/2013

Sergey Chernov, Information Retrieval Basics

The MAT

4/11/2013

Sergey Chernov, Information Retrieval Basics

Tate

4/11/2013

Sergey Chernov, Information Retrieval Basics

The Hermitage

4/11/2013

Sergey Chernov, Information Retrieval Basics

Outline
Open Data
Online and Social Media Data Sources

Multimedia Archives
Hands-on Assignment

Sergey Chernov, Information Retrieval Basics

http://www.slideshare.net/timoreilly/open-up-15195896

Types of OGD Apps

13 categories of apps

identified:
health & safety, entertainment, sports, transportation, environment, education, public safety & law

city services,
real estate,

enforcement, food & dining, development, business & finance, government & civics.

4/11/2013

Sergey Chernov, Information Retrieval Basics

Russian Gov Data Applications

Government tender system
3akupki.ru., bicotender.ru, initpro.ru, ist-budget.ru, is-zakupki.ru,

goszakaz.ru, goszakupki-cpr.ru, marketing.interfax.ru, multitender.ru, my-tender.ru, regionzakaz.ru, seldon.ur.ru, tenderspb.ru Legal cases analysis rospravosudie.com, www.pravo.ru, www.intergrum.ru, sparkinterfax.ru Income declarations declarator.org, publicprofit.ru Budget budget4me.ru, rosspending.ru Public transport rasp.yandex.ru, maps.google.ru, doroga.tv, Rusavtobus.ru
Sergey Chernov, Information Retrieval Basics

4/11/2013

International OGD Apps

http://www.fixmystreet.com/
http://seeclickfix.com/ http://codeforamerica.org/

http://open311.org/
www.recovery.gov http://data.seattle.gov/ (Notifire, MomMaps) https://nycopendata.socrata.com/ (WiFi) http://2011.nycbigapps.com/submissions http://daten.berlin.de/ (Wheelmap, Aircraft noise)
4/11/2013 Sergey Chernov, Information Retrieval Basics

Hands-on assignment
Working in teams of 2-3 people
Search for motivating open data projects on the Web Explore data.mos.ru, openpolice.ru, opengovdata.ru or

any other resource of your choice Propose a project based on some dataset. How the data looks like? What is missing? Which skills and tools you might need? Present your project

4/11/2013

Sergey Chernov, Information Retrieval Basics