Flow Data Post (Reguler)

Cluster HDP IPD Production

Ambari 192.168.180.1
Version HDP 3
Kafka Manager 192.168.180.1
No Hostname IP Core RAM HDD OS Service PIC
Centos
1 master.production.bt 192.168.180.151 24 128GB 2TB master|zk|jn Diki
Linux
Centos namenode|zk|jn|rm|
2 namenode01.production.bt 192.168.180.152 24 128GB 2TB Diki
Linux Hmaster
Centos namenode|zk|jn|rm|
3 namenode02.production.bt 192.168.180.153 24 128GB 2TB Diki
Linux Hmaster
Centos
4 datanode01.production.bt 192.168.180.154 24 128GB 28TB datanode|nm|rs|ignite Diki
Linux
Centos
5 datanode02.production.bt 192.168.180.155 24 128GB 28TB datanode|nm|rs|ignite Diki
Linux
Centos
6 datanode03.production.bt 192.168.180.156 24 128GB 28TB datanode|nm|rs|ignite Diki
Linux
Centos
7 datanode04.production.bt 192.168.180.157 24 128GB 28TB datanode|nm|rs|ignite Diki
Linux
Centos
8 datanode05.production.bt 192.168.180.158 24 128GB 28TB datanode|nm|rs|ignite Diki
Linux
Centos
9 kafka01.production.bt 192.168.180.159 24 128GB 4TB kafka Diki
Linux
Centos
10 kafka02.production.bt 192.168.180.160 24 128GB 4TB kafka Diki
Linux
Centos
11 kafka03.production.bt 192.168.180.161 24 128GB 4TB kafka Diki
Linux
Centos
12 kafka04.production.bt 192.168.180.162 24 128GB 4TB kafka Diki
Linux
Centos
13 kafka05.production.bt 192.168.180.163 24 128GB 4TB kafka Diki
Linux
Centos
14 serviceengine.production.bt 192.168.180.171 24 128GB 2TB hadoop-client Diki
Linux
Centos
15 solr01.production.bt 192.168.180.164 16 64GB 2TB solr Ashif
Linux
Centos
16 solr02.production.bt 192.168.180.165 16 64GB 2TB solr Ashif
Linux

Cluster Elasticsearch IPD Production

192.168.180.188:56
Kibana
01
192.168.180.188:52
Ingest Insert/Query
00
No Hostname IP Core RAM HDD OS Service PIC
elastic01.production.b
1 192.168.180.181 24 128GB 2TB (ssd) Centos Linux eshot Diki
t
elastic02.production.b
2 192.168.180.182 24 128GB 2TB (ssd) Centos Linux eshot Diki
t
elastic03.production.b
3 192.168.180.183 24 128GB 2TB (ssd) Centos Linux eshot Diki
t
elastic04.production.b
4 192.168.180.184 24 128GB 2TB (ssd) Centos Linux eshot Diki
t
elastic05.production.b
5 192.168.180.185 24 128GB 2TB (ssd) Centos Linux eshot Diki
t
elastic06.production.b 2TB +
6 192.168.180.186 24 128GB Centos Linux eswarm | esmaster | esingest Diki
t 28TB
elastic07.production.b 2TB +
7 192.168.180.187 24 128GB Centos Linux eswarm | esmaster | esingest Diki
t 28TB
elastic08.production.b 2TB + eswarm | esmaster | kibana |
8 192.168.180.188 24 128GB Centos Linux Diki
t 28TB haproxy

Elasticsearch Log Data

Kibana 192.168.180.1
No Hostname IP Core RAM HDD OS Service PIC
elasticsearch | kibana | nifi |
1 monitoring.logs.bt 192.168.24.68 16 92GB 2TB Centos Linux Diki
ipd alerting system

NSQ IPD Production

NSQ Admin
No Hostname IP Core RAM HDD OS Service PIC
 nsq|beanstalk|ssdb|nsq
1 nsq01.ipd.bt 192.168.24.81 16 64GB 2TB Centos Linux Ashif
admin
2 nsq02.ipd.bt 192.168.24.82 16 64GB 2TB Centos Linux nsq|beanstalk|ssdb Ashif
nsq|beanstalk|ssdb|
3 nsq03.ipd.bt 192.168.24.90 16 64GB 2TB Centos Linux Ashif
beanstalk admin

Nifi IPD Production

Nifi UI 192.168.180.41/44
No Hostname IP Core RAM HDD OS Service PIC
1 nifi01.ipd.bt 192.168.180.41 24 128GB 2TB Centos Linux Nifi Diki
2 nifi02.ipd.bt 192.168.180.42 24 128GB 2TB Centos Linux Nifi Diki
3 nifi03.ipd.bt 192.168.180.43 24 128GB 2TB Centos Linux Nifi Diki
4 nifi04.ipd.bt 192.168.180.44 24 128GB 2TB Centos Linux Nifi Diki
5 nifi05.ipd.bt 192.168.180.45 24 128GB 2TB Centos Linux Nifi Diki
6 nifi06.ipd.bt 192.168.180.46 24 128GB 2TB Centos Linux Nifi Diki

News Percolator
No Hostname IP Core RAM HDD OS Service PIC
Percolator All Platform |
1 perculator01.cluster.bt 192.168.24.87 16 64GB 2TB Centos Linux Alias ON-PN | Blacklist ON- Ashif
PN
Percolator All Platform |
2 perculator02.cluster.bt 192.168.24.88 16 64GB 2TB Centos Linux Alias ON-PN | Blacklist ON- Ashif
PN
Percolator All Platform |
3 perculator03.cluster.bt 192.168.24.89 16 64GB 2TB Centos Linux Alias ON-PN | Blacklist ON- Ashif
PN

Cluster Graph IPD

No Hostname IP Core RAM HDD OS Service PIC
1 snacore01.cluster.bt 192.168.20.79 16 128GB 2TB Centos Linux Gephi Ulil
2 snacore03.cluster.bt 192.168.20.81 16 100GB 2TB Centos Linux Gephi Ulil
3 snacore04.cluster.bt 192.168.20.82 16 86GB 2TB Centos Linux Gephi Ulil
4 snacore05.cluster.bt 192.168.20.83 16 128GB 2TB Centos Linux Gephi Ulil

Cluster GlusterFs Graph IPD

No Hostname IP Core RAM HDD OS Service PIC
1 snagluster01.cluster.bt 192.168.20.86 16 64GB 11TB Centos Linux GlusterFS Ashif
2 snagluster02.cluster.bt 192.168.20.93 16 64GB 11TB Centos Linux GlusterFS Ashif

Cluster Management IPD

No Hostname IP Core RAM HDD OS Service PIC
1 ipd-70 192.168.24.70 16 64GB 2TB Centos Linux MySQL | MongoDB | Redis Ashif

Server Front End IPD

No Hostname IP Core RAM HDD OS Service PIC
1 ipd-64 192.168.24.64 16 64GB 2TB Centos Linux Nginx Ashif

NSQ PH
No Hostname IP Core RAM HDD OS Service PIC
1 solrSc 192.168.114.238 16 64GB 2TB Centos Linux NSQ Diki
List Topic Kafka IPD
Kafka Broker :
kafka01.production.bt:6667,kafka02.production.bt:6667,kafka03.production.bt:6667,kafka04.production.bt:6667,kafka05.producti
on.bt:6667

Kafka Manager:
http://192.168.24.21:9000/clusters/IPD-Production

Kafka Topic Group Note

sc-twitter-post Stream data raw (twitter)
sc-twitter-profiling Smart Crawler Twitter Stream data raw profile (twitter)
sc-twitter-relation Stream data following - follower (twitter)
sc-instagram-post Stream data raw post (instagram)
sc-instagram-post-web Stream data raw post web (instagram)
sc-instagram-comment Smart Crawler Instagram Stream data raw comment (instagram)
sc-instagram-profiling Stream data raw profile (instagram)
sc-instagram-relation Stream data following - follower (instagram)
sc-youtube-post Stream data raw post (youtube)
sc-youtube-comment Smart Crawler Youtube Stream data raw comment (youtube)
sc-youtube-profiling Stream data raw profile channel (youtube)
sc-facebook-personal-post Stream data raw personal post (facebook)
sc-facebook-personal-comment Stream data raw personal comment (facebook)
sc-facebook-page-post Stream data raw page post (facebook)
sc-facebook-page-comment Stream data raw page comment (facebook)
sc-facebook-page-fans Stream data raw page fans (facebook)
sc-facebook-group-post Stream data raw group post (facebook)
sc-facebook-group-comment Stream data raw group comment (facebook)
sc-facebook-group-member Smart Crawler Facebook Stream data raw group member (facebook)
sc-facebook-profile-personal Stream data raw profiling personal (facebook)
sc-facebook-profile-page Stream data raw profiling page (facebook)
sc-facebook-profile-group Stream data raw profiling group (facebook)
sc-facebook-profile-event Stream data raw profiling event (facebook)
sc-facebook-profile-location Stream data raw profiling location (facebook)
sc-facebook-image Stream data facebook image (facebook)
sc-facebook-relation Stream data raw friends (facebook)
sc-online-news Stream data raw (online-news)
sc-printed-news Smart Crawler News Stream data raw (printed-news)
sc-tv-news Stream data raw (tv-news)
Smart Crawler
sc-website Stream data raw (website/blog)
Blog/Website
sc-ecommerce-indonesia Stream data raw (ecommerce indonesia)
Smart Crawler Ecommerce
sc-ecommerce-malaysia Stream data raw (ecommerce malaysia)
sc-linkedin-profile Smart Crawler Linkedin Stream data raw profile (linkedin)
ipd-log-sc Smart Crawler Log Log data smart crawler

ipd-twitter-post Stream clean content twitter post

Integrated Product Twitter
ipd-twitter-profile Stream blacklist content twitter post
ipd-instagram-post Stream clean content instagram post
Integrated Product
ipd-instagram-comment Stream clean content instagram comment
Instagram
ipd-instagram-profile Stream blacklist content instagram post & comment
ipd-youtube-post Stream clean content youtube post
ipd-youtube-profile Integrated Product Youtube Stream clean content youtube comment
ipd-youtube-spam Stream blacklist content youtube post & comment
ipd-facebook-post Integrated Product Stream clean content facebook post
ipd-facebook-comment Facebook Stream clean content facebook comment
Integrated Product Data
ipd-log-data Stream data log from content filtering engine
Log

ann-twitter-post Stream data nlp result twitter post

Annotation Twitter
ann-twitter-profiling Stream data nlp + cv result twitter profiling
ann-instagram-post Stream data nlp result instagram post
ann-instagram-comment Annotation Instagram Stream data nlp result instagram comment
ann-instagram-profiling Stream data nlp + cv result instagram profiling
ann-youtube-post Stream data nlp result youtube post
ann-youtube-comment Annotation Youtube Stream data nlp result youtube comment
ann-youtube-profiling Stream data nlp + cv result youtube profiling
ann-facebook-post Annotation Facebook Stream data nlp result facebook post
ann-facebook-comment
ann-facebook-profiling
ann-online-news
ann-online-news-statement
ann-printed-news
ann-printed-news-statement
ann-tv-news
ann-tv-news-statement
ann-website
ann-all
ipd-log-ann
Stream data nlp result facebook comment
Stream data nlp + ds + cv result facebook profiling
Stream data nlp result online news
Stream data nlp result printed news
Annotation News
Stream data nlp result tv news
Annotation Website Stream data nlp result website
Annotation All Source Stream data nlp all source
Annotation Log Log data from annotation pipeline

cv-twitter Computer Vision Twitter Stream data comvis result twitter

cv-instagram Computer Vision Instagram Stream data comvis result instagram
cv-facebook Computer Vision Facebook Stream data comvis result facebook
cv-youtube Computer Vision Youtube Stream data comvis result youtube
Computer Vision News &
cv-news Stream data comvis result online news
Blog/Website
cv-all Computer Vision All Source Stream data comvis result all source
Computer Vision News &
ipd-log-cv Log data from comvis pipeline
Blog/Website

ipd-online-news Stream data percolator alias - blacklist online news

ipd-printed-news Stream data percolator alias - blacklist printed news
Integrated Product Engine
ipd-tv-news Stream data percolator alias - blacklist tv news
ipd-image Stream data image criteria