Cloud Cluster Users Meeting

Cluster Characteristics

• Data-Intesive Computing Cluster: Designed for large-scale data processing workloads that are not well served by other computational resources like the ones available at super computing center (e.g., Teragrid), or local group clusters.
• Data-intensive computing software stack: HDFS, Hadoop Map Reduce, HBase, Hive, Pig.
• Managed by the PDL staff: Michael Stroucken and Mitch Franzos.

• Cloud Cluster Team:
  • System administrators: Michael Stroucken and Mitch Franzos.
  • Volunteer Students: Wittawat Tantisiriroj, Lin Xiao, Milo Polte, Kai Ren, Soila Pertet, Swapnil Patil.
  • Faculty: Julio López and Garth Gibson (technical guidance).

What is it being used for:

• Comp-Bio: tissue image processing
• LTI/ML: Read the Web, Worldly Knowledge.
• Tweeter data analysis
• Security / Malware analysis
• Seismology: seismic wavefield analysis and compression.
• Astrophysics/cosmology: Galaxy clustering, Quasar identification, time particle history.

Datasets:

• ClueWeb09: Targeted web crawl corpus
• Tweeter (Meeder/O'Connor).
• Dark Matter simulation output / black holes.
• Seismic wavefield (Quake group) (soon).

Getting Help

• Cloud Cluster Wiki: http://www2.pdl.cmu.edu/~twiki/cgi-bin/view/OpenCloud
• Hadoop / Hive / tools questions:
  • Cloud cluster user's mailing list: opencloud-users@ece.cmu.edu
  • (Hadoop/Pig/Hive/HBase/Thrift) wikis and user lists.
• Cluster operation / IT issues: e-mail disc-help@ece.cmu.edu. This goes into a queue in a request tracking system (RT). The staff process these appropriate requests.

Usage Monitoring

Tools

• http://hadoopjt.opencloud:40030/
• http://monitor.cloud.pdl.cmu.local/ganglia

System Usage

• http://monitor2.pdl.cmu.local/hdfs/history.html
• http://monitor2.pdl.cmu.local/hadoop/

Reducing Space Storage Usage

• RAIDTool (Wittawat).