Skip to main content
 

Getting Technical with Longleaf and Pine


Stepping into a few of the technical details of just what comprises Longleaf and Pine, here’s a list of tidbits:

  • Use different job scheduler (SLURM)
  • Distinguish node provisioning from node configuration (Cobbler, SALT)—to facilitate automated deployment of distinct node types, both locally, remotely (e.g., eventually in the cloud)
  • Use Ethernet only for interconnect (e.g., no Infiniband)
  • Deploy general purpose nodes to big data nodes in a 4:1 ratio (all nodes are from Dell)
    • General Purpose nodes (120):
      • 2x Intel E2680, 2.5GHz; 256GB RAM (24-cores; 2 sockets)
      • 2x 10Gbps Ethernet
      • 1x 400GB Solid State Drive (for GPFS Local Read Only Cache “LRoC”)
      • 1x 300GB 10K RPM SAS drives
    • Big Data nodes (30):
      • 2x Intel E2680, 2.5GHz;
      • 256GB RAM (24-cores; 2 sockets)
      • 2x Mellanox Dual-Port 40Gbps NIC
      • 2x 10Gbps Ethernet
      • 1x 800GB Solid State Drive (for GPFS Local Read Only Cache “LRoC”)
      • 2x 300GB 10K RPM SAS drives

  • Deploy large memory nodes (3TB memory), and plan to add next-gen GPU nodes
  • Dedicated storage nodes (8):
    • 2x Intel E5-2643, 3.4GHz
    • 128GB RAM (12-cores total; 2 sockets)
    • 4x Dual-Port FC HBAs
    • 2x Mellanox Dual-Port 40Gbps NIC
    • 2x 10Gbps Ethernet
    • 2x 300GB 10K RPM SAS drives
  • Add capabilities on an 18-month cycle (rather than a 4-5 year cycle)
  • GPFS parallel filesystem (a.k.a., IBM Spectrum Scale now)
    • SSD on every node for Local Read Only Cache (LRoC)
    • Tiered disk pools
  • High-Performance controllers and disk (Netapp e-Series)
    • 14x 12GB controller, 4x16Gb FC ports
      • 2x for metadata
      • 8x for SSD tier
      • 4x for SAS tier
    • 48x 800GB Solid-State-Disk
    • 192x 1.6TB Solid-State-Disk
    • 720x 4TB SAS Disk


And here’s a glimpse of how the different parts stitch together with interconnections. Note particularly the CPU Frequency, Memory bandwidth per core, Ethernet bandwidth per core, and the number of hops in the interconnect fabric.
[table width =”100%” style =” table-hover” responsive =”false”]
[table_head]
[th_column][/th_column]
[th_column]Kure[/th_column]
[th_column]KillDevil[/th_column]
[th_column]Longleaf
General Use[/th_column]
[th_column]Longleaf
Big Data[/th_column]
[/table_head]
[table_body]
[table_row]
[row_column]CPU Frequency (higher is better)[/row_column]
[row_column]2.8Ghz[/row_column]
[row_column]2.66Ghz[/row_column]
[row_column]2.5Ghz[/row_column]
[row_column]3.4Ghz[/row_column]
[/table_row]
[table_row]
[row_column]Number of cores per socket/system[/row_column]
[row_column]4/8[/row_column]
[row_column]6/12 and 8/16[/row_column]
[row_column]12/24[/row_column]
[row_column]6/12[/row_column]
[/table_row]
[table_row]
[row_column]HTT Enabled for greater memory,
network, and disk parallelism?[/row_column]
[row_column]No[/row_column]
[row_column]No[/row_column]
[row_column]Yes[/row_column]
[row_column]Yes[/row_column]
[/table_row]
[table_row]
[row_column]Memory capacity[/row_column]
[row_column]48GB-to-72GB[/row_column]
[row_column]48GB-to-128GB[/row_column]
[row_column]256GB[/row_column]
[row_column]256GB[/row_column]
[/table_row]
[table_row]
[row_column]Bus Speed (higher is better)[/row_column]
[row_column]6.4GT/s QIP[/row_column]
[row_column]6.4GT/s QIP[/row_column]
[row_column]9.6GT/s QIP[/row_column]
[row_column]9.6GT/s QIP[/row_column]
[/table_row]
[table_row]
[row_column]Max Memory Bandwidth (higher
is better)[/row_column]
[row_column]32GB/s[/row_column]
[row_column]32GB/s[/row_column]
[row_column]68GB/s[/row_column]
[row_column]68GB/s[/row_column]
[/table_row]
[table_row]
[row_column]Memory Speed (higher is better)[/row_column]
[row_column]1333Mhz[/row_column]
[row_column]1333Mhz[/row_column]
[row_column]2133Mhz[/row_column]
[row_column]2133Mhz[/row_column]
[/table_row]
[table_row]
[row_column]Memory per Core (higher is better)[/row_column]
[row_column]6GB-to-9GB[/row_column]
[row_column]4GB-to-8GB[/row_column]
[row_column]10.67GB[/row_column]
[row_column]21.34GB[/row_column]
[/table_row]
[table_row]
[row_column]Ethernet Bandwidth per Core
(higher is better)[/row_column]
[row_column]16MB/s[/row_column]
[row_column]8-to-10.67MB/s[/row_column]
[row_column]106.7MB/s[/row_column]
[row_column]213.34MB/s[/row_column]
[/table_row]
[table_row]
[row_column]Infiniband Bandwidth per Core
(higher is better)[/row_column]
[row_column]4Gb/s[/row_column]
[row_column]2-to-2.67Gb/s[/row_column]
[row_column]N/A[/row_column]
[row_column]N/A[/row_column]
[/table_row]
[/table_body]
[/table]

For those who are “core counters,” Longleaf has a little more than 3,120 cores. That’s about 1.5 times the core count of Kure.

Comments are closed.