Network
The DS-PSI network is a relatively simple network consisting of a pair of Dell S5232F-ON switches (one in each rack) connecting up to a single S3248T-ON switch.
An additional S3248T-ON is used to provide a physically separate out-of-band management network fabric.
![graph "switch-layout" {
A [label="ds-psi-rack0-sw2\nS3248T-ON",shape=box];
{
rank=same;
B [label="ds-psi-rack1-sw1\nS5232F-ON",shape=box];
C [label="ds-psi-rack2-sw1\nS5232F-ON",shape=box];
D [label="ds-psi-rack0-sw1\nS3248T-ON",shape=box];
}
A -- {B C} [color="cyan:invis:cyan",label="10G"];
B -- C [color="magenta:invis:magenta",label="100G"];
}](../_images/graphviz-6dba9a4056bd9c7a48890ce99fd187ee935c9c02.png)
Switches
S3248T-ON
ds-psi-rack0-sw1
This 1G switch is used to provide the out-of-band management network. It is not configured with anything other than a hostname and services such as LLDP and SNMP.
ds-psi-rack0-sw2
This 1G switch is used as the core/aggregation root of the DS-PSI network, and also acts as router using its Layer 3 capabilities. It has BGP configured to allow BGP peers to advertise routes (used for Kubernetes networking), and contains static routes to route between the DS-PSI machines and the TOPIC network.
S5232F-ON
These 100G switches are used as Layer 2 top-of-rack switches for both DS-PSI
racks. They have redundant 10G uplinks to ds-psi-rack0-sw2, and share a
redundant 100G link between each other to carry high-bandwidth storage traffic.
The Rapid Per-VLAN Spanning Tree (Rapid PVST) protocol is used to prevent loops while prioritizing the 100G trunk over the 10G trunks.
Infrastructure-as-Code
Switches in DS-PSI are not managed by hand. Instead their configuration is part of the Ansible inventory in the ska-ds-psi-infrastructure repository and automated using the network Ansible playbook.