The Hoof & Paw
DocsCategoriesTagsView the current conditions from the WolfspyreLabs WeatherstationToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeBack to homepage

Getting things in place

Back to the top...

Stuff yer gonna need:

Proxmox Ceph Cluster

It kinda goes without saying. If you want to use the Ceph storage amalgam provided with Proxmox to enable this, you’ll need a functional Proxmox cluster..

This means at least two, realistically three nodes.

A Front-End gateway.

What’s going to respond to your HTTP(s) traffic?

Ceph’s RADOS gateway endpoint doesn’t provide any intrinsic request routing or load balancing. To maintain a durable front door, and reliable service, you’ll need a load balancer/front end.

How you facilitate this is up to you.

In my environment, I already have a functional, highly available HAProxy rig, backended by OPNsense… so I use that.


You’ll need DNS set up to point everything to the right place.

That will mean having:

Wildcard and A records

I chose as the root subdomain. Since I want to facilitate my offsite hosts to be able to access this as well, I need to enable external and internal resolution of the endpoints.

External records
Public-Facing DNS records:
dog   IN A
*.dog IN A
internal records
NS entries for the new subdomain
dog IN NS
dog IN NS
dog IN NS sub-zone

@             IN A

skwirreltrap  IN A
atticus       IN A
evey          IN A

px-m-40       IN A
px-m-41       IN A
px-m-42       IN A
px-m-43       IN A
px-m-44       IN A
px-m-45       IN A

*             IN A


  • At minimum, you’ll want a wildcard ssl certificate for your s3 apex. (in my case *

  • You may also want a wildcard ssl cert for s3 websites. But I’m not really sure (at the moment) how this works :)

Stuff ya may wanna read:

I’m going to lay out RADOS in alignment with the failure boundaries already established within my existing cluster. You may have different needs.

flowchart RL
  subgraph rm0["fa:fa-bolt Realm (namespace)"]
    subgraph zzz["RADOS Traffic Flow"]
     direction LR 
    subgraph zg0["Zone Group: Barn" ]
        subgraph zzy["Note:"]
          zzya["Zone Groups contain one or more zones. They must have one master zone."]
          direction LR 
      subgraph z0["Zone - PXM Master" ]
        subgraph zzx["Note:"]
          zzxa["Zones define an isolation/replication boundary."]
          direction LR 
        subgraph n40["Physical host: px-m-40"]
          n40v198["Node 40 Ceph Network<br>"]
          r40a["RADOS OSG Process 40A<br>"]
        subgraph n41["Physical host: px-m-41"]
          n41v198["Node 41 Ceph Network<br>"]
          r41a["RADOS OSG Process 41A<br>"]
        subgraph n42["Physical host: px-m-42"]
          n42v198["Node 42 Ceph Network<br>"]
          r42a["RADOS OSG Process 42A<br>"]
        subgraph n43["Physical host: px-m-43"]
          n43v198["Node 43 Ceph Network<br>"]
          r43a["RADOS OSG Process 43A<br>"]
        subgraph n44["Physical host: px-m-44"]
          n44v198["Node 44 Ceph Network<br>"]
          r44a["RADOS OSG Process 44A<br>"]
        subgraph n45["Physical host: px-m-45"]
          n45v198["Node 45 Ceph Network<br>"]
          r45a["RADOS OSG Process 45A<br>"]
  subgraph world["public requests"]
    direction BT
    usera["User Requests"]
    userb["from outside"]
    userc["the cluster"]
  subgraph op["OPNSense Cluster"]
    direction BT
    subgraph OPNHAP["OPNSense HAProxy"]
      direction BT
      zvip0["https://* <br>"]
    opv1["OPNSense VIP Network<br>"]
    opv198["OPNSense CEPH Network <br>"]
    opv2["OPNSense Public Network"]
  r40a -.-> n40v198 --- n40 --> opv198 
  r41a -.-> n41v198 --- n41 --> opv198 
  r42a -.-> n42v198 --- n42 --> opv198 
  r43a -.-> n43v198 --- n43 --> opv198 
  r44a -.-> n44v198 --- n44 --> opv198 
  r45a -.-> n45v198 --- n45 --> opv198 
  opv198 -.-> opv1 -.-> zvip0 ===> r40a & r41a & r42a & r43a & r44a & r45a

   usera & userb & userc -.- world ---> opv2 -.- opv1 -.-> zvip0 ---> world