Goal: Apart from the dev project, all other projects should only have access to the provider network via floating IPs.

Make the provider network private (this is a one-off and should not change (unless we have to recreate all networks yet again ;-)):
openstack network set --no-share provider
Use Role-Based Access Control (RBAC) to manage access to the network. Finding you way around: openstack network rbac list
Object ID is the network the rule applies to, to see which network this is use openstack network show [Object ID]
Now give the dev project direct access to the provider network:
openstack network rbac create --type network --action access_as_shared --target-project dev provider

---

All the sections of the cloud setup log relating to networking.

On cldsk03 as root:
/etc/sysconfig/network-scripts/
ifcfg-enp59s0f0 - main system one (for connection to the grid)
.12 for connection to vlan 12 (private internal cloud vlan) (11 on the tmp cloud)
ifcfg-eno* - not used (machine has six network ports, we are only using one)

[root@cldsk03 network-scripts]# cat ifcfg-enp59s0f0.12
VLAN=yes
DEVICE=enp59s0f0.12
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.0.1.1 <--- from /etc/hosts
NETMASK=255.255.248.0

Same on clddsk04, 05, with appropriate IPADDR

bring up the interface: ifup enp59s0f0.12

Test:
[root@cldsk05 network-scripts]# ping cldsk03.cloud
PING cldsk03.cloud (10.0.1.1) 56(84) bytes of data.
64 bytes from cldsk03.cloud (10.0.1.1): icmp_seq=1 ttl=64 time=0.228 ms

Simon says: "We have now commissioned the network. Apart from the firewall."
Several days later: There are now firewall rules for CEPH Monitor and CEPH OSD