Overall, Kubernetes is a stunningly extensible system - you can make it do almost anything you want.
However, that extensible nature is both a blessing and a curse - it can also be difficult to manage consistently at scale. Pivotal attempts to address a significant part of that by using BOSH and the stemcells it deploys…it certainly helps.
However, it also adds some challenges - the same ‘fixed’ stemcells that provide simpler management also make it more difficult to make changes that impact the system.
I had a customer request to investigate what it would take to make VxFlexOS (nee ScaleIO) work with PKS. VxFlexOS uses a kernel driver on the host to add block devices in
/dev in the form of
/dev/sciniX, which means editing the underlying stemcell to install the kernel driver (
scini.ko) as well as the userspace control binary
Fortunately, BOSH has a tool for adding custom additions to a stemcell without modifying the original upstream source (which is nice for minimizing the footprint of things you need to manage/update). We call these BOSH addons, I’d I recently updated some work done by Gary White and Paul Blum from the EMC/Pivotal teams to work with modern VxFlexOS and PKS. I worked with a colleague Stu to build an updated version of such a release and posted it.
Its fairly easy to install. Using the included manifest under
releases: - name: scaleio-release version: 9 addons: - name: scaleio jobs: - name: setup_sdc release: scaleio-release properties: scaleio: mdm: ips: - 10.192.225.105
Simply update the runtime config for BOSH:
bosh update-runtime-config manifest.yml
And then, if desired, force BOSH to redeploy:
bosh manifest -d current-deployment-name > deployment.yaml
bosh -d current-deployment-name deploy
And wait a bit. Eventually, you’ll end up with brand new VMs with the ScaleIO kernel driver and
In a future post I’ll show how to use with with the VxFlexOS CSI driver in K8S/PKS.
Credits: Lets be honest: I’m a newb with BOSH. Stu Charlton taught me how to do nearly all of this, and is 90% of the reason I got it work. Thanks Stu.