LDoms on Sun SPARC Enterprise T5240

What

It is the third generation Chip Multithreading Technology (CMT) Ultra SPARC 2T Plus CPU. So yes, we purchased two of these beasts (128 virtul CPUs, 64GB RAM, 16 cryptographic units - MAU and total of 16 HDDs).

Why we need them

Drive up utilization and consolidate workloads with built-in virtualization technology - Logical Domains LDoms.

How to install LDoms - control one

Get the file LDoms_Manager-1_1.zip from Sun, unzip it and you'll get directory LDoms_Manager-1_1
Go to LDoms_Manager-1_1/Install and run the command install-ldm -d none
In this way, you'll not harden your system using Solaris Security Toolkit drive, otherwise you may not be able to login agan.
And now you'll have 2 new dirs in /opt - SUNWldm and SUNWjass.

Also don't forget to add /opt/SUNWldm/bin in PATH and /opt/SUNWldm/man in MANPATH
Make sure service svc:/ldoms/ldmd:default is running.
Okay, so 'primary' is now your control domain and will take care of all other LDoms.

• Virtual service for disk server - ldm add-vds vds-ldomname primary
• Virtual network terminal server - ldm add-vcc port-range=5000-5010 vcc-console primary
• Virtual service for network switch - ldm add-vsw net-dev=nxge0 sub-20 primary
• Verify what you did - ldm list-services

• Cryptographic engine - ldm set-mau N primary
• Virtual CPU - ldm set-vcpu N primary
• Memory - ldm set-memory Ng primary

• Check configuration - ldm list-spconfig
• Add configuration - ldm add-spconfig name
• verify resources are allocated to primary - ldm list
• Power off - init 0
• One note here - "init 5" behaives as "init 6", strange. So I "init 0" and from ALOM do poweroff

How to install guest LDoms

okay, so now we need to install guest LDoms. I'll write some notes/ideas from my experience (working together with colleagues Sergey/Andrii).

Basically we installed zfs root on first 2 HDDs (mirror zpool) - 146G.
We also created 6 other zpool (mirror of 2 HDDs) - 73G. Yes we'll have 4 LDoms, but 2 of them will use 2 zpool(s).

So variables used below are like:
LDOM = castro/pacific/oxygen/brahma
SPACE = space-ld1/ld2/ld3/ld4
NET = dco-20/dco-24/dco-dmz/dco-pub/dco-be/dco-bk (this is VSW, one built-in NIC with 4 ports nxge0-3 and additional NIC with 2 ports e1000g0/1)

• Create guest LDom - ldm add-domain ${LDOM}
• Add vcpu - ldm add-vcpu 16 ${LDOM}
• Add RAM - ldm add-memory 8g ${LDOM}
• Add MAU - ldm add-mau 2 ${LDOM}
• Add virtual network device on net switch - ldm add-vnet vnet0 ${NET} ${LDOM}
• Add virtual disk DEVICE to be system disk for LDom- ldm add-vdsdev /dev/zvol/dsk/${SPACE}/${LDOM} zdisk@vds-${LDOM}
• Add virtual DISK - ldm add-vdisk zdisk0 zdisk@vds-${LDOM} ${LDOM}
• Set autoboot to false - ldm set-variable auto-boot\?=false ${LDOM}
• Point to virtual disk device as boot device - ldm set-variable boot-device=/virtual-devices@100/channel-devices@200/disk@0 ${LDOM}
• Bind LDom - ldm bind-domain ${LDOM}
• Start LDom - ldm start-domain ${LDOM}

Now we have LDoms, there are many ways to install the OS. Say use jumpstart installation ( boot net -v - install from 'ok') or create zfs snapshot from 'reference' system, boot from cdrom and restore zfs snapshot to LDom.

Let's write some staps about second approach.

• Create additional virtual DEVICE and disks for 2 LDoms - ldm add-vdsdev /dev/zvol/dsk/${SPACE}/${LDOM} zdisk@vds-${LDOM}
• Create virtual DISK for 2 LDoms- ldm add-vdisk zdisk0 zdisk@vds-${LDOM} ${LDOM}
• Get solaris dvd image, save to /var/tmp
• Add cdrom for LDom you currently wanna install - ldm add-vdsdev /var/tmp/sol-10-u7-ga-sparc-dvd.iso cdrom@vds-misc
• This will require poweroff/poweron from sc>
• Add virtual cdrom - ldm add-vdisk cdrom cdrom@vds-misc ${LDOM}
• Save spconfiguration - ldm add-spconfig intial-new
• Again poweroff/poweron
• Stop domain, like ldm stop-domain castro
• Unbind domain, like ldm unbind castro
• Copy previously saved zfs to new filesystem - zfs send space-ld1/reference@installed | zfs receive -F space-ld1/castro
• Make sure this service is running svc:/ldoms/vntsd:default
• Start domain, like ldm start-domain castro
• Bind domain, like ldm bind castro
• Go to LDoms console - telnet localhost 5001
• You'll get to 'ok' so boot in single user mode from cdrom - boot cdrom -s
• See all zfs you have available for import - zpool import
• Import - zpool import -f space0
• init 6
• If needed change hostname/IP

Okay, I guess I didn't miss any major step(s)