CMS MC on the DOC Cloud

Below is a log of my first attempt of running CMS MC at the DOC cloud. The short version, minus all the detours is here.

(a) Setup
It's not immediately obvious, but here's my best guess:
export SCRAM_ARCH=slc5_amd64_gcc434
. /cvmfs/cms.cern.ch/cmsset_default.sh
scram project CMSSW CMSSW_4_1_5
cd CMSSW_4_1_5
cd src
cmsenv

(b) Monte Carlo
ln -s /opt/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_4_1_5/python/Configuration/Generator/bJpsiX_7TeV_cfi.py .
cmsDriver.py bJpsiX_7TeV_cfi.py -s GEN,SIM,DIGI,RECO -n 10 --no_exec --conditions=auto:startup
(bJpsiX_7TeV_cfi.py is just taken from the release area because it runs without any error locally (i.e. non-grid/non-cloud), but is otherwise arbitrary.)
This generates bJpsiX_7TeV_cfi_py_GEN_SIM_DIGI_RECO.py which I am trying to use with cmsRun:
cmsRun bJpsiX_7TeV_cfi_py_GEN_SIM_DIGI_RECO.py

(b1) SITECONF
It balks immediately:
[dbauer@host-10-40-1-66 src]$ cat bJpsiX_7TeV_3.log
[snip]
Valid site-local-config not found at /opt/cms/SITECONF/local/JobConfig/site-local-config.xml
---- Incomplete configuration END
[snap]
This is probably related to T2_UK_London_IC missing in frontier.cern.ch/geolist.txt.
I try to change it back to RHUL on the webinterface, but the machine doesn't seem to pick it up, though there is no error.
I then force a reboot from the webinterface and it clearly picks up RHUL:
[dbauer@host-10-40-1-66 ~]$ cat /etc/cernvm/site.conf
[snip]
CMS_LOCAL_SITE=T3_UK_London_RHUL
but there is still no /etc/cms/SITECONF/
So I make the directory by hand and try to copy the relevant files into it:
sudo mkdir /etc/cms
sudo mkdir /etc/cms/SITECONF
cd /etc/cms/SITECONF
sudo wget http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/SITECONF/T2_UK_London_IC/JobConfig/cmsset_local.sh
sudo wget http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/SITECONF/T2_UK_London_IC/JobConfig/cmsset_local.csh
sudo wget http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/SITECONF/T2_UK_London_IC/JobConfig/site-local-config.xml
I then stick site-local-config.xml into /opt/cms/SITECONF/local/JobConfig/
The next try:
Did not find frontier-connect in calib-data section in /opt/cms/SITECONF/local/JobConfig/site-local-config.xml
I am going to need some help here. (This is because I downloaded the wrong file, oops.)
But first -- let's start from scratch.....

From scratch:
This just made it worse.
It now recognises T2_UKI_London_IC, but when I log into the newly created machine I see:
[dbauer@host-10-40-1-66 ~]$ ls -l /opt/
total 4
lrwxrwxrwx 1 root root 33 May 10 17:44 classads -> /mnt/cvmfs/grid/3.1.39-0/classads
lrwxrwxrwx 1 root root 32 May 10 17:44 d-cache -> /mnt/cvmfs/grid/3.1.39-0/d-cache
lrwxrwxrwx 1 root root 28 May 10 17:44 edg -> /mnt/cvmfs/grid/3.1.39-0/edg
lrwxrwxrwx 1 root root 33 May 10 17:44 external -> /mnt/cvmfs/grid/3.1.39-0/external
lrwxrwxrwx 1 root root 30 May 10 17:44 glite -> /mnt/cvmfs/grid/3.1.39-0/glite
lrwxrwxrwx 1 root root 31 May 10 17:44 globus -> /mnt/cvmfs/grid/3.1.39-0/globus
lrwxrwxrwx 1 root root 28 May 10 17:44 gpt -> /mnt/cvmfs/grid/3.1.39-0/gpt
drwxr-xr-x 2 root root 4096 May 10 17:44 grid
lrwxrwxrwx 1 root root 28 May 10 17:44 lcg -> /mnt/cvmfs/grid/3.1.39-0/lcg
None of these directories (apart from 'grid') exist.

Advice is to update to cern VM 2.3 though I doubt this has anything to do with this problem.
To test this hypothesis, I make a new machine, using RHUL again.
Still doesn't work, grr.... As it turns out it's just really really slow. Suddenly the links appear !

From scratch, part II
Start a second machine, different IP and all. go to lunch.
Log into web interface, make user, log into machine:
[dbauer@host-10-40-1-67 ~]$ ls -l /opt/
total 0
[dbauer@host-10-40-1-67 ~]$ uptime
13:12:58 up 44 min, 1 user, load average: 0.12, 0.03, 0.01
Change "appliance primary group" to CMS on webinterface and hit save.
[dbauer@host-10-40-1-67 ~]$ ls /opt/
classads cms d-cache edg external glite globus gpt grid lcg
Wow....
The instruction recommend a reboot at this stage, but let's see first:
No /cvmfs, that's a bit of a throwback, let's reboot !
Check that is has rebooted (log in again, uptime) and then press the OK button on the web interface. I realise I have no reached the realm of voodoo.
Now it's back to:
[dbauer@host-10-40-1-67 ~]$ export SCRAM_ARCH=slc5_amd64_gcc434
[dbauer@host-10-40-1-67 ~]$ . /cvmfs/cms.cern.ch/cmsset_default.sh
Your shell is not able to find where cmsset_default.sh is located.
Either you have not set VO_CMS_SW_DIR or OSG_APP correctly
or SCRAM_ARCH is not set to a valid architecture.
/bin/ls: /opt/cms/slc5_amd64_gcc434/etc/profile.d/: No such file or directory
And I get lots of blinking lights here:
[dbauer@host-10-40-1-67 ~]$ ls /opt/
classads d-cache edg external glite globus gpt grid lcg
(uptime is now 10 min)
Let's see if patience is a virtue here... 12 min ... 21 min ...
After 41 min I give up and hit the 'Save' button cernvm/Config/ webpage again, and yes, all the files are back. SIGH.
And to get the CMS_LOCAL_SITE=T2_UK_London_IC back in /etc/cernvm/site.conf I need to hit the 'Save' button in the "Advanced configuration and proxy server setup " again ?! I get the feeling I am missing something fundamental here.
So tried again, clean machine, hit Save everywhere, reboot and indeed, I need to go back and hit 'Save' again to make the files in /opt reappear, though /etc/cernvm/site.conf seems to be preserved.

Still no site config files, so get them by hand
Make a directory for them:
/etc/cms/SITECONF/T2_UK_London_IC/JobConfig
and
[dbauer@host-10-40-1-67 SITECONF]$ pwd
/etc/cms/SITECONF
[dbauer@host-10-40-1-67 SITECONF]$ ls -l
total 4
drwxr-xr-x 4 root root 4096 May 11 14:56 T2_UK_London_IC
lrwxrwxrwx 1 root root 16 May 11 14:57 local -> T2_UK_London_IC/

cd T2_UK_London_IC
sudo wget http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/SITECONF/T2_UK_London_IC/JobConfig/cmsset_local.sh?revision=1.5
sudo mv cmsset_local.sh?revision=1.5 cmsset_local.sh
sudo wget -O cmsset_local.csh http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/SITECONF/T2_UK_London_IC/JobConfig/cmsset_local.csh?revision=1.5
sudo wget -O site-local-config.xml http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/SITECONF/T2_UK_London_IC/JobConfig/site-local-config.xml?revision=1.1

[dbauer@host-10-40-1-67 CMSSW_4_1_5]$ cd src/
[dbauer@host-10-40-1-67 src]$ cmsenv
[dbauer@host-10-40-1-67 src]$ cmsRun bJpsiX_7TeV_cfi_py_GEN_SIM_DIGI_RECO.py
MSTU(12) changed from 0 to 12345
[snip]
Begin processing the 10th record. Run 1, Event 10, LumiSection 1 at 11-May-2011 15:33:58.665 BST
[snip]
[dbauer@host-10-40-1-67 src]$ ls -l *.root
-rw-rw-r-- 1 dbauer dbauer 451160 May 11 15:33 bJpsiX_7TeV_cfi_py_GEN_SIM_DIGI_RECO.root
:-)