thou art thy links

2005.06.07

I've copied and pasted some code and comments at Markus' request to explain what messages should be passed between the ACP and the Sm/Scpu into this file: newmessages.txt.

2005.06.04

Documentation reading day:

• EVB/L3 for Aces by Boris
• EVB/L3 for Experts by Nuno

Questions/Notes:

• Expect Scripting resources

2005.06.03

The task at hand: write java classes that extend daqmsg for communication between the ace control panel (?) and the scpu/proxy. The contents of those message are outlined in the Sm.idl and Scpu.idl files in the cdfevb.sm and cdfevb.scpu packages respectively.

Discussing with Markus what I could use as an example class to guide my learning here...

Also, previous misunderstanding cleared up. The IDL files do not need to be touched. These would need to be changed if we were using idltojava, but these classes will be handwritten by yours truly, so this is not needed.

2005.06.02

Taking the day to learn a bit about Java IDL, modules, etc... so I can understand and modify the GUI widgets for various obsolete functions and the function calls themselves in the ace control panel. Useful refs so far:

• Sun Java IDL Tutorial
• ...

2005.06.01

Starting work on getting rid of obsolete GUI widgets and function calls for the Ace Control Panel (in the SM and SCPU packages of cdfevb). Brief discussion with MK regarding whether I could make a new package in cdfevb2 called ace and copy this code there so as to make a break with the old Panel. After consideration of the extra work involved in that (e.g., there is no SM package in cdfevb2 since there is no SM. This would require serious reworking), I see now that it's preferable to keep this tool under the cdfevb code. More on this tomorrow...

2005.05.23

Did some refinements of the EVB start/stop button. Need to speak to Steve about "Expert's Corner"... Owl shifts are a drag.

2005.05.17

Started working on EVB2 panel. Removed following buttons:

• From Checks: Scramnet, ATM Switch. SM + SCPUs --> SCPUs.
• From EVB Proxy Control: Send Abort.
• From Clean up Actions: Clean up EVB.

New code is implemented for Check EVB Proxy, which checks whether the EVB proxy is running on any of 8 partitions. New code for START EVB proxy that does the same check and then starts the proxies on all partitions.

2005.04.27

Brief convo with Markus last night. He mentioned one of the challenges ahead for us: in the new Event Builder, there can be one proxy per partition, instead of one single proxy. Actions such as "Clean up EVB" or "Stop Proxy" will have to be implemented differently in order to take this into account. Aside from that, This remains a long term issue (weeks) and update will not happen too soon.

2005.06.10

Worked on "Networks" note for EVB2. Markus wrote a first draft and I contributed some comments/corrections. The note is available here.

2005.06.01

Added network diagrams showing the various networks used by the new EVB: private Gigabit Ethernet (online), private Gigabit Ethernet (Test Stand), private Fast Ethernet Boot Network, and connections to the outside world (Fnal "Public" Online Network). These currently reside on b0pcmitxx, but will probably end up elsewhere when they're moved around to be put into this or that note/webpage.

2005.03.01

After about 3 weeks of familiarisation, a first draft of a new EVB documentation is available. It is incomplete at the moment, but was a good way to start learning about the EVB upgrade. So far, only the note for Aces has been drafted.

2005.06.15

From the elog:

Level3 was paged but in fact it was CSL that was giving trouble.
The level3 display was green (waiting to output) and consumer csl monitor was red (presumably not working).
After we paged CSL and they restarted it.
things are back to normal.

2005.06.15

First week of pager starting yesterday. Got a page this morning while aces were running L2 torture: Relay failure. Here is the text from the elog:

Walked into CR as L3 problems were appearing. Relay Server troubles on converter 13. L3 proxy got stuck in trying to recover. Eventually, rebooted c13, cleaned up all L3 processes including relay servers, and back to running.

---khaldoun and arkadiy

2005.06.07

Seems that most issues on the list are ok for now. Ilya strongly recommends reading the L3 Expert Manual found here.

2005.06.04

List of recent L3 pager problems found here.

2005.05.26

Made major changes to script. Kept old one as qsub_fcdflnx_tar since it works just fine for a small tar file (<2GB). New script is called qsub_fcdflnx_scp, and uses a second script to copy things over back from the CAF, qsub_copy_over. Yes, this is a horrible kludge, but after hours of fighting with cshell and then with bash, i just couldn't resolve everything under one roof. So if the current solution works (testing now), I'd be in favor of using it regardless of elegance.

2005.05.25

Previous problem fixed (been meaning to ask what it was... will update when I know). Now we would like to have the script scp the release_dir to the caf instead of tarring it (2GB limit is getting in the way).

2005.04.30

Bruce seems to have found the reason for the problem he was having. From his email:

From your output it looks like I need a directory on fcdfdata051.fnal.gov:
Segment number from 1 to 1
Initial command: ./tmp_caf_20005.tcsh
Output file :
khaldoun@fcdfdata051.fnal.gov:/cdf/scratch/khaldoun/icaf/20005.$.tgz
Submitted : Wed Apr 27 23:17:48 2005
Ended : Wed Apr 27 23:20:10 2005
I seem to not have one:

So we need directories there for $USER == klute, gchouda, and knuteson.

2005.04.26

Latest attempt: success. Perhaps we have different versions of the script?

2005.04.20

Email from Bruce regarding discrepancy in success status between his running of the script and mine:

[...]
Now I execute
cd fcdflnx4:/cdf/scratch/knuteson/cvs/current/
./bin/qsub "ls > yo.txt"

I do not get a yo.txt file, but instead get an email such as the one
attached.
Does qsub "ls > yo.txt" work for you?
Thanks,
[...]


Still need to check this to verify that I can either reproduce his error, or explain why I do not get it...

2005.04.16

qsub_fcdflnx was written to automate the submission of jobs to the main CDF CAF from the fcdflnx nodes. The goal here is to recreate bit by bit the environment we have in our release directory, with data files in certain places, and executables in others, without worrying about the fact that this will run elsewhere. Recreating the directory tree is the main goal. This script was written this summer (summer 2004 version), and brought to near-completion. Other things distracted me, and so I get back to it now to see if we can bring it to completion.

After dealing with several issues related to an old CAF configuration files (grabbed latest .cafrc files from Guillelmo), I now test the script. As far as I have seen, this fulfills the desired purpose. Specifically, it copies the directory structure of the release directory, sends it to the caf, removes broken links and replace with dirs, executes the command, then copies new files back to the release dir on fcdflnx. Emailed Bruce to ask what goes wrong when he executes this.

2005.04.13

Problem: we want to submit jobs (i.e. a quaero analysis) without worrying about computing power available. So we want to use various computer farms where we have accounts. However, we'd like to avoid having to store files in different places, change configurations, environments...

Solution: we wrap the executable in a qsub_nodename script, where nodename is in this particular case fcdflnx. We use this script to automate all adjestments that must be made in order to submit the job on that platform.

2005.05.13

Done. Useful Java classes: JDialog, JPanel, JTextField... Basically, this involved a reworking of the code for logging into the Hardware Database, which was already available in the ACP. Next: final testing and commissioning. However, it would be nice to have this all in CVS first. Time to bug Markus.

2005.05.10

Ilya has asked me whether I could incorporate into the Ace Control Panel a button/option to kill a single L3 node. This is useful if there is a high rate of reformatter errors from one particular node, which has been happening rather often lately. Until now, the solution has been to pop up a window in the main RC that tells the Ace which node has been problematic and instructs him/her to page L3. We can bypass this page if we put this funtion in the Control Panel.

2005.05.05

The new control panel has gone in. Wrap up consists of: putting everything in CVS, adding 0/1 return value to Util.doCommand, and hoping it all goes well.

2005.05.04

The list below was taken care of point by point. It turns out that rgang is perfectly capable of doing the ssh into all ~250 nodes in parallel. The error I saw must have been something else. With a little bit of pain and help from Ron, rgang was installed on pcom2.

Finally, it's worth noting that the installation happened twice b/c of an interesting glitch: certain disks we thought were nfs-mounted (and thus accessible from both pcom1 and 2) turned out to be separate replicas on the two gateways. The code actually resides primarily on pcom1 and gets copied 1/week by a cron job to pcom2. so to install software properly on pcom2, you have to make sure that you make changes on pcom1 so that your changes are not erased when code is copied over.

2005.04.30

Test results:

1. Need to figure out how to ssh into all the nodes at once, or parallelise better. Currently, rgang fails trying all of them at once.
2. Need to install rgang on pcom2 so that l3_proxy_ops can run my script properly.
3. Ask Ilya if I can replace the l3_proxy_ops with my version of the script in the level3 directory since it should not interfere with anything and should allow me to continue testing even during data-taking.

2005.04.30

Setting up the test (can be done before):

• In checkL3farm.sh, change from using test nodelist to actual latest nodelist (optional).
• Switch EventBuilder.java code to use fully integrated button with l3_proxy_ops being called.

During the test:

• Change evproxy's .cshrc file so that it points to my private version of the control panel.
• Change l3proxy's .env.better script, so that $LEVEL3_DIR points to my private ~/level3/control/ directory and picks up my version of l3_proxy_ops (to include new code for $unk==--farm-check case).
• Press the button!
• Toggle rshto from 1 to 5 and back to force/unforce failed check.
• Go through above steps and "undo" each.

2005.04.28

Second order pass:

• Wrote one version of the if() then sequence of checking the farm for testing from $USER == khaldoun when performing tests on my private version of the Ace control panel.
• Wrote another version for full integration which relies on l3_proxy_ops and running under $USER == evbproxy.
• Implemented a confirm dialog box in case the checkL3Farm script indicates some nodes are down so that the Ace can choose either to clean up anyway or cancel and call the expert (Thanks Alberto & Arkadiy!).

That pretty much takes care of integration into the java panel, and matches the action sequence delineated here.

Still unknown:

• How to grab the last line of the JTextArea in the java panel (a preferred way to pass that boolean for seeing whether the test ran fine. The current method relies on lineCount(), and though it works, it's... well... ugly)
• How many nodes you can ssh into at one time -- i.e. the degree of parallelisation you can use in rgang.

The next task is to incorporate this more fully into the java panel. The series of actions we want:

• Ace presses "Clean L3" button
• We pass the "farm-clean" string to inetd
• The l3_proxy_ops script checks farm first
• If all nodes are up, clean farm
• If there is problem, issue warning

The current trouble is that the checkL3Farm script outputs to stdout to tell the Ace if something is wrong. What we'd like is to pass some boolean in the java code in order to test for this condition.

2005.04.21

First order pass, now using rgang: Markus kills two birds with one stone, and shows me a great tool, written by Ron Rechenmacher. rgang is a Python program that has the ability to execute a script/command in parallel, branching at several levels, and even copying itself to nodes that do not have it installed. The cherry on top? It allows you to control timeout for the ssh/rsh.

It's obvious that this has enormous utility for us. With this knowledge, and a little bit of trial and error, I have now a first order pass at this script. It is linked here for archival purposes. It is a shell script that should run under l3proxy and that is called when you want to clean the farm (i.e. as part of l3_proxy_ops). This has been tested and shown to work.

2005.04.20

A thorough talk with Ilya has clarified many aspects of this project. The primary conclusion is that the zeroth order pass explained in previous entries lacks important funcionality and goes through the wrong channels.

In terms of functionality, the primary missing piece is key to the success of this script. Due to the low frequency of problems when cleaning up L3, we must execution time into careful consideration. Rephrasing Ilya's formulation, if we lose 40 minutes every time we get stuck cleaning up, and we get stuck only 1 out of 20 times that we clean up the computer farm, then the script does not have any advantage if it takes 2 minutes to run and it is run each time we do a clean up. The numbers used in this example are pretty representative, so let's set a goal of about 30 seconds for execution time. This requires that we be able to control the ssh timeout, something which is not easily doable in shell, but is effortless in expect. Nothing is for free though, since I have never used expect and would prefer not to learn a new language unless it is truly time-efficient to do so.