Written by Eric Albert
First, thanks to Nat Hellerstein and, especially, Andy Latto for their math insights and suggestions for this puzzle.
This is a hard puzzle to solve mathematically. It probably makes much more sense to write a straightforward computer program that tries every possibility. This entails going through a range of possible numbers (1 to 2000, say) and, treating each number in succession as the "base number," calculating its average expected score. The running time of such a program is under a second.
You calculate the average expected score by looking at the score you'd get for that base number for each of the 216 possible dice rolls (216 = 6 x 6 x 6), then summing all of the scores, and dividing the resulting sum by 216. You need to remember to make a special case of the dice roll "6,6,6" when you're calculating for yourself (but not for the Devil).
Once you've got the average expected score for each of the base numbers, you just sort all these scores to find out the best base number. It turns out that this number is 644. You then read off its expected score, which you've already calculated, which is 69 for each round, on average.
The devil always chooses 666. His expected score is 69 and 17/36 each round.
Then, following the instructions:
644 [reverse] --> 446
[square] --> 198,916
102 [reverse] --> 201 [square] --> 40,401
69 [reverse] --> 96 [square] --> 9,216 [first two digits] --> 92
The resulting digits, in order are:
1 9 8 9 1 6 4 0 4 0 1 9 2
When these digits are added to the given numbers, the results are:
9 14 19 20 18 21 13 5 14 20 1 12 19
which is just the alphabetical positions of the letters of the answer:
I N S T R U M E N T A L S