>It looks like this was supposed to be fixed in SP1, "Indexes on STR() are penalized during Rushmore optimization if SET ANSI is OFF". Unfortunately, we're still seeing the same problem. We've got a 400MB table that has this index:
<snip>

No, it's different problem. In your case it's related to SYS(11) function which is not Rushmore optimizable for whatever reason anymore. You can see that if you replace it with DTOS()

CLEAR 
SET DELETE OFF
SYS(3054, 11)
CREATE TABLE c:\testopt (cust_num C(8), trans_date D)
INDEX ON STR(VAL(CUST_NUM),8) + DTOS(TRANS_DATE)  TAG custhist2
SELECT * ;
  FROM testopt;
WHERE STR(VAL(CUST_NUM),8) + DTOS(TRANS_DATE)   = "  633621" ;
INTO CURSOR Dummy