>Also
>
>I never used NEXT in VFP, but I kept it there because it does not return error message
>Also, there is not ENDFOR in the code, but I think problem is not NESTING error, because
>I get 9 chars of the final string OK, only the last 3 are bad

Try this:

strBookID = "456789123" + "123"
strUserName = "ABC" + SPACE(9)
strBookTitle = "XYZ"  + SPACE(9)


strRegKey=""
strSecretEdit = "xxxxxxxxxxxxx"

strName = ""

*For each of the 12 letters in the input strings
FOR idx3 = 1 TO 12
	IF strBookID = "123" THEN
		ascInStrBookID = 1
	ELSE
		ascInStrBookID = ASC(SUBST(strBookID,idx3,1))
	ENDIF

	ascInStrUserName = ASC(SUBST(strUserName,idx3,1))
	ascInStrBookTitle = ASC(SUBST(strBookTitle,idx3,1))
	ascInStrSecretEdit = ASC(SUBST(strSecretEdit,idx3,1))

	ascKeyChar1 = BITXOR(ascInStrUserName , ascInStrBookTitle , ascInStrSecretEdit , ascInStrBookID)

	IF ascKeyChar1 = 0 THEN
		ascKeyChar1 = 255
	ENDIF

	strName = strName + CHR(ascKeyChar1)
NEXT
FOR idx4 = 1 TO 12
	DO CASE
	CASE idx4 = 1
		intAdd = 6
	CASE idx4 = 2
		intAdd = 14
	CASE idx4 = 3
		intAdd = 9
	CASE idx4 = 4
		intAdd = 15
	CASE idx4 = 5
		intAdd = 3
	CASE idx4 = 6
		intAdd = 5
	CASE idx4 = 7
		intAdd = 1
	CASE idx4 = 8
		intAdd = 18
	CASE idx4 = 9
		intAdd = 13
	CASE idx4 = 10
		intAdd = 8
	CASE idx4 = 11
		intAdd = 12
	CASE idx4 = 12
		intAdd = 19
	ENDCASE

	ascKeyChar2 = ASC(SUBS(strName,idx4,1)) + intAdd
	DO WHILE ascKeyChar2 >  90
		ascKeyChar2 = (ascKeyChar2-25)
		LOOP
	ENDDO

	DO WHILE ascKeyChar2 < 65
		ascKeyChar2 = (ascKeyChar2+25)
		LOOP
	ENDDO
	strRegKey = strRegKey + CHR(ascKeyChar2)
NEXT

You had a leading space in your declaration of strSecretEdit, strName should be initialized to an empty string - and I was wrong about the DO WHILE test value. The sample values above gave me the same result in VBSCRIPT. You could also shorten it:

strBookID = "456789123" + "123"
strUserName = "ABC" + SPACE(9)
strBookTitle = "XYZ"  + SPACE(9)

strRegKey=""
strSecretEdit = "xxxxxxxxxxxxx"

strName = ""
slookup = CHR(6)+CHR(14)+CHR(9)+CHR(15)+CHR(3)+CHR(5)+CHR(1)+CHR(18)+CHR(13)+CHR(8)+CHR(12)+CHR(19)

FOR i = 1 TO 12
	IF strBookID = "123" THEN
		ascInStrBookID = 1
	ELSE
		ascInStrBookID = ASC(SUBST(strBookID,i,1))
	ENDIF

	ascKeyChar1 = BITXOR(ASC(SUBST(strUserName,i,1)) ,ASC(SUBST(strBookTitle,i,1)) ,ASC(SUBST(strSecretEdit,i,1)) , ;
		 IIF(strBookID = "123" ,1,ASC(SUBST(strBookID,i,1))))

	IF ascKeyChar1 = 0 THEN
		ascKeyChar1 = 255
	ENDIF

	ascKeyChar1 = ascKeyChar1 + ASC(SUBSTR(slookup,i,1))
	DO WHILE ascKeyChar1 >  90
		ascKeyChar1 = (ascKeyChar1-25)
	ENDDO
	DO WHILE ascKeyChar1 < 65
		ascKeyChar1 = (ascKeyChar1+25)
	ENDDO

	strRegKey = strRegKey + CHR(ascKeyChar1)
NEXT

HTH,
Viv