Try this:

SELECT test1
scan
   c3   = ALLTRIM(LEFT(examcode ,5))
   replace nmonth WITH Rep1(c3)
ENDSCAN

FUNCTION Rep1
   PARAMETERS cMes
   LOCAL cRes
   cRes = 0
   DO case 
      CASE UPPER(cMes) = "NOV02" 
           cRes = 11 
      CASE UPPER(cMes) = "OCT02" 
           cRes = 10 
   ENDCASE

RETURN cRes

When You define cRes as public VFP automaticly creates that variable as bolean & cRes = .f.
If cRes is not assigned to numeric value (program not go in any CASE) cRes is still bolean. Therefor You Get Data Type Missmatch.