>
>I'm using md5 to calculate a hash of a password.
>
>I start off with the hash of the password. Then, I do a series of hashes where each new hash is done with the previous hash + a substring of the password
>
>The number of additional hashes depends on (1) the password length and (2) the byte value of each of the bytes modulo a number
>
>If you don't know how I hash, how can you crack it - I wonder

One thought - if you create an account (or accounts) on a system you will be stealing the password file from, you will already know the password. So you'll be able to attempt all of the basic hashing schemes before realizing something else is happening. You could then create more accounts with various "basic" passwords to attempt to analyze the types of transformations that are occurring.

Honestly, even just doing something slightly differently would probably weed out 99% of the hackers, leaving only people who are really interested in cryptology (or who like a challenge) to continue the attempt.

If it were me, I'd probably decide that a direct approach isn't going to work. If I was able to compromise their password file/table, can I get a hold of the files the site actually runs on (ex. DLL's, assemblies, etc.) and then attempt to decompile or reverse-engineer it to determine what types of transforms are occurring? Seems like that would be much simpler.