There are multiple ways to manage this. If you can figure out a way to have a user provide a password for a keystore when the application starts up the most appropriate way would be to encrypt all the values using a key, and store this key in the keystore. The command line interface to the keystore is by using keytool. However JSE has APIs to programmatically access the keystore as well.
If you do not have an ability to have a user manually provide a password to the keystore on startup (say for a web application), one way to do it is to write an exceptionally complex obfuscation routine which can obfuscate the key and store it in a property file as well. Important things to remember is that the obfuscation and deobfuscation logic should be multi layered (could involve scrambling, encoding, introduction of spurious characters etc. etc.) and should itself have at least one key which could be hidden away in other classes in the application using non intuitive names. This is not a fully safe mechanism since someone with a decompiler and a fair amount of time and intelligence can still work around it but is the only one I know of which does not require you to break into native (ie. non easily decompilable) code.