There will come a time when all your problem solving skills will hit a brick wall. These are some simple but effective tips to help kick start the process whether you are debugging, optimizing performance or whatever it is you're working on.
Scribble It
I'm a firm believer that nobody can hold everything in their head, this is why I recommend using a whiteboard or notepad to scribble out ideas, notes, lists and diagrams that can help your problem solving. Seeing everything laid out before you can also help give a clearer picture of a problem or reveal relationships you had not considered.
Discuss It
Vocally describing a problem to somebody else is a great way to spark ideas. The act of talking about it seems to allow your brain to follow new trains of thought and find new inspiration. It doesn't even matter if the other person fully understands what you're talking about. Inanimate objects don't really work for me, but if nobody else is available then even your cat or dog might help get you back on track.
Find Your Thinking Place
Everybody has a special thinking place where the ideas just seem to flow better. It doesn't really matter where it is, it only matters that you've identified it so that you can use it to your advantage. It could be walking to the local coffee shop, riding a bike, driving home, or walking the dog in the park. For me it's in the shower, I get all my best ideas there and actually write most of these posts in my head in the shower.
Sleep On It
Sometimes the best approach is to step away from a problem and leave it to background processing. If you've been thinking about something for a large part of the day, your subconcious will keep trying to make sense of it while you sleep. I couldn't count the number of times the answer has suddenly come to me at 3AM.