It can be expensive. It just depends on how diligent and patient you are on waiting on deals.
A simple gaming PC that can play most games on decent specs could start at around $600. A high end rig can go up towards $1800.
As for the building aspect, it can be easy as long as you don't rush. I would say the most difficult part would be placing the CPU into the motherboard. Once the motherboard is in place everything else just snaps in, really. If this is your first build, I recommend using a static strap around your wrist so that you don't short out any of your parts.
Overall, patience is the name of the game. You rush and you could be paying too much or having to dismantle your rig to see which part isn't working and why.
Oh, and order your stuff from Newegg.com exclusively.
I'm going to have to slightly disagree with that. At times, Amazon has better deals than newegg and usually I don't have to pay tax and/or shipping. When it comes to PC parts, shopping around helps save money.