I actually enjoy it. I sit at a desk all day, so digging in the dirt is actually fun for me.
Our first year in our house I cut down eight 70-year-old ewe trees. The roots for each of them weighed about 100 lbs each, and I dug them out by hand. Last year I moved 8 cubic yards of new topsoil to create new planting beds - by wheelbarrow, and also removed a raised island garden. I had to rent a bobcat to take care of that.
Fun stuff. This year we're expanding our beds, so I'll probably need another 10 cubic yards of top soil. Planting is the fun part.