Inconsistent voltage readings on an outside light fixture?

Holy crap! That's the back of my house! Same brick! Color wire means nothing to electricity. Your box is not grounded or you didn't have good contact with the box. Yes, that will work. Use the 5.0 as the hot.