During full-hole core drilling, wire-line coring compound drill string is easy to have scratch, indentation and wear marks. Based on nonlinear finite element method and using three-dimensional pipe unit, two composite sections of compound drill string were analyzed about its dynamics behavior including whirl orbit, whirl speed, lateral vibration and longitudinal vibration. The results showed that with the increase of rotary speed and WOB, the collision time of the drill string and the well wall significantly increased; under the same WOB, along with the increase of the rotary speed, the collision probability of drilling string and well wall increased and the contact trajectory became lengthened, which led to more sliding friction between the drill string and the well wall; with the increase of WOB and rotary speed, the mean value of whirl speed increased, the effect of rotary speed was larger than WOB; the increase of rotary speed would increase the lateral oscillation frequency of drill string, the increase of WOB would increase the longitudinal oscillation amplitude of drill string cross section.