In the current high efficiency video coding development, each LCU has its own quantization parameter (QP) and the difference between a current QP and a reference QP is transmitted instead of the QP value itself. Since the LCU is much larger than the macroblock of AVC/H.264, using one delta QP per LCU may cause rate control unable to adapt to the bitrate quickly enough. Consequently there is a need to adopt delta QP in units smaller than LCU to provide more granular rate control. Furthermore, it is desirable to develop a system that is capable of facilitating more flexible delta QP. Accordingly, adaptive quantization parameter processing is disclosed where a QP minimum CU size for incorporating or deriving a delta QP is indicated selectively in the slice level or in the sequence/picture level. In one variation, the delta QP is incorporated or derived only if nonzero quantized transform coefficients exist in a leaf CU that is larger than the QP minimum CU size. In another variation, quantization parameter information is only incorporated or derived when a leaf CU has any nonzero quantized transform coefficient. In yet another variation, a flag is used to indicate whether one delta QP per LCU is transmitted if the LCU has any nonzero quantized transform coefficient or one delta QP per leaf CU is transmitted if the leaf CU has any nonzero quantized transform coefficient.