Instability of Actor-Critic Algorithms diagnosed by DPO

Instability of Actor-Critic Algorithms diagnosed by DPO