With the development of technology and the diversification of information acquisition methods, video has become an important component of our daily life and work, and the security of video information is of increasing concern. In this work, a general video encryption algorithm is proposed and implemented. The algorithm extracts a video file frame by frame to obtain video frames and encrypts the video by encrypting its frames. The bilinear interpolation method is used to reduce the size of the video frames, and then one of the four fractional order chaotic systems and one of three scrambling methods are selected to scramble and diffuse the video frames. This work conducted in-depth feasibility testing on the encryption and decryption process, analyzed the resistance to statistical attacks, and completed tests on the histogram, correlation of adjacent pixels, and information entropy, as well as sensitivity analysis and noise resistance analysis. Moreover, the ability of encryption and restoration were compared with other methods. In addition, the performance comparison of three scrambling methods was also conducted. The encryption effect of these methods is not very different, but the restoration ability of the pixel scrambling is not as good as the other two methods. The experimental results show that this algorithm has better security and encryption efficiency.