Video Processing Issue and Common Fixes

This document outlines the steps and checks to follow when troubleshooting video processing issues.

!!! failure For many of these things you must have access to the prod configuration in the Railbird backend repo! You can check by running cat $(echo $RB_SECRET_PROD_TOML) in that repo and ensuring the result is nonempty.

Common Quick Fixes

Debugging a Failed Processing

1. Hatchet Task Check

Verify whether a Hatchet task (workflow run in their terminology) was created for the video by visiting:

hatchet.railbird.ai

2. Worker Check

Ensure the following workers are running:

VideoProcessing
Encoding/Playlist Updating

You can request a login via Slack or use the credentials available at: vaultwarden.railbird.ai

3. Evaluate Video State in Railbird Ipython

To evaluate the state of the video, follow these steps:

Start a session in Railbird Ipython:
```
railbird ipython --prod
```
Load the video:
```
video = load.video(id=[your video id])
```
Check if there is current processing:
```
video.current_processing
```

Review the current state and state history:

video.current_processing.status
video.current_processing.statuses

If you find a failed state, you can typically find the traceback that caused it by looking at the error_traceback on the failed status
```
video.current_processing.statuses[-1].error_traceback
```
or
```
print(video.current_processing.statuses[-1].error_traceback)
```

for a human readable version.

4. Check Logs

For further troubleshooting, check the logs: Logs

The following presets may be useful:

Video id in logs field pane (Can be used to filter to a specific video)
ProcessVideo Filter to ProcessVideo only
playlist-updater For fragmented mp4/ios
encoder for chunked mp4/android (although this is less important and technically orthogonal to processing)

Reenqueue/reprocess a video

In the Railbird backend repo using just prod railbird hatchet process <video_id>. You can check Hatchet to ensure its there as described below.