Efficient recording settings for OBS and x264

x264 is very capable when set up correctly, but is a bit unintuitive to tune. This should be a good starting point. It’s nothing ground-breaking, but I get asked about this a lot and want to put it somewhere more public as a reference.

This is one way to get good results, not the only way. x264 tuning is at times more of an art than a science.

This is mostly about recording, because that’s what I’m more familiar with. I’ve put in some notes on streaming where I know the differences, but I’m likely missing some subtleties in that area.

x264 versus NVENC

NVENC (Nvidia’s hardware encoder) is mostly the gold standard in this space, and x264 is the runner-up. Here are some reasons you might want to use one or the other:

• NVENC tends to produce somewhat higher quality results at settings matched to produce the same file sizes, or somewhat smaller files at settings matched to produce the same quality. This gap gets bigger as you tune x264 to be more efficient with CPU time.

• x264 is extremely tunable, which may be able to close the performance gap to NVENC for more specific scenarios. (That’s one reason I’m writing this: x264 is more dependent on appropriate tuning to get good results.)

• NVENC is hardware, and if you don’t have that hardware (as part of an Nvidia GPU) it isn’t an option. x264 is software and can theoretically run on pretty much anything, but its capabilities depend a lot on how fast your CPU is.

• Recording a game with x264 slows down the game’s CPU-side work, where NVENC slows down the game’s GPU-side work. x264 may slow things down less if your game is GPU-bound and NVENC will probably slow things down less if your game is CPU-bound.

Other encoders

AMD’s VCN hardware encoder isn’t very good. You’ll probably have a better time with x264 instead unless you have a very weak CPU.

Intel’s QuickSync hardware encoder is the one I’m least familiar with. The most recent iterations are purportedly excellent.

OBS ships with two more software encoders: SVT-AV1 and AOM AV1. They don’t seem all that effective for game recording or streaming right now (I think memory bandwidth contention with games is the issue), but they are in active development and may get better at that.

x264 settings

Your encoder settings in the output tab should look something like the image below. You’ll have to go through the settings from top to bottom, as several of them affect which options are displayed further down the page.

CRF

This is the main way to control quality. Lower numbers look better but take more disk space, while higher numbers look worse but take less disk space. This setting targets a quality instead of a bitrate, so it keeps CPU load low and file sizes small when there’s not a whole lot of detail or motion to encode, and steps up as far as it needs to to keep things looking nice when the scenes get complex and fast. Any 6 point change to CRF approximately doubles or halves file sizes. The same CRF numbers are good regardless of your recording resolution and framerate, unlike bitrate controls.

16 to 24 is a good range. The quality improvements from going below 16 are tough to distinguish, and the quality problems from going above 24 get tough to ignore. Up to 30 or so is still reasonable (though ugly). If a CRF of 24+ seems like the thing to do, consider lowering your recording resolution instead. Reducing resolution gives larger performance improvements than increasing CRF.

Bitrate

This might be better named maximum bitrate. It’s here to patch over a specific problem with CRF control: with only CRF, if the scene being recorded is sufficiently complex and fast, the encoder can easily try for more detail than your CPU can handle and end up dropping frames. This puts a cap on that, so that when your CPU wouldn’t be able to keep up the recording’s quality drops and it stays smooth. Setting this to avoid dropped frames with minimum quality loss takes a bit of experimentation specific to the computer, the other recording settings, and the game being recorded.

Ideally the bitrate is mostly controlled by the CRF, not this maximum, and so the maximum doesn’t have much effect on file sizes and in-game framerates. The lower you set this bitrate and the lower you set CRF, the less this holds true.

If you expect a later re-encoding (like Youtube’s process) to badly damage your recording’s quality anyway, it will probably do it in more of a bitrate-limited pattern than a CRF-limited pattern, and you can probably save some space in the meantime by setting bitrate much lower. This is a more obvious quality penalty in the initial recording, but it may not be a noticeable difference in the final re-encoded product.

For streaming, bitrate will need to be set much lower, to match your internet connection and whatever your streaming service accepts.

CPU Usage Preset

This lets you trade off between bitrate efficiency and CPU time efficiency. If you’re having trouble getting the recording quality you want without dropping frames or in-game performance is taking too much of a hit, try ultrafast. If the performance is fine but the file sizes are uncomfortable, try veryfast. Settings slower than veryfast are rarely useful for recording.

Since streaming needs to fit decent quality into a fixed low bitrate budget, veryfast is a better default for streaming, and even slower settings than that may sometimes be good.

me and subme

When not manually set these are a part of the CPU usage preset. They seem to be particularly effective ways to spend a bit of extra CPU time when encoding and gaming on the same system (I’m guessing that this is because they don’t have as much potential to cause cache and memory bandwidth contention).

The defaults are:

I recommend:

qcomp

This affects how bursty the bitrate can be. The default of 0.6 is a bit low for recording gameplay, and fast-moving scenes look noticeably more solid at 0.7. Streaming’s bitrate consistency requirements push it the other direction, maybe to 0.5.

nr

This is noise reduction, useful for reducing bitrate when CRF-limited or reducing artifacting when bitrate-limited. It has little effect on performance in either direction. Since it purposefully removes some detail, the results won’t look quite as true to the source material when using it, even if they’re nicer to look at overall. If you want to use it, try 200-400 to keep the effect very subtle or 1000 for more aggressive compression.

Recording Format

mkv and mp4 are both good. mkv has some technical advantages, but mp4 support is nearly universal, while some programs don’t know what to do with an mkv file. For instance, Discord can play back mp4s directly but not mkvs.

General OBS settings

Set your output resolution and framerate in the video tab. Reduced resolution (like 720p) with a low CRF can look good enough and perform very well (better than a higher resolution and higher CRF), but Youtube’s re-encoding pass is very stingy with bitrate for anything below 1080p, so that might not stay a win once it’s published.

If a game is the only source that will be visible in the recording, the base resolution should match the output resolution (and Rescale Output over in the encoder settings should be disabled as well). Reducing the resolution as early as possible in the process saves the GPU a bit of work, which means if you need to it’s best to do it in a scaling filter applied to the source:

There’s one more gotcha in this setup. Not all capture methods are created equal. Game capture usually works better than window capture or screen capture, but still isn’t bulletproof, especially on games running borderless. If you’ve got performance problems you can’t get rid of with encoder settings, it’s probably related to this. Make sure any games you’re trying to record are running in true fullscreen if possible, and experiment with different capture settings in OBS if nothing else works.