REQUIREMENTS

First things first, you need a good cpu for 1080p x265 hdr encoding and a beast of cpu for 4k x265 hdr encoding, otherwise you'll have to wait days for an encode.
I'd recommand for 1080p at least an ryzen 1600 and for 4k at least i9 9900k or even threadripper first gen.
And please don't encode nonHDR sources with x265, you'll waste your time if you do that, x264 is much faster but can't keep the hdr10 metadata, this is why we use x265 for hdr encoding, because x265 can do it.

DOWNLOAD

AviSynthPlus - AviSynth+ is a fork of the well-known frameserver featuring new functionality and optimizations not available in mainline. In the hope of modernizing AviSynth to meet present day needs and expectations, the project encourages everybody to help reach our goals: add long-sought features, energize the community, and improve developer and user friendliness. AviSynth+ is fully compatible with plugins and scripts written for AviSynth.
AvsP - AvsPmod is a text editor built for AviSynth, a video scripting language. AvsPmod is built on the discontinued AvsP project. Requires AVISYNTH installed.
x265 - I don't think this one needs any introduction, although download a version that your cpu is compatible with.
eac3to - eac3to is able to convert E-AC3, TrueHD, PCM and DTS-HD audio tracks to AC3, DTS or FLAC. Converts E-AC3, TrueHD, PCM, DTS-HD and FLAC tracks to AC3/DTS/FLAC. Removes Dialog Normalization from AC3, E-AC3, DTS and TrueHD tracks. Extracts AC3 frames from Blu-Ray AC3/TrueHD tracks. Extracts DTS core from DTS-HD tracks. Can remux the video track of a EVO/VOB source to Matroska(mkv). Command line tool but GUI(UsEac3to) also available. Requires external audio decoders.
MKVToolNix - MKVToolNix is a set of tools to create, alter, split, join and inspect Matroska files(mkv). With these tools one can get information about (mkvinfo) Matroska files, extract tracks/data from (mkvextract) Matroska files and create (mkvmerge) Matroska files from other media files.
FFMS - FFmpegSource (usually known as FFMS or FFMS2) is a cross-platform wrapper library around FFmpeg. It gives you an easy, convenient way to say "open and decompress this media file for me, I don't care how you do it" and get frame- and sample-accurate access (usually), without having to bother with the sometimes less than straightforward and less than perfectly documented FFmpeg API.
avs2pipemod - a CLI tool to pipe raw/y4m video and wav/extwav/raw audio to various audio/video encoders. It has other useful functions like benchmarking, saving pixel values to a text file, it can suggest x264 settings for Blu-ray disc encoding, and also has support for high bit depth video. avs2pipemod is the continued development of avs2pipe.
HDRTools_v0_4_2.7z - Avisynth HDR Tools plugin.
MediaInfo_GUI_18.12_Windows.exe - MediaInfo is a convenient unified display of the most relevant technical and tag data for video and audio files.

FIRST STEP

Run AviSynthPlus-MT-r2772.exe install it and don't forget to check both x86 and x64 versions.
Unzip AvsPmod 2.5.1.zip and save it anywhere you want, you can save it in Program Files for example.
At this point you can create a folder where you'll save the files for your encoding process, this way it's easier to keep track of the files.
Unzip the archive for x265, you'll only need x265.exe (64bit version) and save it in the encoding folder that I told you to create earlier.
Now create a new folder in D: for example where you'll store all the files for eac3to, unzip the eac3to archive in that folder.
In case you want to do matroska muxing, audio encoding/decoding with eac3to, you'll have to download&install eac3to Decoder Pack 1.4 + Haali Media Sp

Just install the mkvtoolnix anywhere you want, you don't need to do anything special with this one.
Unzip the ffms2 archive anywhere you want, grab the dlls from x64 and x86 folders and FFMS2.avsi, put them in avisynth+/plugins+ and avisynth+/plugins+64, ffms2 from x86 and ffms2.avsi go to plugins+ and ffms2.dll from x64 goes to plugins+64.
Unzip the avs2pipemod archive, grab avs2pipemod64.exe and store it in the encoding folder.
Unzip HDRTools archive and do the same thing like you done with ffms2, store HDRTools dlls into plugins+ and plugins+64 folders.
ColorMatrix - store this one in plugins+ too.

STEP TWO

eac3to:
Now you have to download a 4k hdr blu-ray disc from here
Demux the video from the disc, for that you have to use eac3to.
Encode the lossless audio track into a lossy track like dolby digital 5.1 640kbps
I'll skip explaining those steps because it's already explained here
Demux the video.hevc/h265 or mkv, all those containers can store the HDR metadata (doing this you just change the container, you don't encode it).
Save the video into the encoding folder.

Scripting:
Create a AvsP script.
Go to your encoding folder, right click, create txt file.
Open the txt with AvsP, to do that you have to select the path to your AvsP folder.
Now that you opened AvsP, load the video into it using ffvideosource.
The command is FFvideoSource("path\to\video.mkv"
Change .mkv with .hevc or .h265 if your container isn't mkv.
You have to wait a little bit not because ffvidesource is now indexing the video.
After the video its indexed you can crop it or resize it (depends on the disc aspect ratio and your needs).
If your video doesn't have black bars it means that you don't need to crop but you can resize it to 1920x1080 (in case you want to encode it fullhd and not 4k).
For 1080p resize use spline36Resize(1920,1080).
In case you your video has black bars, you'll have to cut them, to do that use Crop(left crop, top crop, right crop, bottom crop) and crop until you can't see the black bars anymore.
You have to change 'left crop, top crop' etc with numbers, like that (0, 20, 0, 20) and you can't use odd numbers.
And for 1080p resize you have to keep the aspect ratio, if your video has the 1:85 aspect ratio after you crop it your 1080p resolution will be 1920x1038, resize calculator from AvsP can help you with that, you can find it in Tools -> Resize calculator
Make sure you use these settings in resize calculator:
1.jpg

Also add selectrangeevery(floor(framecount()/30),300) in your script, this line is for selecting ~9000 frames from different scenes of the movie, you'll encode those 9000 frames.

Setting everything up to start encoding:
Create a .bat file in your encoding folder
In that bat file you'll have to add these settings:

start /low /b /wait avs2pipemod64.exe -y4mp nameofyourscript.avs | x265.exe --input - --y4m --crf 14 --preset slow --input-csp i420 --profile main10 --bframes 8 --level-idc 5.1 --high-tier --merange 32 --ctu 32 --input-depth 10 --rd 3 --tu-intra-depth 3 --aq-mode 1 --tu-inter-depth 3 --rdoq-level 1 --psy-rdoq 1.0 --aq-strength 0.80 --qcomp 0.60 --cbqpoffs -2 --crqpoffs -2 --vbv-bufsize 160000 --vbv-maxrate 160000 --pbratio 1.30 --ipratio 1.40 --subme 5 --me umh --weightb --rc-lookahead 40 --max-cll "9977,240" --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --range limited --chromaloc 2 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635, 16450)L(10000000,1)" --hdr-opt --repeat-headers --deblock -3:-3 --no-sao --psy-rd 1.0 --no-strong-intra-smoothing --stats "stats.log" --output "testing.mkv"

Ofc, you can tweak the settings a little bit and even add some more in case that you want your encode to be top notch, but for the beginning you can start to do some tests and maybe some encodes with those.
All the x265 commands are explained HERE
Now run a test, open cmd, type the path to your encoding folder and type the name of your bat file, the encode test should start right away.
Wait for those 9000 frames to encode and after that you'll compare the video source to your encode test.
To do that you'll have to open 2 tabs in AvsP, in first one you'll have the source with selectrangeevery(floor(framecount()/30),300) and in the second one just the your test, you can add FFinfo()in both tabs so you can have the OSD active.
Compare the source with the encode and if you notice easily major loss of details it means that you need more bitrate, if you don't notice any difference between the source and the encode it means that your test has too much bitrate.
To change the bitrate you'll use crf, a higher number means less bitrate and a lower number means more bitrate.
Run some more tests till you'll find a bitrate to suit you.
After you're satisfied with your tests, remove selectrangeevery(floor(framecount()/30),300) from your source script and save the script without it, run again the bat and you'll encode the whole source.

STEP THREE

Muxing everything together:
Here's not much to say, after your encode is finished, open mkvtoolnix.
Drag your encode, audio track, subtitles, chaper into mkvtoolnix and mux them.
To make sure that everything is muxed right, you can check that mediainfo.

Comparisons:
Open 2 tabs in AvsP, load the source and your encode with ffvideosource.
Add in both source and encode tabs:
ConvertBits(16)

ConvertYUVtoXYZ(Color=0, OutputMode=1, HDRMode=0, fullrange=false)

ConvertXYZ_Mobius_HDRtoSDR(exposure_X=17.0, peak_X=1.1)

ConvertXYZtoYUV(pColor=0)
That's for tonemaped because most trackers that allow x265 hdr encodes require tonemaped comparisons.
Also add FFInfo(cfrtime=false, vfrtime=false, colorspace=false, colorrange=false, cropping=false, sar=false, version=false, showprefix=false) for OSD.
To save screenshots go to Video -> Save screenshots as...
They have to be saved in png and the frame type has to be b for both source and encode or p for source and b for encode.

That's all, have fun.
Press REP and LIKE button for supporting more useful tutorials like this one. Thanks

good GPU can speed up encoding. ffmpeg has option to use it.

Good guide!