- If you are selecting a WCG calibration in the display, whether DCI-P3 or BT2020, you need to check "Report BT2020" in MadVR's calibration tab so that the correct colorspace is selected in the projector. This has nothing to do with the calibration software. This is necessary for me only when outputting 12bits, not when outputting 8bits, at least with the driver I use currently (385.28). If you don't do this, you end up with the kind of saturations you posted.
- You need to match the calibration you are using in madVR with the calibration you are selecting on the PJ. So if you tell madVR to use a DCI-P3 LUT, you need to select a DCI-P3 calibration on the PJ. If you tell it to use a BT2020 LUT, you have to select a BT2020 calibration in the PJ. I highly recommend not to use BT2020 if you use a 3D LUT, as this tends to cause a lot of posterization. You want to select a DCI-P3 calibration and create a DCI-P3 LUT with a gamma 2.2 target. In that case, do not select a BT2020 LUT in madVR, or madVR will select the largest LUT available when playing HDR content.
- You NEVER use HDR patterns to calibrate or verify a calibration for madVR's tonemapping. You only use SDR DCI-P3 in your software, with a gamma target of 2.2 if using a LUT. The gamma target of the display doesn't matter. If you use a 3D LUT, give it a 2.2 target, if you don't use a LUT tell madVR to which gamma the display is calibrated. It will always end up as PQ when playing HDR content with the tonemapping.
Please don't ask any more question on this here. As I said a few times, I will start a thread when the next public build of madVR is released. The reason why I'm waiting for this is because 1) I don't have much time at the moment and 2) although calibration won't change, the HDR settings will change and I want to be able to test the final settings before making any recommendation.
There are a couple of threads offering general support on madVR, that's where to post in the meantime.
Good luck