....well, this may be a reason Handbrake IS required! First, if you haven't already tried it I would try a short project using each of the Sony XAVC/XAVC S formats (XAVC Intra, XAVC Long, and XAVC S Long) as well as each of the Sony MXF formats (HDCAM SR, and HDCAM SR Lite, and HDCAM SR 444), to determine if any are compatible with MKV ToolNix and is the quality satisfactory. If so, that would then eliminate the requirement for Handbrake.
If not, then I would assure, if you haven't already, that within Handbrake you are using the highest quality setting within Handbrake to render the MKV.
BTW: in your reply you state "Only thing is now, I got some problems with importing .MKF files into MKV ToolNix". Is the MKF a typo and you meant to say MXF?
Although I haven't made a comparison of file sizes MXF -vs- MKV, I would assume that MKV would be smaller and more compressed, although the viewer quality (with the right settings) could appear to be of equal quality, as MKV is a pure delivery format intended to be a smaller size as is for example most mp4 (h.264) files compared to avi files (which are normally an acquisition format rather than delivery format).