LoRAs and ControlNets are trained against a specific main model. They only work with checkpoints built on the same base model. In the Blender to ComfyUI SDXL image-to-image render pipeline that base is SDXL.
This rule is why the base model is picked by the size of its add-on ecosystem. See Choosing the base model by add-on ecosystem (SDXL vs Flux vs SD 3.5).