Fix ClassDB not checking for editor classes properly

[mihe]

This fixes the _instantiate_internal, can_instantiate and is_virtual methods of ClassDB to not only check whether ClassInfo::api is API_EDITOR but also whether it's API_EDITOR_EXTENSION.

This bug led to a crash when running @qarmin's fuzzer on Godot Jolt, as seen in godot-jolt/godot-jolt#868, as it tried to instantiate an editor-only type that erroneously reported a successful can_instantiate.

@dsnopek You wouldn't happen to know if this needs the same change? I couldn't quite figure out what it's for:

godot/core/object/class_db.cpp

Line 592 in b7feebe

placeholder_extension->editor_class = ti->api == API_EDITOR;

[adamscott]

@mihe @dsnopek What's weird is that it's assigning a boolean as the value instead of the enum value.

godot/core/object/class_db.cpp

Line 592 in b7feebe

placeholder_extension->editor_class = ti->api == API_EDITOR;

[dsnopek]

@dsnopek You wouldn't happen to know if this needs the same change?

No, I don't think it should need it there. That's inside a block that will only execute if the class isn't an extension class.

[dsnopek]

Thanks! Looks good to me :-)

[dsnopek]

@dsnopek What's weird is that it's assigning a boolean as the value instead of the enum value.

Yeah, that one is a boolean. This code is making a "fake" GDExtension class to represent a placeholder for a runtime class, and ObjectGDExtension::editor_class is a boolean.

[akien-mga]

Thanks!