No. Performance is not degraded by widgets that are not in use on a page.
Elementor does not load widget-specific assets in pages where the widget is not in use, except for a small amount of basic CSS and javascript that should not be fragmented. Likewise, Elementor does not load assets for fonts or icons on pages where they are not in use. Thus, there is no need to disable any native Elementor widgets, fonts or icons as they will only load their assets if they are actually in use on a page.
Some third-party plugins may allow disabling of their widgets, but that is a different scenario, as they may require actively disabling to prevent them from loading assets even when not in use. Elementor, however, does not have this issue, therefore, no disabling is needed.
In summary, Elementor eliminates unnecessary asset-loading bloat, allowing your site to load faster, by only loading those assets necessary for the specific page being loaded.