Do portlets inherently make a site less accessible?

No, portlets aren't inherently inaccessible. They can be inacessible or they can be an improvement to accessibility. There are techniques and strategies for implementing accessible portlets and portals.

Portlets are not inherently inaccessible.

Portlets have an opportunity to have poor usability and accessibility. It is the nature of portals that there are potentially multiple portlets on the screen at a time. If this is not coordinated well, these portlets may have significantly different UIs and conventions, may not attend to good accessibility development practices (e.g. well-defined tab order for form fields), or may even have conflicts in their attempts at accessibility (e.g. conflicting desires to use keyboard shortcuts for navigating to form fields.)

Portlets can be an improvement to accessibility. Portlets, especially standards-compliant portlets, often implement simplified, bite-sized, easily understood and navigated UIs that represent aspects of, portions of, or views on larger, more complex processes and applications. Some users may find the self-service portlet surfacing a portion of an application's functionality and data more usable than interacting with the full application.

Portlets can inherit some of the UI features of their portals, including the option of skins that are more usable or even deliberately higher-contrast.

Portlets can be written with more or less usability in mind. For instance, the Fluid Infusion components are intended to be particularly usable and accessible. Portlets that are written to make use of these sorts of UI components may be more usable and accessible than portlets that make use of other tooling.

PS: Shameless plug: Unicon offers portlet development consulting services. Consider involving Unicon in helping you make portlets that are more usable and accessible, perhaps even making use of Fluid Infusion.