Clathrin-mediated endocytosis (CME) plays a central role in cellular homeostasis and is mediated by clathrin-coated pits (CCPs). Live-cell imaging has revealed a remarkable heterogeneity in CCP assembly kinetics, which can be used as an intrinsic source of mechanistic information on CCP regulation but also poses several major problems for unbiased analysis of CME dynamics. The backbone of unveiling the molecular control of CME is an imaging-based inventory of the full diversity of individual CCP behaviors, which requires detection and tracking of structural fiduciaries and regulatory proteins with an accuracy of >99.9%, despite very low signals. This level of confidence can only be achieved by combining appropriate imaging modalities with self-diagnostic computational algorithms for image analysis and data mining.