Context. Nucleation is a common phenomenon in all types of galaxies and at least 70% of them host nuclear star clusters (NSCs) in their centres. Many of the NSCs co-habit with supermassive black holes and follow similar scaling relations with host galaxy properties. Unlike black holes, NSCs, preserve the signature of their evolutionary path imprinted onto their kinematics and stellar populations. Thus their study provides us with important information about the formation of galactic nuclei. Aims. In this paper we explored the angular momentum of the nuclei of six intermediate mass (9.7 > log(M_dyn/M_⊙) > 10.6) early-type galaxies that host NSCs and are located in the Fornax cluster. Our goal was to derive a link between the nuclear angular momentum and the proposed formation scenarios of NSCs. Methods. We used adaptive optics assisted IFU observations with VLT/SINFONI to derive the spatially resolved stellar kinematics of the galaxy nuclei. We measured their specific stellar angular momenta λ_Re, and compared these with Milky Way globular clusters (GCs) and N-body simulations of NSC formation. Results. We found that all studied nuclei exhibit varied stellar kinematics. Their λ_Re and ellipticities are similar to Milky Way GCs. Five out of six galaxy nuclei are consistent with the λ_Re − ϵ_e of simulated NSCs embedded in a contaminating nuclear bulge that have formed via the in-spiralling and merging of GCs. Conclusion. It has previously been suggested that the NSCs in higher mass galaxies, such as those studied in this paper, form via dissipational sinking of gas onto the galactic nuclei with hints that some might also involve the merger of GCs. In this work we show that we cannot exclude the pure GC merging scenario as a viable path for the formation of NSCs.