R Cluster Agnes - Molecular fingerprint

Hi !

I'm currently using this protocol in order to cluster a dataset of molecules !! The criteria is a Fingerprint who looks like ECFP, such as a column as below :

786456
56486
4564
8748974
...

The problem, is that the clustering didnt provides good results. Is there any possibility to make a cluster from a binary fingerprint ?
Or is it possible to cluster from a similarity matrix ?

Best regards