Removing Reference from Item

I am trying to remove a reference from an Item. My code works fine when I add a Reference to the Item through code.

But when I add a reference to an Item manually through the Item Explorer, I am not able to Remove this Reference. Any Ideas?

    Private Function Item_RemoveFile(ByRef Item As IEdmItem, FileName As String, FileID As Integer) As String

            Dim RemoveRef(0) As EdmItemRef

            Dim refFlags As Integer = EdmRefFlags.EdmRef_File

            Dim ExistingRefs() As EdmItemRef = Nothing

            ' Get Existing References

            Item.GetReferences(refFlags, ExistingRefs)

            For Each ExistingRef As EdmItemRef In ExistingRefs

                If ExistingRef.moNamePathOrID = FileID Then

                  

                     ' Found Reference

                    RemoveRef(0) = ExistingRef

                    ' Remove Reference

                    Item.UpdateReferences(Nothing, RemoveRef)

                    Item.Refresh()

                    ' This Function Checks the Item Back In

                     Update_ItemStatus(Item, True, "Removed File Reference")

                    Return ""

                End If

            Next ExistingRef

SolidworksSolidworks Pdm enterprise Pdm