{"id":12058,"date":"2022-11-11T12:12:21","date_gmt":"2022-11-11T17:12:21","guid":{"rendered":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/?p=12058"},"modified":"2026-04-14T23:57:27","modified_gmt":"2026-04-15T03:57:27","slug":"shared-manifold-learning-using-a-triplet-network-for-multiple-sensor-translation-and-fusion-with-missing-data","status":"publish","type":"post","link":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/2022\/11\/11\/shared-manifold-learning-using-a-triplet-network-for-multiple-sensor-translation-and-fusion-with-missing-data\/","title":{"rendered":"Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data"},"content":{"rendered":"

Abstract:<\/h2>\n

Heterogeneous data fusion can enhance the robustness and accuracy of an algorithm on a given task. However, due to the difference in various modalities, aligning the sensors and embedding their information into discriminative and compact representations is challenging. In this paper, we propose a Contrastive learning based MultiModal Alignment Network (CoMMANet) to align data from different sensors into a shared and discriminative manifold where class information is preserved. The proposed architecture uses a multimodal triplet autoencoder to cluster the latent space in such a way that samples of the same classes from each heterogeneous modality are mapped close to each other. Since all the modalities exist in a shared manifold, a unified classification framework is proposed. The resulting latent space representations are fused to perform more robust and accurate classification. In a missing sensor scenario, the latent space of one sensor is easily and efficiently predicted using another sensor’s latent space, thereby allowing sensor translation. We conducted extensive experiments on a manually labeled multimodal dataset containing hyperspectral data from AVIRIS-NG and NEON, and LiDAR (light detection and ranging) data from NEON. Lastly, the model is validated on two benchmark datasets: Berlin Dataset (hyperspectral and synthetic aperture radar) and MUUFL Gulfport Dataset (hyperspectral and LiDAR). A comparison made with other methods demonstrates the superiority of this method. We achieved a mean overall accuracy of 94.3% on the MUUFL dataset and the best overall accuracy of 71.26% on the Berlin dataset, which is better than other state-of-the-art approaches.<\/p>\n

Links:\"IEEE<\/a>\"arXiv<\/a>\"github\"<\/a><\/h2>\n

Citation:<\/h2>\n
A. Dutt, A. Zare and P. Gader, \"Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion With Missing Data,\" in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 9439-9456, 2022, doi: 10.1109\/JSTARS.2022.3217485.<\/code><\/pre>\n
@article{dutt2022shared,\n  title={Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data},\n  author={Dutt, Aditya and Zare, Alina and Gader, Paul},\n  journal={IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},\n  year={2022},\n  publisher={IEEE}\n}<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
Abstract: Heterogeneous data fusion can enhance the robustness and accuracy of an algorithm on a given task. However, due to the difference in various modalities, aligning the sensors and embedding their information into discriminative and compact representations is challenging. In this paper, we propose a Contrastive learning based MultiModal Alignment Network (CoMMANet) to align data […]<\/p>\n","protected":false},"author":28,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"single-templates\/single-sidebar-none.php","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"featured_post":"off","footnotes":"","_links_to":"","_links_to_target":""},"categories":[19,13],"tags":[161,177,243,485],"class_list":["post-12058","post","type-post","status-publish","format-standard","hentry","category-journal_paper","category-publication","tag-commanet","tag-contrastive-learning","tag-discriminative-manifold","tag-multimodal-triplet-autoencoder"],"acf":[],"_links":{"self":[{"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/posts\/12058","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/users\/28"}],"replies":[{"embeddable":true,"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/comments?post=12058"}],"version-history":[{"count":3,"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/posts\/12058\/revisions"}],"predecessor-version":[{"id":16531,"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/posts\/12058\/revisions\/16531"}],"wp:attachment":[{"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/media?parent=12058"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/categories?post=12058"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/faculty.eng.ufl.edu\/machine-learning\/wp-json\/wp\/v2\/tags?post=12058"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}