Sanghyuk Chun (He/Him)

Sanghyuk Chun

profile

I am currently serving as a lead research scientist at ML Research team in NAVER AI Lab, where my focus lies in the domains of machine learning, multi-modal learning (e.g., vision-language, language-audio, and audio-visual), and computer vision. At NAVER, my primary research goal aims to the development of generalizable machine learning models to challenging yet practical scenarios. Prior to joining NAVER, I held a position as a research engineer at KAKAO Corp from 2016 to 2018, where my work focused on recommendation systems and machine learning applications.

News

See older news
  • 12/2024 : Giving a talk at POSTECH AI Day (topic: Probabilistic Language-Image Pre-training) [slide]
  • 12/2024 : Serving as an area chair at ICML 2025
  • 12/2024 : 1 paper [ReWaS] is accepted at AAAI 2025.
  • 11/2024 : 1 paper [FairDRO extension] is accepted at Neural Networks.
  • 10/2024 : 1 paper [ReWaS] is accepted at NeurIPS 2024 Workshopon Video-Language Models.
  • 10/2024 : Serving as an area chair at AISTATS 2025
  • _9/2024 : 1 paper [CKD] is accepted at NeurIPS 2024 D&B track.
  • _9/2024 : Giving a talk at SKKU (topic: "Realistic challenges and limitations of AI") [slide]
  • _8/2024 : RoCOCO[RoCOCO] is accepted as ECCV 2024 Synthetic Data for Computer Vision Workshop and selected as Oral presentation!
  • _8/2024 : Giving a talk at HUST AI Summer School on "Generative AI" (topic: "CompoDiff") [slide]
  • _8/2024 : Serving as an area chair at ICLR 2025
  • _8/2024 : HYPE[HYPE] is selected as Oral presentation at this ECCV!
  • _7/2024 : 1 paper [CompoDiff] is accepted at TMLR.
  • _7/2024 : 3 papers [HYPE][SAT][LUT] are accepted at ECCV 2024.
  • _4/2024 : 1 paper [CompoDiff] is accepted at CVPR 2024 SynData4CV Workshop.
  • _4/2024 : Serving as an area chair at NeurIPS 2024 Datasets and Benchmarks Track.
  • _3/2024 : Serving as an area chair at NeurIPS 2024.
  • _3/2024 : 1 paper [RegionVLP] is accepted at NAACL 2024 main track.
  • _3/2024 : Giving a talk at UNIST (topic: "Probabilistic Image-Text Representations") [slide]
  • _2/2024 : 1 paper [LinCIR] is accepted at CVPR 2024.
  • _2/2024 : Giving a talk at IEIE AI Signal Processing Society Winter School (topic: "Probabilistic Image-Text Representations") [slide]
  • _1/2024 : 2 papers [PCME++][AD Correctness] are accepted at ICLR 2024. One paper[PCME++] is my sole authored paper πŸ€—, and one paper[AD Correctness] is selected as spotlight! (Top-5% paper)
  • 12/2023 : Giving a talk at Dankook University (topic: "Probabilistic Image-Text Representations") [slide]
  • 12/2013: We finally released SynthTriplets18 dataset!
  • 11/2013: Being nominated as NeurIPS 2023 top reviewers (10%) [link].
  • _9/2023 : Giving a talk at HUST AI Summer School on "Modern Machine Learning: Foundations and Applications" (topic: "Probabilistic Image-Text Representations") [slide]
  • _9/2023 : 1 paper [PCME++ short] is accepted at the non-archival track of ICCV 2023 Workshop on Closing The Loop Between Vision And Language (CLVL).
  • _8/2023 : Giving a talk at Yonsei University (topic: "CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion") [slide]
  • _7/2023 : 1 paper [SeiT] is accepted at ICCV 2023.
  • _7/2023 : Serving as a TMLR Action Editor.
  • _6/2023 : Being nominated as a TMLR Expert Reviewer [link].
  • _6/2023 : Giving a talk at Sogang University (topic: "Probabilistic Image-Text Representations") [slide]
  • _5/2023 : Serving as an area chair at NeurIPS 2023 Datasets and Benchmarks Track.
  • _4/2023 : We released "Graphit: A Unified Framework for Diverse Image Editing Tasks" [GitHub] [Graphit], The technical report will be released soon!
  • _4/2023 : 1 paper [3D-Pseudo-Gts] is accepted at CVPR 2023 Workshop on Computer Vision for Mixed Reality (CV4MR).
  • _1/2023 : 1 paper [FairDRO] is accepted at ICLR 2023.
  • _9/2022 : Giving a talk at Sogang University (topic: "ECCV Caption") [slide]
  • _9/2022 : 1 paper [MSDA theorem] is accepted at NeurIPS 2022.
  • _8/2022 : Starting a new chapter in life with Song Park πŸ€΅β€οΈπŸ‘°.
  • _7/2022 : 1 paper [LF-Font journal] is accepted at IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI).
  • _7/2022 : 2 papers [ECCV Caption] [MIRO] are accepted at ECCV 2022.
  • _7/2022 : Giving a talk at UNIST AIGS (topic: "Towards Reliable Machine Learning: Challenges, Examples, Solutions") [slide]
  • _6/2022 : Giving a tutorial on "Shortcut learning in Machine Learning: Challenges, Analysis, Solutions" at FAccT 2022. [ tutorial homepage | slide | video ]
  • _5/2022 : Receiving an outstanding reviewer award at CVPR 2022 [link].
  • _5/2022 : 1 paper [DCC] is accepted at ICML 2022.
  • _4/2022 : 1 paper [WSOL Eval journal] is accepted at IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI).
  • _4/2022 : Organizing ICLR 2022 ML in Korea Social
  • _3/2022 : Giving guest lectures at KAIST and SNU (topic: "Towards Reliable Machine Learning") [slide]
  • _3/2022 : Co-organizing FAccT 2022 Translation/Dialogue Tutorial: "Shortcut learning in Machine Learning: Challenges, Analysis, Solutions" (slides, videos and web pages will be released soon)
  • _3/2022 : 1 paper [CGL] is accepted at CVPR 2022.
  • _2/2022 : Giving a talk at POSTECH AI Research (PAIR) ML Winter Seminar 2022 (topic: "Shortcut learning in Machine Learning: Challenges, Examples, Solutions") [slide]
  • _1/2022 : 2 papers [ViDT] [WCST-ML] are accepted at ICLR 2022.
  • 12/2021 : Co-hosting NeurIPS'21 workshop on ImageNet: Past, Present, and Future with 400+ attendees!
  • 12/2021 : Giving a talk at University of Seoul (topic: "Realistic challenges and limitations of AI") [slide]
  • 11/2021 : Giving a talk at NAVER and NAVER Labs Europe (topic: Mitigating dataset biases in Real-world ML applications) [slide]
  • 11/2021 : Giving a guest lecture at UNIST (topic: Limits and Challenges in Deep Learning Optimizers) [slide]
  • 10/2021 : Releasing an unified few-shot font generation framework! [code]
  • _9/2021 : 2 papers [SWAD] [NHA] are accepted at NeurIPS 2021.
  • _8/2021: Reaching a research milestone of 1,000 citations at Google Scholar and Semantic Scholar!
  • _7/2021 : Co-organizing the NeurIPS Workshop on ImageNet: Past, Present, and Future! [webpage]
  • _7/2021 : 2 papers [MX-Font] [PiT] are accepted at ICCV 2021.
  • _7/2021 : Giving a talk at Computer Vision Centre (CVC), UAB (topic: PCME and AdamP) [info] [slide]
  • _6/2021 : Giving a talk at KSIAM 2021 (topic: AdamP). [slide]
  • _6/2021 : Giving a guest lecture at Seoul National University (topic: few-shot font generation) .[slide]
  • _5/2021 : Receiving an outstanding reviewer award at CVPR 2021 [link].
  • _4/2021 : 1 paper [LF-Font] is accepted at CVPR 2021 workshop (also appeared at AAAI).
  • _3/2021 : 2 papers [PCME] [ReLabel] are accepted at CVPR 2021.
  • _1/2021 : 1 paper [AdamP] is accepted at ICLR 2021.
  • 12/2020 : 1 paper [LF-Font] is accepted at AAAI 2021.
  • _7/2020 : 1 paper [DM-Font] is accepted at ECCV 2020.
  • _6/2020 : Receiving the best paper runner-up award at AICCW CVPR 2020 [DM-Font WS].
  • _6/2020 : Receiving an outstanding reviewer award at CVPR 2020 [link].
  • _6/2020 : Giving a talk at CVPR 2020 NAVER interative session.
  • _6/2020 : 1 paper [ReBias] is accepted at ICML 2020.
  • _4/2020 : 1 paper [DM-Font short] is accepted at CVPR 2020 workshop.
  • _2/2020 : 1 paper [wsoleval] is accepted at CVPR 2020.
  • _1/2020 : 1 paper [HCNN] is accepted at ICASSP 2020.
  • 10/2019 : 1 paper [HCNN short] is accpeted at ISMIR late break demo.
  • 10/2019 : Working at Naver Labs Europe as a visiting researcher (Oct - Dec 2019)
  • _7/2019 : 2 papers [CutMix] [WCT2] are accepted at ICCV 2019 (1 oral presentation).
  • _6/2019 : Giving a talk at ICML 2019 Expo workshop.
  • _5/2019 : 2 papers [MTSA] [RegEval] are accepted at ICML 2019 workshops (1 oral presentation).
  • _5/2019 : Giving a talk at ICLR 2019 Expo talk.
  • _3/2019 : 1 paper [PRM] is accepted at ICLR 2019 workshop.

Current Service Appointments

  • Area Chair - NeurIPS 2025 ICML 2025 ICLR 2025 AISTATS 2025
  • Action Editor - TMLR
  • Reviewer - ICLR Blog track 2025 CVPR 2025 ICCV 2025

Research

I am interested in machine learning and its reliability. My research philosophy aims to (1) scalability and (2) theoretical or conceptual soundness. Recently, I have focused on multi-modal learning (e.g., vision-language and audio-visual), but I am open to exploring a broader range of topics. More details about my research can be found in my research statement (as of December 2023). Representative works that reflect my research approach and significant contributions are highlighted, with the most significant works emphasized in orange.

(C: peer-reviewed conference, W: peer-reviewed workshop, A: arxiv preprint, O: others)
(❋authors contributed equally)
See also at my Google Scholar.

2025

  • Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning.
  • Emergence of Text Readability in Vision Language Models.
    • Jaeyoo Park, Sanghyuk Chun, Wonjae Kim, Sangdoo Yun, Bohyung Han
    • CVPR 2025 EVAL-FoMo Workshop. paper | bibtex
  • LongProLIP: A Probabilistic Vision-Language Model with Long Context Text.
  • DNNs May Determine Major Properties of Their Outputs Early, with Timing Possibly Driven by Bias.
    • Song Park❋, Sanghyuk Chun❋, Byeongho Heo, Dongyoon Han
    • preprint. paper | bibtex
  • Probabilistic Language-Image Pre-Training.
  • Read, Watch and Scream! Sound Generation from Text and Video.
    • Yujin Jeong, Yunji Kim, Sanghyuk Chun, Jiyoung Lee
    • AAAI 2025 | NeurIPS 2024 Workshop on Video-Language Models. paper | code | project page | bibtex
  • FairDRO: Group Fairness Regularization via Classwise Robust Optimization.
    • Taeeon Park, Sangwon Jung, Sanghyuk Chun, Taesup Moon
    • Neural Networks. paper | code | bibtex

2024

  • Do Counterfactually Fair Image Classifiers Satisfy Group Fairness? -- A Theoretical and Empirical Study.
  • HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts. Oral presentation
    • Wonjae Kim, Sanghyuk Chun, Taekyung Kim, Dongyoon Han, Sangdoo Yun
    • ECCV 2024. paper | code | bibtex
  • Similarity of Neural Architectures using Adversarial Attack Transferability.
    • Jaehui Hwang, Dongyoon Han, Byeongho Heo, Song Park, Sanghyuk Chun❋, Jong-Seok Lee❋
    • ECCV 2024. paper | code | bibtex
  • Learning with Unmasked Tokens Drives Stronger Vision Learners.
    • Taekyung Kim❋, Sanghyuk Chun, Byeongho Heo, Dongyoon Han❋
    • ECCV 2024. paper | code | bibtex
  • RoCOCO: Robust Benchmark of MS-COCO to Stress-test Robustness of Image-Text Matching Models. Oral presentation
    • Seulki Park, Daeho Um, Hajung Yoon, Sanghyuk Chun, Sangdoo Yun
    • ECCV 2024 Synthetic Data for Computer Vision Workshop (SyntheticData4CV 2024). paper | code | bibtex
  • CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion.
  • An Efficient Post-hoc Framework for Reducing Task Discrepancy of Text Encoders for Composed Image Retrieval.
    • Jaeseok Byun❋, Seokhyeon Jeong❋, Wonjae Kim, Sanghyuk Chun†, Taesup Moon†
    • preprint. paper | bibtex
  • Toward Interactive Regional Understanding in Vision-Large Language Models
    • Jungbeom Lee, Sanghyuk Chun❋, Sangdoo Yun❋
    • NAACL 2024. paper | code | bibtex
  • Language-only Efficient Training of Zero-shot Composed Image Retrieval.
    • Geonmo Gu❋, Sanghyuk Chun❋, Wonjae Kim, Yoohoon Kang, Sangdoo Yun
    • CVPR 2024. paper | code | bibtex
  • What Does Automatic Differentiation Compute for Neural Networks? Spotlight presentation
    • Sejun Park❋, Sanghyuk Chun❋, Wonyeol Lee
    • ICLR 2024. paper | code | bibtex
  • Improved Probabilistic Image-Text Representations.

2023

  • SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storage.
    • Song Park❋, Sanghyuk Chun❋, Byeongho Heo, Wonjae Kim, Sangdoo Yun
    • ICCV 2023. paper | code | slide | bibtex
  • Three Recipes for Better 3D Pseudo-GTs of 3D Human Mesh Estimation in the Wild.
    • Gyeongsik Moon, Hongsuk Choi, Sanghyuk Chun, Jiyoung Lee, Sangdoo Yun
    • CVPR 2023 Workshop on Computer Vision for Mixed Reality (CV4MR). paper | code | bibtex
  • Re-weighting based Group Fairness Regularization via Classwise Robust Optimization.
    • Sangwon Jung❋, Taeeon Park❋, Sanghyuk Chun, Taesup Moon
    • ICLR 2023. paper | code | bibtex

2022

  • Group Generalized Mean Pooling for Vision Transformer.
    • Byungsoo Ko, Han-Gyu Kim, Byeongho Heo, Sangdoo Yun, Sanghyuk Chun, Geonmo Gu, Wonjae Kim
    • preprint. paper | bibtex
  • A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective.
    • Chanwoo Park❋, Sangdoo Yun❋, Sanghyuk Chun
    • NeurIPS 2022. paper | code | bibtex
  • ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO.
  • Domain Generalization by Mutual-Information Regularization with Pre-trained Models.
    • Junbum Cha, Kyungjae Lee, Sungrae Park, Sanghyuk Chun
    • ECCV 2022. paper | code | bibtex
  • Few-shot Font Generation with Weakly Supervised Localized Representations.
    • Song Park❋, Sanghyuk Chun❋, Junbum Cha, Bado Lee, Hyunjung Shim
    • IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 2022. (IF:24.314)
    • PAMI. paper | code (old) | code (new) | project page | bibtex
  • Evaluation for Weakly Supervised Object Localization: Protocol, Metrics, and Datasets.
    • Junsuk Choe❋, Seong Joon Oh❋, Sanghyuk Chun, Seungho Lee, Zeynep Akata, Hyunjung Shim
    • IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 2022. (IF:24.314)
    • PAMI. paper | code and dataset | bibtex
  • An Extendable, Efficient and Effective Transformer-based Object Detector.
    • Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, Ming-Hsuan Yang
    • preprint. paper | code | bibtex
  • Dataset Condensation with Contrastive Signals.
    • Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, Sungroh Yoon
    • ICML 2022. paper | bibtex
  • Learning Fair Classifiers with Partially Annotated Group Labels.
    • Sangwon Jung, Sanghyuk Chun❋, Taesup Moon❋
    • CVPR 2022. paper | code | bibtex
  • ViDT: An Efficient and Effective Fully Transformer-based Object Detector.
    • Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, Ming-Hsuan Yang
    • ICLR 2022. paper | code | bibtex
  • Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective.
    • Luca Scimeca❋, Seong Joon Oh❋, Sanghyuk Chun, Michael Poli, Sangdoo Yun
    • ICLR 2022. paper | bibtex

2021

  • SWAD: Domain Generalization by Seeking Flat Minima.
    • Junbum Cha, Sanghyuk Chun❋, Kyungjae Lee❋, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, Sungrae Park
    • NeurIPS 2021. paper | code | bibtex
  • Neural Hybrid Automata: Learning Dynamics with Multiple Modes and Stochastic Transitions.
    • Michael Poli❋, Stefano Massaroli❋, Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Atsushi Yamashita, Hajime Asama, Jinkyoo Park, Animesh Garg
    • NeurIPS 2021. paper | bibtex
  • StyleAugment: Learning Texture De-biased Representations by Style Augmentation without Pre-defined Textures.
  • Rethinking Spatial Dimensions of Vision Transformers.
    • Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, Seong Joon Oh
    • ICCV 2021. paper | code | tweet | bibtex
  • Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts.
    • Song Park, Sanghyuk Chun, Junbum Cha, Bado Lee, Hyunjung Shim
    • ICCV 2021. paper | code | bibtex
  • Probabilistic Embeddings for Cross-Modal Retrieval.
  • Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels.
    • Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, Sanghyuk Chun
    • CVPR 2021. paper | code | bibtex
  • AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights.
  • Few-shot Font Generation with Localized Style Representations and Factorization.
    • Song Park❋, Sanghyuk Chun❋, Junbum Cha, Bado Lee, Hyunjung Shim
    • AAAI 2021. CVPR Workshop 2021. paper | code | project page | bibtex

2020

  • Few-shot Compositional Font Generation with Dual Memory.
    • Junbum Cha, Sanghyuk Chun, Gayoung Lee, Bado Lee, Seonghyeon Kim, Hwalsuk Lee
    • ECCV 2020. paper | code | video | bibtex
  • Learning De-biased Representations with Biased Representations.
  • Toward High-quality Few-shot Font Generation with Dual Memory. Oral presentation The best paper runner-up award
    • Junbum Cha, Sanghyuk Chun, Gayoung Lee, Bado Lee, Seonghyeon Kim, Hwalsuk Lee
    • CVPR Workshop 2020. paper | bibtex
  • Evaluating Weakly Supervised Object Localization Methods Right.
  • Data-driven Harmonic Filters for Audio Representation Learning.

2019

  • Neural Approximation of Auto-Regressive Process through Confidence Guided Sampling.
    • YoungJoon Yoo, Sanghyuk Chun, Sangdoo Yun, Jung-Woo Ha, Jaejun Yoo
    • preprint. paper | bibtex
  • Toward Interpretable Music Tagging with Self-attention.
  • CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. Oral presentation
  • Photorealistic Style Transfer via Wavelet Transforms.
  • Automatic Music Tagging with Harmonic CNN.
    • Minz Won, Sanghyuk Chun, Oriol Nieto, Xavier Serra
    • ISMIR LBD 2019. paper | code | bibtex
  • An Empirical Evaluation on Robustness and Uncertainty of Regularization methods.
    • Sanghyuk Chun, Seong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, Youngjoon Yoo
    • ICML Workshop 2019. paper | bibtex
  • Visualizing and Understanding Self-attention based Music Tagging. Oral presentation
  • Where To Be Adversarial Perturbations Added? Investigating and Manipulating Pixel Robustness Using Input Gradients.
    • Jisung Hwang❋, Younghoon Kim❋, Sanghyuk Chun❋, Jaejun Yoo, Ji-Hoon Kim, Dongyoon Han
    • ICLR Workshop 2019. paper | bibtex

~ 2018

  • Multi-Domain Processing via Hybrid Denoising Networks for Speech Enhancement.
  • A Study on Intelligent Personalized Push Notification with User History.
    • Hyunjong Lee, Youngin Jo, Sanghyuk Chun, Kwangseob Kim
    • Big Data 2017. paper | bibtex
  • Scalable Iterative Algorithm for Robust Subspace Clustering: Convergence and Initialization.
    • Master's Thesis, Korea Advanced Institute of Science and Technology, 2016 (advised by Jinwoo Shin) paper | code

Academic Activities

Professional Service

  • Conference Area Chair:
    • ICLR 2025
    • AISTATS 2025
    • NeurIPS 2024-2025
    • NeurIPS Dataset and Benchmark (D&B) track 2023-2024
    • ICML 2025
  • Tutorial / Workshop / Social Organizer:
    • FAccT 2022 Translation/Dialogue Tutorial: "Shortcut learning in Machine Learning: Challenges, Analysis, Solutions"
    • NeurIPS 2021 Workshop on ImageNet: Past, Present, and Future
      • Co-organized by Zeynep Akata, Lucas Beyer, Sanghyuk Chun, Almut Sophia Koepke, Diane Larlus, Seong Joon Oh, Rafael Sampaio de Rezende, Sangdoo Yun, Xiaohua Zhai

Awards

Talks

See older talks
Code and Data

Datasets

  • CUB v2 and OpenImages 30k [C5] [J2]: These datasets were designed and collected for the WSOL evaluation project. We newly collected bird images for CUB v2 (200 classes) and re-organize OpenImages V5 for OpenImages 30k (100 classes)
  • Biased MNIST, 9-Class ImageNet and ImageNet cluster labels [C6]: We proposed these datasets for measuring how ML models can be generalized to bias shift. Biased MNIST is a synthetic dataset based on MNIST, while each image has background colors which highly correlates with the labels (controllable). ImageNet-9 contains 9 super-classes (dog, cat, frog, turtle, bird, monkey, fish, crab, insect) and 57k training samples. We also proposes "unbiased accuracy" by using cluster labels (K=9), which empicially matches to Shell, Grass, Close-up, Eye, Human, Sand and Mammal.
  • CUB Caption [C11]: CUB was not majorly used for image-text matching (ITM) retrieval. However, while doing the PCME project, we proposed to use the CUB dataset as a ITM benchmark for measuring the impact of many-to-many correspondence; if an image and a description are in the same class, then we treat it as "positive" otherwise "negative". We carefully devide the training-validation (150 classes) and test splits (50 classes) following Xian et al. 2017.
  • ECCV Caption [C21]: Although CUB Caption can evaluate the impact of many-to-many correspondence in ITM, this dataset is still "synthetic". For a more practical usage, we proposed the ECCV Caption benchmark, by correcting the false negatives (FNs) in the MS-COCO Caption dataset. By machine and human annotators, we collected x8.47 positive images and x3.58 positive captions compared to the original COCO Caption.
  • RoCOCO [W9]: Is an ITM model robust to a malicious manipulation on captions or images? Here, we investigated the impact of altered captions (same concept, different concept, rand voca attack, and dangerous word) and mixed images. Even state-of-the-art ITM models showed substantial performance degradations on our benchmark.

Softwares

  • Few-shot Font Generation Benchmark.
    • Song Park, Sanghyuk Chun
    • open source. code
  • Graphit: A Unified Framework for Diverse Image Editing Tasks.
    • Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Sangdoo Yun, Yoohoon Kang
    • open source. code | demo
Industry Experience

NAVER AI Lab (2018 ~ Now)

  • Hangul
    Hangul
    DM-Font teasor
    Hangul Handwriting Font Generation

    Distributed at 2019 Hangul's day (ν•œκΈ€λ‚ ), [Full font list]

    • Hangul (Korean alphabet, ν•œκΈ€) originally consists of only 24 sub-letters (γ„±, γ…‹, γ„΄, γ„·, γ…Œ, ㅁ, γ…‚, ㅍ, γ„Ή, γ……, γ…ˆ, γ…Š, γ…‡, γ…Ž, γ…‘, γ…£, γ…—, ㅏ, γ…œ, γ…“, γ…›, γ…‘, γ… , γ…•), but by combining them, there exist 11,172 valid characters in Hangul. For example, "ν•œ" is a combination of γ…Ž, ㅏ, and γ„΄, and "쐰" is a combination of γ……, γ……, γ…—, γ…£, and γ„΄. It makes generating a new Hangul font be very expensive and time-consuming. Meanwhile, since 2008, Naver has distributed Korean fonts for free (named Nanum fonts, λ‚˜λˆ” κΈ€κΌ΄).
    • In 2019, we developed a technology for fully-personalized Hangul generation only with 152 characters. We opened an event page where users can submit their own handwriting. The full generated font list can be found in [this link]. Details for the generation technique used for the service was presented in Deview 2019 [Link].
    • This work was also extended to the few-shot generation based on the compositionality. See the papers in AI for Content Creation Workshop (AICCW) at CVPR 2020 (short paper) [Link], ECCV 2020 (full paper) [Link], AAAI 2021 [Link], ICCV 2021 [Link], and journal extension [Link].
    • [BONUS] You can play with my handwriting here
  • example sticker
    Example emoji from LINE sticker shop.
    Emoji Recommendation (LINE Timeline)

    Deployed in Jan. 2019

    • LINE is a major messenger player in east asia (Japan, Taiwan, Thailand, Indonesia, and Korea). In the application, users can buy and use numerous emoijs a.k.a. LINE Sticker.
    • In this project, we recommended emojis to users based on their profile picture (cross-domain recommendation).
    • I developed and researched the entire pipeline of the cross-domain recommendation system and operation tools.
  • Mentees / Short-term post-doctoral collaborators / Internship students

    Topics: Reliable ML Vision-Language Modality-specific tasks Generative models Other topics

    • _    Junwon Lee (KAIST, 2025) -- AVL representation learning
    • _    Sehyun Kwon (Seoul National University, 2024) -- VL representation learning
    • _    Jaeyoo Park (Seoul National University, 2024) -- VL representation learning
    • _    Jungin Park (Visiting researcher, 2024) -- VL representation learning
    • _    Yujin Jeong (Korea University, 2024) [C33/W10] -- AVL representation learning
    • _ _ Heesun Bae (KAIST, 2023) -- VL representation learning under noisy environment
    • _    Jungbeom Lee (Visiting researcher, 2023) [C28] -- VL representation learning
    • _    Eunji Kim (Seoul National University, 2022) -- XAI + Probabilistic Machine (the internship project is published at ICML 2023 [paper])
    • _    Jaehui Hwang (Yonsei University, 2022) [C30] -- Adversarial robustness and XAI
    • _    Chanwoo Park (Seoul National University, 2021-2022) [C22] -- Deep learning theory
    • _    Gyeongsik Moon (Visiting researcher, 2022) [W7] -- Semi-supervised learning for 3D Human Mesh Estimation
    • _    Hongsuk Choi (Visiting researcher, 2022) [W7] -- Semi-supervised learning for 3D Human Mesh Estimation
    • _ _ Seulki Park (Seoul National University, 2022) [W9] -- VL robustness benchmark
    • _    Saehyung Lee (Seoul National University, 2021-2022) [C19] -- Data condensation
    • _    Sangwon Jung (Seoul National University, 2021-2023) [C18] [C19] -- Fairness with not enough group labels, group fairness
    • _    Luca Scimeca (A short-term post-doctoral collaborator, 2021) [C16] [C14] -- Understanding shortcut learning phenomenon in feature space
    • _    Michael Poli (KAIST, 2021) [C14] [C16] -- Neural hybrid automata
    • _    Hyemi Kim (KAIST, 2021) -- Test-time training for robust prediction
    • _    Jun Seo (KAIST, 2021) -- Self-supervised learning
    • _    Song Park (Yonsei University, 2020-2021) [C8/W6] [C12] [A4] [J2] -- Few-shot font generation
    • _    Hyojin Bahng (Korea University, 2019) [C6] -- De-biasing
    • _    Junsuk Choe (Yonsei University, 2019) [C5] [J1] -- Reliable evaluation for WSOL
    • _    Naman Goyal (IIT RPR, 2019) -- Robust representation against shift
    • _    Minz Won (Music Technology Group, Universitat Pompeu Fabra, 2018-2019) [W2] [W4] [A2] [C4] -- Audio representation learning
    • _    Byungkyu Kang (Yonsei University, 2018) [C2] -- Image-to-image translation and style transfer
    • _    Jang-Hyun Kim (Seoul National University, 2018) [A1] -- Audio representation learning
    • _    Jisung Hwang (University of Chicago, 2018) [W1] -- Adversarial robustness
    • _    Younghoon Kim (Seoul National University, 2018) [W1] -- Adversarial robustness

Kakao Advanced Recommendation Technology (ART) team (2016 ~ 2018)

  • Kakao
    Recommender Systems (Kakao services)

    Feb. 2016 - Feb. 2018

    • I developed and maintained a large-scale real-time recommender system (Toros [PyCon Talk] [AI Report]) for various services in Daum and Kakao. I mainly worked with content-based representation modeling (for textual, visual, and musical data), collaborative filtering modeling, user embedding, user clustering, and ranking system based on Multi-armed Bandit.
    • Textual domain: Daum News similar article recommendation, Brunch (blog service) similar post recommendation, Daum Cafe (community service) hit item recommendation.
    • Visual domain: Daum Webtoon and Kakao Page similar item recommendation, video recommendation for a news article (cross-domain recommendation).
    • Audio domain: music recommendation for Kakao Mini (smart speaker), Melon and Kakao Music.
    • Online to offline: Kakao Hairshop style recommendation.
  • IPPN
    System overview.
    Personalized Push Notification with User History (Daum, Kakao Page)

    Deployed in 2017

    • The mobile push service (or alert system) is widely-used in mobile applications to attain a high user retention rate. However, a freqeunt push notification makes a user feel fatigue, resulting on the application removal. Usually, the push notification system is a rule-based system, and managed by human labor. In this project, we researched and developed a personalized push notification system based on user activity and interests. The system has been applied to Daum an Kakao Page mobile applications. More details are in our paper.
  • Daum Shopping
    Large-Scale Item Categorization in e-Commerce (Daum Shopping)

    Deployed in 2017

    • An accurate categorization helps users to search desired items in e-Commerce based on the category, e.g., clothes / shoes / sneakers. However, the categorization is usually performed based on rule-based systems or human labor, which leads to low coverage of categorized items. Even the automatic item categorization is difficult due to its web-scale data size, the highly unbalanced annotation distribution, and noisy labels. I developed a large-scale item categorization system for Daum Shopping based on a deep network, from the operation tool to the categorization API.

Internship

  • Naver Labs
    Research internship (Naver Labs)

    Aug. 2015 - Dec. 2015

    • During the internship, I implemented batch normalization (BN) to AlexNet, Inception v2 and VGG on ImageNet using Caffe. I also researched batch normalization for sequential models, e.g., RNN using Lua Torch.
  • IUM-SOCIUS
    Software engineer (IUM-SOCIUS)

    Jun. 2012 - Jan. 2013

    • I worked as web developer at IUM-SOCIUS. During the internship, I developed and maintained internal batch services (JAVA spring batch), internal statistics service (Python Flask, MongoDB), internal admin tools (Python Django, MySQL), and main service systems (JAVA spring, Ruby on Rails, MariaDB).
Education and Career
  • M.S. (2014.03 - 2016.02), School of Electrical Engineering, KAIST
  • B.S. (2009.03 - 2014.02), School of Electrical Engineering and School of Management Science (double major), KAIST