publications

2024

1. Preprint

   EPT: Explosive Prompt Tuning For Parameter-Efficient with Large Norm Prompt

   Wonyong Jo, Taewon Park, Minho Lee, and 1 more author

   2024
2. NeurIPS

   Discrete Dictionary-based Decomposition Layer for Structured Representation Learning

   Taewon Park, Hyun-Chul Kim, and Minho Lee

   arXiv preprint, 2024

   Abs PDF

   Neuro-symbolic neural networks have been extensively studied to integrate symbolic operations with neural networks, thereby improving systematic generalization. Specifically, Tensor Product Representation (TPR) framework enables neural networks to perform differentiable symbolic operations by encoding the symbolic structure of data within vector spaces. However, TPR-based neural networks often struggle to decompose unseen data into structured TPR representations, undermining their symbolic operations. To address this decomposition problem, we propose a Discrete Dictionary-based Decomposition (D3) layer designed to enhance the decomposition capabilities of TPR-based models. D3 employs discrete, learnable key-value dictionaries trained to capture symbolic features essential for decomposition operations. It leverages the prior knowledge acquired during training to generate structured TPR representations by mapping input data to pre-learned symbolic features within these dictionaries. D3 is a straightforward drop-in layer that can be seamlessly integrated into any TPR-based model without modifications. Our experimental results demonstrate that D3 significantly improves the systematic generalization of various TPR-based models while requiring fewer additional parameters. Notably, D3 outperforms baseline models on the synthetic task that demands the systematic decomposition of unseen combinatorial data.

3. ICLR

   Attention-based Iterative Decomposition for Tensor Product Representation

   Taewon Park, Inchul Choi, and Minho Lee

   In International Conference on Learning Representation , 2024

   Abs PDF

   In recent research, Tensor Product Representation (TPR) is applied for the systematic generalization task of deep neural networks by learning the compositional structure of data. However, such prior works show limited performance in discovering and representing the symbolic structure from unseen test data because of the incomplete bindings to the structural representations. In this work, we propose an Attention-based Iterative Decomposition (AID) module that can effectively improve the binding for the structured representations encoded from the sequential input features with TPR. Our AID can be easily adapted to any TPR-based model and provides enhanced systematic decomposition through a competitive attention mechanism between input features and structured representations. In our experiments, AID shows effectiveness by significantly improving the performance of TPR-based prior works on the series of systematic generalization tasks. Moreover, in the quantitative and qualitative evaluations, AID produces more compositional and well-bound structural representations than other works.

2022

1. ICONIP

   Learning Associative Reasoning Towards Systematicity Using Modular Networks

   Jun-Hyun Bae*, Taewon Park*, and Minho Lee

   In International Conference on Neural Information Processing , 2022

   Abs HTML

   Learning associative reasoning is necessary to implement human-level artificial intelligence even when a model faces unfamiliar associations of learned components. However, conventional memory augmented neural networks (MANNs) have shown degraded performance on systematically different data since they lack consideration of systematic generalization. In this work, we propose a novel architecture for MANNs which explicitly aims to learn recomposable representations with a modular structure of RNNs. Our method binds learned representations with a Tensor Product Representation (TPR) to manifest their associations and stores the associations into TPR-based external memory. In addition, to demonstrate the effectiveness of our approach, we introduce a new benchmark for evaluating systematic generalization performance on associative reasoning, which contains systematically different combinations of words between training and test data. From the experimental results, our method shows superior test accuracy on systematically different data compared to other models. Furthermore, we validate the models using TPR by analyzing whether the learned representations have symbolic properties.

2021

1. Neural Networks

   Distributed associative memory network with memory refreshing loss

   Taewon Park*, Inchul Choi*, and Minho Lee

   Neural Networks, 2021

   Abs HTML

   Despite recent progress in memory augmented neural network (MANN) research, associative memory networks with a single external memory still show limited performance on complex relational reasoning tasks. Especially the content-based addressable memory networks often fail to encode input data into rich enough representation for relational reasoning and this limits the relation modeling performance of MANN for long temporal sequence data. To address these problems, here we introduce a novel Distributed Associative Memory architecture (DAM) with Memory Refreshing Loss (MRL) which enhances the relation reasoning performance of MANN. Inspired by how the human brain works, our framework encodes data with distributed representation across multiple memory blocks and repeatedly refreshes the contents for enhanced memorization similar to the rehearsal process of the brain. For this procedure, we replace a single external memory with a set of multiple smaller associative memory blocks and update these sub-memory blocks simultaneously and independently for the distributed representation of input data. Moreover, we propose MRL which assists a task’s target objective while learning relational information existing in data. MRL enables MANN to reinforce an association between input data and task objective by reproducing stochastically sampled input data from stored memory contents. With this procedure, MANN further enriches the stored representations with relational information. In experiments, we apply our approaches to Differential Neural Computer (DNC), which is one of the representative content-based addressing memory models and achieves the state-of-the-art performance on both memorization and relational reasoning tasks.

2. ICEIC

   4W1H Keyword Extraction based Summarization Model

   Seungyeon Lee, Taewon Park, and Minho Lee

   In International Conference on Electronics, Information, and Communication , 2021

   Abs HTML

   In this internet era, with rapidly growing online information, there is a need for automatic summarization of textual documents from plethora of available information, making it an interesting area of research. Automatic keyword extraction and text summarization are Natural Language Processing (NLP) tasks for extracting relevant information from the large text documents. 4W1H (Who, When, Where, What, How) keywords are crucial for sentence generation. Despite the potential of 4W1H keywords, there have not been approaches that utilize the keywords in NLP tasks, particularly summarization. In this paper, we propose a new summarization method based on 4W1H keywords extraction which extracts the answer to a question corresponding to each event in QA format. We apply our methods to BERT and ELECTRA models to generate a summary, which are well-known pre-trained Language Models (LMs) in NLP domain, as a baseline. In experiments, our 4W1H keyword extraction method shows promising performance on AI Hub Machine Reading Comprehension (MRC) dataset, recording an extraction performance of an F1-score as 84.93%. Moreover, we show the results of generating a rule-based summarization using keywords extracted with 4W1H.