PyTorch Huggingface BERT-NLP for Named Entity Recognition

[AshwinAmbal]

I have been using your PyTorch implementation of Google’s BERT by HuggingFace for the MADE 1.0 dataset for quite some time now. Up until last time (11-Feb), I had been using the library and getting an F-Score of 0.81 for my Named Entity Recognition task by Fine Tuning the model. But this week when I ran the exact same code which had compiled and run earlier, it threw an error when executing this statement:

input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts], maxlen=MAX_LEN, dtype=”long”, truncating=”post”, padding=”post”)

ValueError: Token indices sequence length is longer than the specified
maximum sequence length for this BERT model (632 > 512). Running this
sequence through BERT will result in indexing errors

The full code is available in this colab notebook.

To get around this error I modified the above statement to the one below by taking the first 512 tokens of any sequence and made the necessary changes to add the index of [SEP] to the end of the truncated/padded sequence as required by BERT.

input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt[:512]) for txt in tokenized_texts], maxlen=MAX_LEN, dtype=”long”, truncating=”post”, padding=”post”)

The result shouldn’t have changed because I am only considering the first 512 tokens in the sequence and later truncating to 75 as my (MAX_LEN=75) but my F-Score has dropped to 0.40 and my precision to 0.27 while the Recall remains the same (0.85). I am unable to share the dataset as I have signed a confidentiality clause but I can assure all the preprocessing as required by BERT has been done and all extended tokens like (Johanson –> Johan ##son) have been tagged with X and replaced later after the prediction as said in the BERT Paper.

Has anyone else faced a similar issue or can elaborate on what might be the issue or what changes the PyTorch (Huggingface) has done on their end recently?

[AshwinAmbal]

I've found a fix to get around this.
Running the same code with pytorch-pretrained-bert==0.4.0 solves the issue and the performance is restored to normal.
There's something messing with the model performance in BERT Tokenizer or BERTForTokenClassification in the new update which is affecting the model performance.
Hoping that HuggingFace clears this up soon. :)
Thanks.

[jplehmann]

There's something messing with the model performance in BERT Tokenizer or BERTForTokenClassification in the new update which is affecting the model performance.
Hoping that HuggingFace clears this up soon. :)

Sounds like the issue should remain open?

[AshwinAmbal]

Oh. I didn't know I closed the issue. Let me reopen it now. Thanks.

…

On Tue, 5 Mar, 2019, 10:57 AM John Lehmann, ***@***.***> wrote: There's something messing with the model performance in BERT Tokenizer or BERTForTokenClassification in the new update which is affecting the model performance. Hoping that HuggingFace clears this up soon. :) Sounds like the issue should remain open? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#328 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcM_oJw9xYJ3ppFG_egEJTrMQq22ERKhks5vTr4wgaJpZM4bVcDQ> .

[AshwinAmbal]

Sorry about that. Didn't realise I closed the issue.
Reopened it now. :)

[thomwolf]

Seems strange that the tokenization changed.

So you were only having sequence with less than 512 tokens before and now some sequences are longer?

Without having access to your dataset I can't really help you but if you can compare the tokenized sequences in your dataset with pytorch-pretrained-bert==0.4.0 versus sequences tokenized with the current pytorch-pretrained-bert==0.6.1 to identify a sequence which is tokenized differently it could help find the root of the issue.

Then maybe you can just post some part of a sequence or example which is tokenized differently without breaching your confidentiality clause?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[savindi-wijenayaka]

I had the same issue when trying to use it with Flair for text classification. Can I know the root cause of this issue? Does this mean that my text part in the dataset is too long?

[thomwolf]

Yes, BERT only accepts inputs smaller or equal to 512 tokens.

[Ezekiel25c17]

Seems strange that the tokenization changed.

So you were only having sequence with less than 512 tokens before and now some sequences are longer?

Without having access to your dataset I can't really help you but if you can compare the tokenized sequences in your dataset with pytorch-pretrained-bert==0.4.0 versus sequences tokenized with the current pytorch-pretrained-bert==0.6.1 to identify a sequence which is tokenized differently it could help find the root of the issue.

Then maybe you can just post some part of a sequence or example which is tokenized differently without breaching your confidentiality clause?

I think I found a little bug in tokenization.py that may be related to this issue.
I was facing a similar problem that using the newest version leads to a huge accuracy drop (from 88% to 22%) in a very common multi-class news title classification task. Using pytorch-pretrained-bert==0.4.0 was actually a workaround so I did a comparison of the tokenization logs of these two versions.

the main problem was that many tokens have different ids during training and evaluation. Compared to 0.4.0, the newest version has an additional function that saves the vocabulary to the output_dir/vocab.txt after training and then loads this generated vocab.txt instead during evaluation.
In my case, this generated vocab.txt differs from the original one because in https://github.com/huggingface/pytorch-pretrained-BERT/blob/3763f8944dc3fef8afb0c525a2ced8a04889c14f/pytorch_pretrained_bert/tokenization.py#L65 the tokenizer deletes all the trailing spaces. This actually strips different tokens, say a normal space and a non-break space into an identical empty token "". After changing this line to "token = token.rstrip("¥n") ", I was able to reproduce the expected accuracy using the newest version

[thomwolf]

@Ezekiel25c17 I'm a bit surprised that training spaces would be important in the vocabulary so I would like to investigate this deeper.

Can you give me the reference of the following elements you were using in your tests:

• the python version,
• versions of pytorch-pretrained-bert
• the pretrained model,
• the vocabulary (probably same as the model I guess),
• the example script.

So I can reproduce the behavior