In case anyone ever encounters a similar or related issue in the future: What eventually helped me solve the issue was to shift the definition of the tokenizer outside of __name__ == '__main__':. This enabled the proper serialization and unpickling of the tokenizer in every sub-process. Thanks to @lhoestq for proposing the fix!