transformers RobertaWithFeatures license:mit endpoints_compatible region:us
---
license: mit
---


# Project Name

Provide a brief introduction to what the project does and its target audience. Describe the problems it solves or the functionality it offers.

## Features

- Custom integration of numerical features with text data using RoBERTa.
- Ability to handle complex text classification tasks with additional context from numerical data.

## Prerequisites

Before you begin, ensure you have met the following requirements:
- Python 3.6+
- PyTorch 1.4+
- Transformers 3.5+

## Installation

Install the required packages using pip:

```bash
pip install torch transformers

import torch
import torch.nn as nn
from transformers import RobertaModel

class RobertaWithFeatures(nn.Module):
    def __init__(self, num_features, num_labels):
        super(RobertaWithFeatures, self).__init__()
        # Load the pretrained RoBERTa base model
        self.roberta = RobertaModel.from_pretrained('roberta-base')
        # Define a linear layer to process numerical features
        self.feature_processor = nn.Linear(num_features, 768)
        # Final classifier that takes the concatenated output of text + numerical features
        self.classifier = nn.Linear(768 * 2, num_labels)

    def forward(self, input_ids, attention_mask, features):
        outputs = self.roberta(input_ids=input_ids, attention_mask=attention_mask)
        sequence_output = outputs.pooler_output
        features_processed = self.feature_processor(features).squeeze(1)
        combined_features = torch.cat((sequence_output, features_processed), dim=1)
        logits = self.classifier(combined_features)
        return logits

```

## Instantiate the model


```

model = RobertaWithFeatures(num_features=8, num_labels=2)
model.load_state_dict(torch.load('roberta_with_features_v1.pth'))
model.eval()  # Set the model to evaluation mode
```

