Agentic· paper-reproduction
SUPER
Allen AI (AI2) · 2024
Evaluates agents on setting up and executing tasks from research code repositories.
Examples
Browse full dataset ↗task_id
unified-prompt-selection
github_repo
https://github.com/soheeyang/unified-prompt-selection
git_commit
76c764b1e75f4707f612537126d172c6091d1e8e
query
Run probability-based prompt selection on the SST-2 dataset using opt-125m as the base model with the script `run_prompt_selection.py`. Report metrics.
Additional instructions:
If any details are missing from the task description, you are allowed to make arbitrary assumptions and decisions to fulfill the task.
To reduce execution time:
1. Load/preprocess only the first 10 rows of each set in the dataset. 2. Only run a single epoch (when training). 3. Make sure you only run a single experiment, disabling any grid searchor hyperparameter tuning.
Git repository: https://github.com/soheeyang/un …
query_components
{"e2e_task":"Run probability-based prompt selection on the SST-2 dataset using opt-125m as the base model with the script `run_prompt_selection.py`.","scenario_task":"None","report":"Report metrics.","instructions":"If any details are missing from the task description, you are allowed to make arbitrary assumptions and decisions to fulfill the task.\nTo reduce execution time:\n1. Load/preprocess only the first 10 rows of each set in the dataset. 2. Only run a single epoch (when training). 3. Make sure you only run a single experiment, disabling any grid searchor hyperparameter tuning. "}
entrypoint
run_prompt_selection.py
task_id
gener
github_repo
https://github.com/dmis-lab/gener
git_commit
b797b365cbea1a690fcbe5f41801aa1d38a19eb2
query
Perform self-training for NER on the CoNLL-2003 dataset using roberta-base as the base model with the script `self_training/run_ner.py`. Report metrics.
Additional instructions:
If any details are missing from the task description, you are allowed to make arbitrary assumptions and decisions to fulfill the task.
To reduce execution time:
1. Load/preprocess only the first 10 rows of each set in the dataset. 2. Only run a single epoch (when training). 3. Make sure you only run a single experiment, disabling any grid searchor hyperparameter tuning.
Git repository: https://github.com/dmis-lab/ge …
query_components
{"e2e_task":"Perform self-training for NER on the CoNLL-2003 dataset using roberta-base as the base model with the script `self_training/run_ner.py`.","scenario_task":"None","report":"Report metrics.","instructions":"If any details are missing from the task description, you are allowed to make arbitrary assumptions and decisions to fulfill the task.\nTo reduce execution time:\n1. Load/preprocess only the first 10 rows of each set in the dataset. 2. Only run a single epoch (when training). 3. Make sure you only run a single experiment, disabling any grid searchor hyperparameter tuning. "}
entrypoint
self_training/run_ner.py
Real rows from the Hugging Face datasets server · long values truncated