Scientific LLM Benchmarks
GitHub
← All benchmarks
Biology· code-generation

BioCoder

Yale University (Gerstein Lab) · 2023

Fuzz-tested benchmark evaluating LLMs on generating bioinformatics code with cross-file dependencies and domain knowledge.

Biology
GitHub stars
Task type
code-gen
Modality
code
Access
open
Size
207 items
License
CC-BY-4.0
Metrics
Pass@k
promptSummaryOnly
This is in python write a function that takes in a string as an argument and returns a sanitized version of the string suitable for LaTeX formatting. The function should follow these steps: 1. Replace any occurrences of four backslashes (\\\\) with the string \\textbackslash. 2. Escape certain characters, including $, %, _, }, {, &, and #, by adding a backslash before them. 3. Replace special characters such as tilde (~) with the corresponding LaTeX equivalent (e.g. $\\sim$). The function should be named sanitize_tex and should have one argument named original_text. It should return the sanit …
test_case_id
61beb3529846e024cdff01d3e2ba1a1ec4212dd64426028deb1065f1975bd376
goldenCode
def sanitize_tex(original_text): """Sanitize TeX text. Args: original_text (str): the text to sanitize for LaTeX Text is sanitized by following these steps: 1. Replaces ``\\\\`` by ``\\textbackslash`` 2. Escapes certain characters (such as ``$``, ``%``, ``_``, ``}``, ``{``, ``&`` and ``#``) by adding a backslash (*e.g.* from ``&`` to ``\\&``). 3. Replaces special characters such as ``~`` by the LaTeX equivalent (*e.g.* from ``~`` to ``$\\sim$``). """ sanitized_tex = original_text.replace('\\', '\\textbackslash ') sanitized_tex = re. …
contextCode
import random import hashlib import numpy as np import skimage import skimage.measure import scipy.ndimage import os import logging from functools import wraps from scipy import stats import sys import math import subprocess from pathlib import PurePath from itertools import islice import pysam import pandas as pd from scipy.signal import savgol_coeffs, savgol_filter from scipy.stats import norm import re import fileinput import warnings from scipy.stats import scoreatpercentile, chisquare from sklearn.preprocessing import scale from sklearn.cluster import KMeans, AgglomerativeClustering _char …
repository
pgxcentre/genipe
content
def sanitize_tex(original_text): """Sanitize TeX text. Args: original_text (str): the text to sanitize for LaTeX Text is sanitized by following these steps: 1. Replaces ``\\\\`` by ``\\textbackslash`` 2. Escapes certain characters (such as ``$``, ``%``, ``_``, ``}``, ``{``, ``&`` and ``#``) by adding a backslash (*e.g.* from ``&`` to ``\\&``). 3. Replaces special characters such as ``~`` by the LaTeX equivalent (*e.g.* from ``~`` to ``$\\sim$``). """ sanitized_tex = original_text.replace('\\', '\\textbackslash ') sanitized_tex = re. …
numLines
27
filePath
genipe/reporting/utils.py
lineEnd
84
comment
Sanitize TeX text. Args: original_text (str): the text to sanitize for LaTeX Text is sanitized by following these steps: 1. Replaces ``\\`` by ``\textbackslash`` 2. Escapes certain characters (such as ``$``, ``%``, ``_``, ``}``, ``{``, ``&`` and ``#``) by adding a backslash (*e.g.* from ``&`` to ``\&``). 3. Replaces special characters such as ``~`` by the LaTeX equivalent (*e.g.* from ``~`` to ``$\sim$``).
signature
def sanitize_tex(original_text)
lineStart
58
language
Python
promptSummaryOnly
This is in python Write a function called "filter_kmers" that takes in three parameters: "kmers", "kmer_len", and "rate". The function should return a clean set of k-mers in a tuple. The function should filter out low-complexity and low-frequency kmers. Inside the function, create a list called "low_comp" which uses a regular expression to match to each base in 'ACGTN' and divide "kmer_len" by 2. Initialize "i" and "x" both to -1. Use a while loop to iterate until "x" is equal to the length of "low_comp". Inside the loop, increment "i" by 1 and set "x" to the sum of the boolean output of each …
test_case_id
98b443b728881547cc9d48a5ccb5f26c65aa1cbafd482b962b85a4b2e4725d3f
goldenCode
def filter_kmers(kmers, kmer_len, rate): """Return a clean set of k-mers in tuple. Filter low-complexity and low-frequency kmers. """ low_comp = [re.compile(base * (kmer_len // 2)) for base in 'ACGTN'] i, x = -1, -1 while x != len(low_comp): i += 1 x = sum([(not p.findall(kmers[i][0])) for p in low_comp]) max_hits = kmers[i][1] clean = [] total = 0 for s, n in kmers[i:]: if sum([(not p.findall(s)) for p in low_comp]) != len(low_comp): continue if float(max_hits) / n > rate: break clean.a …
contextCode
import random import hashlib import numpy as np import skimage import skimage.measure import scipy.ndimage import os import logging from functools import wraps from scipy import stats import sys import math import subprocess from pathlib import PurePath from itertools import islice import pysam import pandas as pd from scipy.signal import savgol_coeffs, savgol_filter from scipy.stats import norm import re <<insert solution here>> def main(): random.seed(<|int;range=0,100|>) letters = ['A', 'C', 'G', 'T', 'N'] kmers = [] for _ in range(5): kmers.append((''.join(random.ch …
repository
jnktsj/DNApi
content
def filter_kmers(kmers, kmer_len, rate): """Return a clean set of k-mers in tuple. Filter low-complexity and low-frequency kmers. """ low_comp = [re.compile(base * (kmer_len // 2)) for base in 'ACGTN'] i, x = -1, -1 while x != len(low_comp): i += 1 x = sum([(not p.findall(kmers[i][0])) for p in low_comp]) max_hits = kmers[i][1] clean = [] total = 0 for s, n in kmers[i:]: if sum([(not p.findall(s)) for p in low_comp]) != len(low_comp): continue if float(max_hits) / n > rate: break clean.a …
numLines
22
filePath
dnapilib/kmer.py
lineEnd
50
comment
Return a clean set of k-mers in tuple. Filter low-complexity and low-frequency kmers.
signature
def filter_kmers(kmers, kmer_len, rate)
lineStart
29
language
Python

Real rows from the Hugging Face datasets server · long values truncated