101.md

Results of Code LLama 34B on HumanEval/101

Date: 01 September, 2023
Model: codellama-34b-instruct
Temperature: 0.2
Passing runs: 0
Total runs: 10
Success rate: 0%

Prompt

def words_string(s):
    """
    You will be given a string of words separated by commas or spaces. Your task is
    to split the string into words and return an array of the words.
    
    For example:
    words_string("Hi, my name is John") == ["Hi", "my", "name", "is", "John"]
    words_string("One, two, three, four, five, six") == ["One", "two", "three", "four", "five", "six"]
    """

The above is an incomplete Python code fragment. Return the complete and correct code with no additional text.

Correct solution

def words_string(s):
    """
    You will be given a string of words separated by commas or spaces. Your task is
    to split the string into words and return an array of the words.
    
    For example:
    words_string("Hi, my name is John") == ["Hi", "my", "name", "is", "John"]
    words_string("One, two, three, four, five, six") == ["One", "two", "three", "four", "five", "six"]
    """if not s:
        return []

    s_list = []

    for letter in s:
        if letter == ',':
            s_list.append(' ')
        else:
            s_list.append(letter)

    s_list = "".join(s_list)
    return s_list.split()

Unit tests

def check(candidate):

    # Check some simple cases
    assert True, "This prints if this assert fails 1 (good for debugging!)"
    assert candidate("Hi, my name is John") == ["Hi", "my", "name", "is", "John"]
    assert candidate("One, two, three, four, five, six") == ["One", "two", "three", "four", "five", "six"]
    assert candidate("Hi, my name") == ["Hi", "my", "name"]
    assert candidate("One,, two, three, four, five, six,") == ["One", "two", "three", "four", "five", "six"]

    # Check some edge cases that are easy to work out by hand.
    assert True, "This prints if this assert fails 2 (also good for debugging!)"
    assert candidate("") == []
    assert candidate("ahmed     , gamal") == ["ahmed", "gamal"]

Runs

❌ Run 1

Start time: 2023-09-01 02:47:01
Duration: 3.38s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 2

Start time: 2023-09-01 02:50:00
Duration: 3.69s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 3

Start time: 2023-09-01 02:52:56
Duration: 2.44s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 4

Start time: 2023-09-01 02:55:54
Duration: 2.19s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 5

Start time: 2023-09-01 02:58:51
Duration: 4.25s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 6

Start time: 2023-09-01 03:01:55
Duration: 3.15s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 7

Start time: 2023-09-01 03:04:55
Duration: 3.06s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 8

Start time: 2023-09-01 03:07:59
Duration: 4.21s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 9

Start time: 2023-09-01 03:10:55
Duration: 3.37s
Result: Fail

def words_string(s):
    return s.split()

❌ Run 10

Start time: 2023-09-01 03:13:55
Duration: 3.82s
Result: Fail

def words_string(s):
    return s.split()