Reference Documentation

More high-level information about the Python grader can be found on the Python Autograder page.

Class to provide user feedback and correctness checking of various datatypes, including NumPy arrays, Matplotlib plots, and Pandas DataFrames.

set_score classmethod

set_score(score: float) -> None

Set the score for the test case, should be a floating point value between 0 and 1.

Examples:

>>> Feedback.set_score(0.75)

add_feedback classmethod

add_feedback(text: str) -> None

Adds some text to the feedback output for the current test.

Examples:

>>> Feedback.add_feedback("The return value is not correct.")

finish classmethod

finish(fb_text: str) -> NoReturn

Complete grading immediately, additionally outputting the message in fb_text.

Examples:

>>> Feedback.finish("Invalid format")

not_allowed staticmethod

not_allowed(*_args: Any, **_kwargs: Any) -> NoReturn

Used to hook into disallowed functions, raises an exception if the student tries to call it.

Note that because Python is a highly-dynamic language, this method can be bypassed by students with sufficient knowledge of Python. For stronger guarantees about which functions are or are not used, consider using more advanced static analysis techniques, which are beyond the scope of what this autograder offers. You can also perform verification by hand with manual grading.

check_numpy_array_sanity classmethod

check_numpy_array_sanity(
    name: str, num_axes: int, data: ArrayLike | None
) -> None

Perform a sanity check on a NumPy array, making sure that it is in fact defined and has the correct dimensionality. If the checks fail then grading will automatically stop.

Parameters:

Name	Type	Description	Default
name	str	Name of the array that is being checked. This will be used to give feedback.	required
num_axes	int	Number of axes that the array should have.	required
data	ArrayLike | None	NumPy array to check.	required

Examples:

>>> Feedback.check_numpy_array_sanity(name, num_axes, data)

check_numpy_array_features classmethod

check_numpy_array_features(
    name: str,
    ref: NDArray[Any],
    data: None | ArrayLike,
    accuracy_critical: bool = False,
    report_failure: bool = True,
    report_success: bool = True,
) -> bool | None

Check that a student NumPy array has the same shape and datatype as a reference solution NumPy array.

Parameters:

Name	Type	Description	Default
name	str	Name of the array that is being checked. This will be used to give feedback.	required
ref	NDArray[Any]	Reference NumPy array.	required
data	None | ArrayLike	Student NumPy array to be checked. Do not mix this up with the previous array! This argument is subject to more strict type checking.	required
accuracy_critical	bool	If true, grading will halt on failure.	False
report_failure	bool	If true, feedback will be given on failure.	True
report_success	bool	If true, feedback will be given on success.	True

Examples:

>>> Feedback.check_numpy_array_features("b", self.ref.a, self.st.b, accuracy_critical=True)

check_numpy_array_allclose classmethod

check_numpy_array_allclose(
    name: str,
    ref: NDArray[Any],
    data: ArrayLike,
    accuracy_critical: bool = False,
    rtol: float = 1e-05,
    atol: float = 1e-08,
    report_success: bool = True,
    report_failure: bool = True,
) -> bool

Check that a student NumPy array has similar values to a reference NumPy array. Note that this checks value according to the numpy.allclose function, which goes by the following check: absolute(a - b) <= (atol + rtol * absolute(b))

Parameters:

Name	Type	Description	Default
name	str	Name of the array that is being checked. This will be used to give feedback.	required
ref	NDArray[Any]	Reference NumPy array.	required
data	ArrayLike	Student NumPy array to be checked. Do not mix this up with the previous array! This argument is subject to more strict type checking.	required
rtol	float	Maximum relative tolerance between values.	1e-05
atol	float	Maximum absolute tolerance between values.	1e-08
accuracy_critical	bool	If true, grading will halt on failure.	False
report_failure	bool	If true, feedback will be given on failure.	True

Examples:

>>> Feedback.check_numpy_array_allclose("G", self.ref.G, self.st.G)

check_dict classmethod

check_dict(
    name: str,
    ref: dict[Any, Any],
    data: dict[Any, Any],
    *,
    target_keys: None | list[str] | set[str] = None,
    accuracy_critical: bool = False,
    report_failure: bool = True,
    report_success: bool = True,
) -> bool

Checks that a student dict (data) has all correct key-value mappings with respect to a reference dict (ref).

Parameters:

Name	Type	Description	Default
name	str	Name of the dict that is being checked. This will be used to give feedback.	required
ref	dict[Any, Any]	Reference dict.	required
data	dict[Any, Any]	Student dict to be checked. Do not mix this up with the previous dict! This argument is subject to more strict type checking.	required
target_keys	None | list[str] | set[str]	If not None, it will only only compare the keys listed.	None
accuracy_critical	bool	If true, grading will halt on failure.	False
report_failure	bool	If true, feedback will be given on failure.	True
report_success	bool	If true, feedback will be given on success.	True

check_list classmethod

check_list(
    name: str,
    ref: list[Any],
    data: list[Any] | None,
    entry_type: Any | None = None,
    accuracy_critical: bool = False,
    report_failure: bool = True,
    report_success: bool = True,
) -> bool

Check that a student list has correct length with respect to a reference list. Can also check for a homogeneous data type for the list.

Parameters:

Name	Type	Description	Default
name	str	Name of the list that is being checked. This will be used to give feedback.	required
ref	list[Any]	Reference list.	required
data	list[Any] | None	Student list to be checked. Do not mix this up with the previous list! This argument is subject to more strict type checking.	required
entry_type	Any | None	If not None, requires that each element in the student solution be of this type.	None
accuracy_critical	bool	If true, grading will halt on failure.	False
report_failure	bool	If true, feedback will be given on failure.	True
report_success	bool	If true, feedback will be given on success.	True

Examples:

>>> Feedback.check_list(name, ref, data)

check_tuple classmethod

check_tuple(
    name: str,
    ref: tuple[Any],
    data: tuple[Any] | None,
    accuracy_critical: bool = False,
    report_failure: bool = True,
    report_success: bool = True,
) -> bool

Check that a student tuple has correct length with respect to a reference tuple, and same values.

Parameters:

Name	Type	Description	Default
name	str	Name of the tuple that is being checked. This will be used to give feedback.	required
ref	tuple[Any]	Reference tuple.	required
data	tuple[Any] | None	Student tuple to be checked. Do not mix this up with the previous tuple! This argument is subject to more strict type checking.	required
accuracy_critical	bool	If true, grading will halt on failure.	False
report_failure	bool	If true, feedback will be given on failure.	True
report_success	bool	If true, feedback will be given on success.	True

Examples:

>>> Feedback.check_tuple(name, ref, data)

check_scalar classmethod

check_scalar(
    name: str,
    ref: complex | number[Any],
    data: complex | number[Any] | None,
    accuracy_critical: bool = False,
    rtol: float = 1e-05,
    atol: float = 1e-08,
    report_success: bool = True,
    report_failure: bool = True,
) -> bool

Check that a student scalar has correct value with respect to a reference scalar. This will mark a value as correct if it passes any of the following checks:

• abs(ref - data) < ref(ref) * rtol
• abs(ref - data) < atol

One of rtol or atol can be omitted (set to None) if that check is unwanted. Or both, but then nothing would be graded :)

Parameters:

Name	Type	Description	Default
name	str	Name of the scalar that is being checked. This will be used to give feedback.	required
ref	complex | number[Any]	Reference scalar.	required
data	complex | number[Any] | None	Student scalar to be checked. Do not mix this up with the previous value! This argument is subject to more strict type checking.	required
accuracy_critical	bool	If true, grading will halt on failure.	False
rtol	float	Maximum relative tolerance.	1e-05
atol	float	Maximum absolute tolerance.	1e-08
report_failure	bool	If true, feedback will be given on failure.	True
report_success	bool	If true, feedback will be given on success.	True

Examples:

>>> Feedback.check_scalar("y", self.ref.y, self.st.y)

call_user classmethod

call_user(
    f: Callable[..., T],
    *args: Any,
    stop_on_exception: bool = True,
    **kwargs: Any,
) -> Any

Attempts to call a student defined function, with any arbitrary arguments. If the student code raises an exception, this will be caught and user feedback will be given. If the function call succeeds, the user return value will be returned from this function.

Note that the keyword argument stop_on_exception will not be passed to the student defined function f, but all other keyword arguments will.

Parameters:

Name	Type	Description	Default
f	Callable[..., T]	Student defined function to be called.	required
*args	Any	Arbitrary positional arguments to be passed to the student function.	()
stop_on_exception	bool	If true, grading will halt on failure.	True
**kwargs	Any	Arbitrary keyword arguments to be passed to the student function.	{}

Examples:

>>> user_val = Feedback.call_user(self.st.fib, 5)

check_plot classmethod

check_plot(
    name: str,
    ref: Axes,
    plot: Axes,
    check_axes_scale: Literal[None, "x", "y", "xy"] = None,
    accuracy_critical: bool = False,
    report_failure: bool = True,
    report_success: bool = True,
) -> bool

Checks that a student plot has the same lines as a reference plot solution. Can optionally check the axis scales to ensure they are the same as the reference.

Parameters:

Name	Type	Description	Default
name	str	Name of plot scalar that is being checked. This will be used to give feedback.	required
ref	Axes	Reference plot.	required
plot	Axes	Student plot to be checked. Do not mix this up with the previous value! This argument is subject to more strict type checking.	required
check_axes_scale	Literal[None, 'x', 'y', 'xy']	Signals which axis scale should be checked against the reference solution.	None
accuracy_critical	bool	If true, grading will halt on failure.	False
report_failure	bool	If true, feedback will be given on failure.	True
report_success	bool	If true, feedback will be given on success.	True

Examples:

>>> Feedback.check_plot("plot", self.ref.plot, self.st.plot, check_axes_scale="xy")

check_dataframe classmethod

check_dataframe(
    name: str,
    ref: DataFrame,
    data: DataFrame,
    subset_columns: list[str] | None = None,
    check_values: bool = True,
    allow_order_variance: bool = True,
    display_input: bool = False,
    report_success: bool = True,
) -> bool

Checks and adds feedback regarding the correctness of a pandas! DataFrame.

Author: Wade Fagen-Ulmschneider (waf)

By default, checks if the student DataFrame data contains the same contents as the reference DataFrame ref by using pandas.testing.assert_frame_equal after basic sanity checks.

Parameters:

Name	Type	Description	Default
name	str	The human-readable name of the DataFrame being checked	required
ref	DataFrame	The reference (correct) DataFrame	required
data	DataFrame	The student DataFrame	required
subset_columns	list[str] | None	If subset_columns is an empty array, all columns are used in the check. Otherwise, only columns named in subset_columns are used in the check and other columns are dropped.	None
check_values	bool	Check the values of each cell, in addition to the dimensions of the DataFrame	True
allow_order_variance	bool	Allow rows to appear in any order (so long as the dimensions and values are correct)	True
display_input	bool	Display the student's answer in the feedback area.	False
report_success	bool	Report success if the DataFrame is correct	True

Base class for test suites, using the Python unittest library. Handles automatic setup and teardown of testing logic.

Methods here do not need to be overridden by test suites.

setUpClass classmethod

setUpClass() -> None

On start, run the user code and generate answer tuples.

tearDownClass classmethod

tearDownClass() -> None

Close all plots and increment the iteration number on test finish

display_plot classmethod

display_plot() -> None

get_total_points classmethod

get_total_points() -> float

Get the total number of points awarded by this test suite, including cases where the test suite is run multiple times.

Test suite that includes plot grading. Will automatically check plots for appropriate labels.